Justification optional: ChatGPT’s advice can still influence human judgments about moral dilemmas

Abstract Why do users follow moral advice from chatbots? Arguably, a chatbot is not an authoritative moral advisor, but it can generate plausible arguments. We conducted a large pre-registered vignette experiment ( N = 1269) that controlled for the effect that ethical justification has on subjects’ propensity to accept the chatbot’s advice. Furthermore, to study the influence of the source of the advice, we manipulated subjects’ belief of being advised by a chatbot or another human. In our experiment, we find that users did not accept reasoned more readily than unreasoned advice. However, this was also true if we attributed advice to a moral advisor, not a chatbot. Hence, we suggest that advice might offer users an easy way to escape from a moral dilemma. This is a concern that chatbots do not raise, but they may exacerbate it as they make advice on versatile issues easily accessible. We conclude that it may take ethical in addition to digital literacy to protect users against moral advice from chatbots.