Article In: Pragmatics and Society: Online-First Articles
Six guidelines for trustworthy, ethical and responsible automation design
This content is being prepared for publication; it may be subject to changes.
Abstract
Calibrated trust (Lee, John D., and Katrina A. See. 2004. “Trust in Automation: Designing for Appropriate Reliance.” Human Factors 46 (1): 50–80. ) is critical for safe and seamless integration of automated systems into society. Users should only rely on a system recommendation when it is actually correct and reject it when it is factually wrong. One requirement to achieve this goal is an accurate trustworthiness assessment, ensuring that the user’s perception of the system’s trustworthiness aligns with its actual trustworthiness, allowing users to make informed decisions about the extent to which they can rely on the system (Schlicker, Nadine, Kevin Baum, Alarith Uhde, Sarah Sterz, Martin C. Hirsch, and Markus Langer. 2025. “How do we assess the trustworthiness of AI? Introducing the trustworthiness assessment model (TrAM).” Computers in Human Behavior 1701: 108671. ).
The field of pragmatics offers valuable insights into communication processes emphasizing the importance of context and mutual understanding in communication. These are essential for developing accurate trustworthiness assessments bridging the gap between user expectations and system capabilities.
We propose six design guidelines to help designers foster accurate trustworthiness assessments, and thus ethical and responsible human-automation interactions. The guidelines draw on human-computer interaction, cognitive psychology, automation research, user-experience design, ethics, as well as pragmatics — specifically, the cultivation of common ground (Clark, Herbert. 1996. Using Language. Cambridge: Cambridge University Press. ) and Gricean communication maxims (Grice, Herbert P. 1975. “Logic and Conversation.” In Speech Acts, ed. by Peter Cole and Jerry L. Morgan, 41–58. New York: Academic Press. ). Pragmatic principles provide valuable insights into these contexts, because the user’s perception of the system’s trustworthiness is shaped by both environmental contexts, such as organizational culture or societal norms, but also situational context, including the specific circumstances or scenarios in which the interaction occurs (Hoff, Kevin A., and Masooda Bashir. 2015. “Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust.” Human Factors 57 (3): 407–34. ).
The guidelines provide actionable insights for designers to create automated systems that make relevant trustworthiness cues available and also serve as a tool for evaluating to what extent existing systems enable users to accurately assess a system’s trustworthiness.
Article outline
- 1.Introduction
- 1.1Significance of trust in human-automation interactions
- 1.2Pragmatics as a framework for trust calibration
- 1.3Effective communication as the foundation of an accurate trustworthiness assessment
- 2.Guidelines
- Guideline 1 — Allow verification of automation operation
- Design guidance
- Supporting evidence
- Example
- Caution
- Guideline 2 — Explain automation actions
- Design Guidance
- Supporting evidence
- Example
- Caution
- Guideline 3 — Convey automation uncertainty
- Design guidance
- Supporting evidence
- Example
- Caution
- Guideline 4 — Provide methods to recover from automation errors
- Design guidance
- Supporting evidence
- Example
- Caution
- Guideline 5 — Design automation to conform with social norms and etiquette
- Design Guidance
- Supporting evidence
- Example
- Caution
- Guideline 6 — Provide user training
- Design guidance
- Supporting evidence
- Example
- Caution
- Guideline 1 — Allow verification of automation operation
- 3.Discussion
- 3.1Contributions
- 3.2The need for trustworthiness heuristics
- 3.3How previous lists are not adequate
- 3.4Implications and future research directions
- 4.Conclusion
- Notes
References
References (95)
Arrieta, Alejandro Barredo, Natalia Díaz-Rodríguez, Javier Del Ser, et al. 2019. “Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI.” arXiv.
Axelsson, Minja., Micol Spitale, and Hatice Gunes. 2024. ““Oh, Sorry, I Think I Interrupted You’’: Designing Repair Strategies for Robotic Longitudinal Well-Being Coaching.” In Proceedings of International Conference on Human-Robot Interaction, 13–22. New York: ACM.
Bach, Anne Kathrine Petersen, Trine Munch Nørgaard, Jens Christian Brok, and Niels van Berkel. 2023. “‘If I Had All the Time in the World’: Ophthalmologists’ Perceptions of Anchoring Bias Mitigation in Clinical AI Support.” In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–14. New York:ACM.
Bahner, J. Elin, Monika F. Elepfandt, and Dietrich Manzey. 2008. “Misuse of Diagnostic Aids in Process Control: The Effects of Automation Misses on Complacency and Automation Bias.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 52 (September):1330–34.
Bahner, J. Elin, Anke-Dorothea Hüper, and Dietrich Manzey. 2008. “Misuse of Automated Decision Aids: Complacency, Automation Bias and the Impact of Training Experience.” International Journal of Human-Computer Studies 66 (9): 688–99.
Beller, Johannes, Matthias Heesen, and Mark Vollrath. 2013. “Improving the Driver–Automation Interaction: An Approach Using Automation Uncertainty.” Human Factors 55 (6): 1130–41.
Bonnefon, Jean-François, Azim Shariff, and Iyad Rahwan. 2016. “The Social Dilemma of Autonomous Vehicles.” Science 352 (6293): 1573–1576.
Brown, Penelope, and Stephen C. Levinson. 1987. Politeness: Some Universals in Language Usage. Cambridge: Cambridge University Press.
Buçinca, Zana, Maja Barbara Malaya, and Krzysztof Z. Gajos. 2021. “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making.” Proceedings of the ACM on Human-Computer Interaction 51: 188:1–188:21.
Castelfranchi, Christiano, and Rino Falcone. 2010. Trust Theory: A Socio-Cognitive and Computational Model. Chichester: John Wiley & Sons.
Chen, Jessie, Shan Lakhmani, Kimberly Stowers, Anthony Selkowitz, Julia Wright, and Michael Barnes. 2018. “Situation Awareness-Based Agent Transparency and Human-Autonomy Teaming Effectiveness.” Theoretical Issues in Ergonomics Science 19 (May): 259–282.
Chien, Shih-Yi, Michael Lewis, Katia Sycara, Jyi-Shane Liu, and Asiye Kumru. 2016. “Influence of Cultural Factors in Dynamic Trust in Automation.” In 2016 IEEE SMC, 002884–89.
Clark, Herbert, and Kerstin Fischer. 2023. “Social Robots as Depictions of Social Agents.” Behavioral and Brain Sciences 46 (March): 1–33.
Cohen, Marvin, Raja Parasuraman, Daniel Serfaty, and Robert Andes. 1997. “Trust in decision aids: A model and a training strategy.” Arlington, VA: Cognitive Technologies, Inc.
Culley, Kimberly E., and Poornima Madhavan. 2013. “A Note of Caution Regarding Anthropomorphism in HCI Agents.” Computers in Human Behavior. 29 (3): 577–579.
Cummings, M. L. 2019. “Adaptation of Human Licensing Examinations to the Certification of Autonomous Systems.” In Safe, Autonomous and Intelligent Vehicles, ed. by Huafeng Yu, Xin Li, Richard M. Murray, S. Ramesh, and Claire J. Tomlin. Cham: Springer International Publishing.
de Visser, Ewart J., Marvin Cohen, Amos Freedy, and Raja Parasuraman. 2014. A Design Methodology for Trust Cue Calibration in Cognitive Agents. Vol. 85251.
de Visser, Ewart J., Samuel S. Monfort, Ryan McKendrick, et al. 2016. “Almost Human: Anthropomorphism Increases Trust Resilience in Cognitive Agents.” Journal of Experimental Psychology: Applied. 22 (3): 331–349.
de Visser, Ewart J., Marieke M. M. Peeters, Malte F. Jung, et al. 2020. “Towards a Theory of Longitudinal Trust Calibration in Human–Robot Teams.” International Journal of Social Robotics. 12 (2): 459–478.
de Visser, Ewart J., Ali Momen, James C. Walliser, et al. 2023. “Mutually Adaptive Trust Calibration in Human-AI Teams.” Hybrid Human-Artificial Intelligence, Munich, Germany.
de Winter, Joost, and Dimitra Dodou. 2014. “Why the Fitts List Has Persisted throughout the History of Function Allocation.” Cognition, Technology & Work 16 (1): 1–11.
Degani, Asaf, Michael Shafto, and Alex Kirlik. 1999. “Modes in Human-Machine Systems: Constructs, Representation, and Classification.” International Journal of Aviation Psychology 9 (2): 125–138.
“DIB Guide: Detecting Agile BS.” 2018. U.S. Department of Defense. [URL]
Dorton, Stephen L., and Jeff C. Stanley. 2024. “Minding the Gap: Tools for Trust Engineering of Artificial Intelligence.” Ergonomics in Design 33 (3): 10648046241249903.
Du, Na, Kevin Y. Huang, and X. Jessie Yang. 2020. “Not All Information Is Equal: Effects of Disclosing Different Types of Likelihood Information on Trust, Compliance and Reliance, and Task Performance in Human-Automation Teaming.” Human Factors 62 (6): 987–1001.
Dzindolet, Mary T., Scott A. Peterson, Regina A. Pomranky, Linda G. Pierce, and Hall P. Beck. 2003. “The Role of Trust in Automation Reliance.” International Journal of Human-Computer Studies, Trust and Technology 58 (6): 697–718.
Endsley, Mica R. 2023. “Supporting Human-AI Teams: Transparency, Explainability, and Situation Awareness.” Computers in Human Behavior 1401: 107574.
European Parliament. 2024. “Artificial Intelligence Act.” P9_TA(2024)0138. [URL]
Ezer, Neta, Sylvain Bruni, Yang Cai, et al. 2019. “Trust Engineering for Human-AI Teams.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 631: 322–326.
Ferrario, Andrea, and Michele Loi. 2022. “How Explainability Contributes to Trust in AI.” In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency: 1457–1466. New York: ACM.
Finger, Richard, and Ann M. Bisantz. 2002. “Utilizing Graphical Formats to Convey Uncertainty in a Decision-Making Task.” Theoretical Issues in Ergonomics Science. 3 (1): 1–25.
Fischer, Kerstin, Hanna M. Weigelin, and Leon Bodenhagen. 2018. “Increasing trust in human–robot medical interactions: effects of transparency and adaptability.” Paladyn, Journal of Behavioral Robotics 9 (1): 95–109.
Gajos, Krzysztof Z., and Lena Mamykina. 2022. “Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning.” In Proceedings of the 27th International Conference on Intelligent User Interfaces, 794–806. New York: ACM.
Grice, Herbert P. 1975. “Logic and Conversation.” In Speech Acts, ed. by Peter Cole and Jerry L. Morgan, 41–58. New York: Academic Press.
Hancock, Peter A., Deborah R. Billings, Kristin E. Schaefer, et al. 2011. “A meta-analysis of factors affecting trust in human-robot interaction.” Human Factors 53(5): 517–527.
Hayes, Caroline C., and Christopher A. Miller. 2011. Human-Computer Etiquette: Cultural Expectations and the Design Implications They Place on Computers and Technology. Boca Raton: CRC Press.
Helldin, Tove, Göran Falkman, Maria Riveiro, and Staffan Davidsson. 2013. “Presenting System Uncertainty in Automotive UIs for Supporting Trust Calibration in Autonomous Driving.” In Proceedings 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications.
Ho, Nhut, Garrett G. Sadler, Lauren C. Hoffmann, et al. 2017. “A longitudinal field study of auto-GCAS acceptance and trust: First-year results and implications.” Journal of Cognitive Engineering and Decision Making 11 (3): 239–251.
Hoff, Kevin A., and Masooda Bashir. 2015. “Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust.” Human Factors 57 (3): 407–34.
Jakobson, Roman. 1960. “Linguistics and Poetics.” In Style in Language, ed. by Thomas A. Sebeok, 350–377. Cambridge, MA: MIT Press.
Jakubik, Johannes, Jakob Schoeffer, Vincent Hoge, Michael Vössing, and Niklas Kühl. 2022. “An Empirical Evaluation of Predicted Outcomes as Explanations in Human-AI Decision-Making.” In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 353–368. Cham: Springer Nature Switzerland.
Jobin, Anna, Marcello Ienca, and Effy Vayena. 2019. “The Global Landscape of AI Ethics Guidelines.” Nature Machine Intelligence 11: 389–399.
Kaber, David B. 2018. “A conceptual framework of autonomous and automated agents.” Theoretical Issues in Ergonomics Science 19(4): 406–430.
Klien, Glen, David D. Woods, Jeffrey M. Bradshaw, Robert R. Hoffman, and Paul J. Feltovich. 2004. “Ten Challenges for Making Automation a ‘Team Player’ in Joint Human-Agent Activity.” IEEE Intelligent Systems 19 (6): 91–95.
Kopp, Tobias, Marco Baumgartner, and Steffen Kinkel. 2023. “‘It’s Not Paul, It’s a Robot’: The Impact of Linguistic Framing and the Evolution of Trust and Distrust in a Collaborative Robot during a Human-Robot Interaction.” International Journal of Human-Computer Studies 1781: 103095.
Langer, Markus, Daniel Oster, Timo Speith, et al. 2021. “What Do We Want from Explainable Artificial Intelligence (XAI)? — A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research.” Artificial Intelligence 2961:103473.
Lee, John D., and Katrina A. See. 2004. “Trust in Automation: Designing for Appropriate Reliance.” Human Factors 46 (1): 50–80.
Liu, Le, Mirzargar, M., Kirby, R. M., Whitaker, R. and D. H. House. 2015. “Visualizing Time-Specific Hurricane Predictions, with Uncertainty, from Storm Path Ensembles.” Computer Graphics Forum 34 (3): 371–380.
Lucas, Gale M., Jonathan Gratch, Aisha King, and Louis-Philippe Morency. 2014. “It’s Only a Computer: Virtual Humans Increase Willingness to Disclose.” Computers in Human Behavior 371: 94–100.
Lyons, Joseph B., Kerianne Hobbs, Steve Rogers, and Scott H. Clouse. 2023. “Responsible (Use of) AI.” Frontiers in Neuroergonomics 41.
Madhavan, Poornima, and Douglas A. Wiegmann. 2007. “Similarities and Differences between Human–Human and Human–Automation Trust: An Integrative Review.” Theoretical Issues in Ergonomics Science 8 (4): 277–301.
Madhavan, Poornima, Douglas Wiegmann, and Frank Lacson. 2006. “Automation Failures on Tasks Easily Performed by Operators Undermine Trust in Automated Aids.” Human Factors 481: 241–256.
Mayer, Roger C., James H. Davis, and F. David Schoorman. 1995. “An Integrative Model of Organizational Trust.” The Academy of Management Review 20 (3): 709–734.
McBride, Sara E., Wendy A. Rogers, and Arthur D. Fisk. 2014. “Understanding Human Management of Automation Errors.” Theoretical Issues in Ergonomics Science 15 (6): 545–577.
McVay, Jennifer C., and Ewart J. de Visser. 2025. “Challenges and Choices when Evaluating Alignment in Human-AI Systems.” Proceedings of the AAAI Symposium Series 7 (1): 214–222.
Mercado, Joseph E., Michael A. Rupp, Jessie Y. C. Chen, Michael J. Barnes, Daniel Barber, and Katelyn Procci. 2016. “Intelligent Agent Transparency in Human–Agent Teaming for Multi-UxV Management.” Human Factors 58 (3): 401–415.
Miller, Christopher A. 2021. “Trust, Transparency, Explanation, and Planning: Why We Need a Lifecycle Perspective on Human-Automation Interaction.” In Trust in Human-Robot Interaction, ed. by Chang S. Nam, and Joseph B. Lyons, 233–257. San Diego: Academic Press.
Miller, Christopher, Daniel Barber, Eric Holder, et al. 2023. “LifeCycle Transparency: Why, and How, Transparency Information Exchange Should Be Distributed throughout the Life of Technology Usage.” In Proceedings of the Human Factors and Ergonomics Society Annual Meeting 67(1): 409–413.
Miller, Tim. 2023. “Explainable AI Is Dead, Long Live Explainable AI! Hypothesis-Driven Decision Support Using Evaluative AI.” In Proceedings of the Conference on Fairness, Accountability, and Transparency. New York: ACM.
Momen, Ali., Ewart J. de Visser, Marlena R. Fraune, et al. 2023. “Group trust dynamics during a risky driving experience in a Tesla Model X.” Frontiers in Psychology 141: 1129369.
Mori, Masahiro, Karl MacDorman, and Norri Kageki. 2012. “The Uncanny Valley [From the Field].” Robotics & Automation Magazine 19 (2): 98–100.
Nass, Clifford, and Youngme Moon. 2000. “Machines and mindlessness: Social responses to computers.” Journal of Social Issues 56(1): 81–103.
Okamura, Kazuo, and Seiji Yamada. 2020. “Adaptive Trust Calibration for Human-AI Collaboration.” PLOS ONE 15 (2): e0229132.
Onnasch, Linda, and Eileen Roesler. 2019. “Anthropomorphizing Robots.” Proceedings of the Human Factors and Ergonomics Society Annual Meeting 63(1): 1311–1315.
Onnasch, Linda, Christopher D. Wickens, Huiyang Li, and Dietrich Manzey. 2014. “Human performance consequences of stages and levels of automation: An integrated meta-analysis.” Human Factors 56 (3): 476–488.
Onnasch, Linda, Eileen Roesler, Lionel P. Robert, and Ewart de Visser. 2026. “Trust (worthiness) Issues with Trust in Human-Robot Interaction.” ACM Transactions on Human-Robot Interaction 15 (3): 1–19.
Pak, Richard, Nicole Fink, Margaux Price, Brock Bass, and Lindsay Sturre. 2012. “Decision Support Aids with Anthropomorphic Characteristics Influence Trust and Performance in Younger and Older Adults.” Ergonomics 55 (9): 1059–1072.
Parasuraman, Raja, and Dietrich Manzey. 2010. “Complacency and Bias in Human Use of Automation: An Attentional Integration.” Human Factors 52 (3): 381–410.
Parasuraman, Raja, and Christopher A. Miller. 2004. “Trust and Etiquette in High-Criticality Automated Systems.” Commun. ACM 47 (4): 51–55.
Parasuraman, Raja, and Victor Riley. 1997. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors 39 (2): 230–253.
Parasuraman, Raja, and Christopher Wickens. 2008. “Humans: Still Vital After All These Years of Automation.” Human Factors 501: 511–520.
Pelau, Corina, Dan-Cristian Dabija, and Irina Ene. 2021. “What Makes an AI Device Human-like? The Role of Interaction Quality, Empathy and Perceived Psychological Anthropomorphic Characteristics in the Acceptance of Artificial Intelligence in the Service Industry.” Computers in Human Behavior 1221: 106855.
Pflanzer, Michael, Zachary Traylor, Joseph B. Lyons, Veljko Dubljević, and Chang S. Nam. 2023. “Ethics in Human–AI Teaming: Principles and Perspectives.” AI Ethics 3 (3): 917–35.
Phillips, Elizabeth K., and Ewart J. de Visser. 2026. “Humanlikeness as Design, Anthropomorphism as Inference: A Conceptual Framework for Human–Robot Interaction.” Frontiers in Cognition 51: 1786256.
Reeves, Byron, and Clifford Nass. 1996. The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge: Cambridge University Press.
Rice, Stephen, and Jason S. McCarley. 2011. “Effects of Response Bias and Judgment Framing on Operator Use of an Automated Aid in a Target Detection Task.” Journal of Experimental Psychology: Applied 17 (4): 320–331.
Rieger, Tobias, Dietrich Manzey, Benigna Meussling, Linda Onnasch, and Eileen Roesler. 2023. “Be Careful What You Explain: Benefits and Costs of Explainable AI in a Simulated Medical Task.” Computers in Human Behavior: Artificial Humans 1 (2): 100021.
Ries, Anthony J., Stéphane Aroca-Ouellette, Alessandro Roncone, and Ewart J. de Visser. 2025. “Gaze-Informed Signatures of Trust and Collaboration in Human-Autonomy Teams.” Computers in Human Behavior: Artificial Humans 51: 100171.
Roesler, Eileen, Dietrich Manzey, and Linda Onnasch. 2021. “A Meta-Analysis on the Effectiveness of Anthropomorphism in Human-Robot Interaction,” Science Robotics 6 (58).
Rouse, William B., and Nancy M. Morris. 1986. “On Looking into the Black Box: Prospects and Limits in the Search for Mental Models.” Psychological Bulletin 100 (3): 349–363.
Sanchez, Julian, Wendy A. Rogers, Arthur D. Fisk, and Ericka Rovira. 2014. “Understanding Reliance on Automation: Effects of Error Type, Error Distribution, Age and Experience.” Theoretical Issues in Ergonomics Science 15 (2): 134–160.
Schlicker, Nadine, Kevin Baum, Alarith Uhde, Sarah Sterz, Martin C. Hirsch, and Markus Langer. 2025. “How do we assess the trustworthiness of AI? Introducing the trustworthiness assessment model (TrAM).” Computers in Human Behavior 1701: 108671.
Sorkin, Robert D., Barry H. Kantowitz, and Susan C. Kantowitz. 1988. “Likelihood Alarm Displays.” Human Factors 30 (4): 445–459.
Spiegelhalter, David, Mike Pearson, and Ian Short. 2011. “Visualizing Uncertainty About the Future.” Science 3331: 1393–1400.
Sterz, Sarah, Kevin Baum, Sebastian Biewer, et al. 2024. “On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives.” In ACM Conference on Fairness, Accountability, and Transparency. New York: ACM.
Verberne, Frank M. F., Jaap Ham, and Cees J. H. Midden. 2012. “Trust in Smart Systems: Sharing Driving Goals and Giving Information to Increase Trustworthiness and Acceptability of Smart Systems in Cars.” Human Factors 54 (5): 799–810.
Walliser, James C., Ewart J. de Visser, Eva Wiese, and Tyler H. Shaw. 2019. “Team Structure and Team Building Improve Human–Machine Teaming With Autonomous Agents.” Journal of Cognitive Engineering and Decision Making 13 (4): 258–278.
Walliser, James C., Ewart J. de Visser, and Tyler H. Shaw. 2023. “Exploring System Wide Trust Prevalence and Mitigation Strategies with Multiple Autonomous Agents.” Computers in Human Behavior 1431: 107671.
Wickens, Christopher D., and Stephen R. Dixon. 2007. “The Benefits of Imperfect Diagnostic Automation: A Synthesis of the Literature.” Theoretical Issues in Ergonomics Science 8 (3): 201–212.