Doing social robotics with conversation analysis. Towards the development of an automatic system for the prediction of
disengagement
Nicolas Rollet | Télécom Paris | Institut polytechnique de Paris | I3 CNRS
Chloé Clavel | Télécom Paris | Institut polytechnique de Paris | Université Paris-Saclay
This article presents an applied discussion of the possibility of integrating conversation analysis (CA)
methodology into that of machine learning. The aim is to improve the detection of that which resembles disengagement in the
interaction between a robot and a human. We offer a novel analytical assemblage at the heart of the two disciplines, and namely on
the level of the annotation schemes provided by conversation analysis transcription methods. First, we demonstrate that the need
for a stable structure in establishing an interaction scenario and in designing robot behaviours does not prevent the emergence of
ordinariness or creativity among the participants engaged in this interaction. Secondly, based on an actual case, we emphasize the
possibility of systematicness in CA transcription to support the choice (a) of the categories targeted by prediction methods and
defined by the annotation scheme, and (b) of the verbal and non-verbal features used to create prediction models.
Bartneck, C., Belpaeme, T., Eyssel, F., Kanda, T., Keijsers, M., & Šabanović, S.
(2019) Human-robot interaction: An introduction. Cambridge: Cambridge University Press.
Beach, W. A.
(1993) Transitional regularities for ‘casual’ “Okay” usages. Journal of Pragmatics, 191, 325–352.
Ben-Youssef, A., Clavel, C., Essid, S., Bilac, M., Chamoux, M., & Lim, A.
(2017) UE- HRI: a new dataset for the study of user engagement in spontaneous human- robot interactions. Paper presented at the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK.
Breazeal, C.
(2003) Toward sociable robots. Robotics and Autonomous Systems, 421, 167–175.
Button, G.
(1991) Conversation-in-a-series. In D. Boden & D. H. Zimmerman (Eds.), Talk and social structure. Studies in Ethnomethodology and Conversation Analysis (pp. 251–277). Cambridge: Polity Press.
Campano, S., Clavel, C., & Pélachaud, C.
(2015) “I like this painting too”: when an ECA shares appreciations to engage users. Paper presented at the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15), Istanbul, Turkey.
Cassell, J., Torres, O., & Prevost, S.
(1999) Turn taking vs. discourse structure: How best to model multimodal conversation. In Y. Wilks (Ed.), Machine Conversations (pp. 143–154). The Hague: Kluwer.
(2011) Fiction support for realistic portrayals of fear-type emotional manifestations. Computer Speech & Language, 25(1), 63–83.
Clavel, C., Cafaro, A., Campano, S., & Pelachaud, C.
(2016) Fostering User Engagement in Face-to-Face Human-Agent Interactions: A Survey. In A. Esposito & L. C. Jain (Eds.), Toward Robotic Socially Believable Behaving Systems – Volume II: Modeling Social Signals (pp. 93–120). Cham: Springer International Publishing.
Clavel, C., & Callejas, Z.
(2016) Sentiment analysis: from opinion mining to human-agent interaction. IEEE Transactions on affective computing, 7(1), 74–93.
Cowie, R., & Cornelius, R. R.
(2003) Describing the emotional states that are expressed in speech. Speech communication, 40(1–2), 5–32.
Dautenhahn, K.
(2007) Socially intelligent robots: dimensions of human-robot interaction. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 362(1480), 679–704.
(1997) Linguistic anthropology. New York: Cambridge University Press.
Duranti, A., & Goodwin, C.
(1992) Rethinking context: an introduction. In A. Duranti & C. Goodwin (Eds.), Rethinking Context, language as an interactive phenomenon (pp. 1–42). Cambridge University Press.
Fong, T., Nourbakhsh, I., & Dautenhahn, K.
(2003) A survey of socially interactive robots. Robotics and Autonomous Systems, 42(3–4), 143–166.
Ford, C., & Thompson, S.
(1996) Interactional units in conversation: syntactic, intonational, and pragmatic resources for the management of turns. In E. Ochs, E. Schegloff, & S. Thompson (Eds.), Interaction and Grammar (134-184): Cambridge University Press.
Garfinkel, H.
(1967) Studies in ethnomethodology. Engelwood Cliffs: Prentice-Hall.
Goffman, E.
(1973) La mise en scène de la vie quotidienne (Vol. 21. Les relations en public). Paris: Les Editions de minuit.
Goffman, E.
(1983) The interaction order. American Sociological Review, 48(1), pp: 1–17.
Goodwin, C.
(1981) Conversational organization: interaction between speakers and hearers. New York: Academic Press.
Heritage, J.
(1991) L’Ethnométhodologie : une approche procédurale de l’action et de la communication. Réseaux CNET, 501, 89–130.
Jefferson, G.
(1978) Sequential aspects of storytelling in conversation. In J. Schenkein (Ed.), Studies in the organization of conversational interaction (pp. 219–248). New York: New York Academic Press.
Jones, R. A.
(2017) What makes a robot “social”?Social Studies of Science, 47(4), 556–579.
Langlet, C., & Clavel, C.
(2014) Modelling user’s attitudinal reactions to the agent utterances: focus on the verbal content. Paper presented at the 5th International Workshop on Corpora for Research on Emotion, Sentiment & Social Signals (ES3 2014), Reykjavik, Iceland.
Langlet, C., & Clavel, C.
(2018) Detecting User’s Likes and Dislikes for a Virtual Negotiating Agent. Paper presented at the 20th ACM International Conference on Multimodal Interaction, Boulder, USA.
Levinson, S. C.
(1983) Pragmatics. Cambridge University Press.
Licoppe, C., & Figeac, J.
(2014) L’Organisation temporelle des engagements visuels dans des situations de multi-activité équipée en milieu urbain. Activités, 11(1).
Licoppe, C., & Rollet, N.
(2020, in press) « Je dois y aller ». Analyses de séquences de clôtures entre humains et robot. Réseaux. La Découverte.
Mondada, L.
(2006) Participants’ online analysis and multimodal practices: projecting the end of the turn and the closing of the sequence. Discourse Studies, 8(1), pp. 117–129.
Mondada, L.
(2008) Documenter l’articulation des ressources multimodales dans le temps : la transcription d’enregistrements vidéos d’interactions. In M. Bilger (Ed.), Données orales. Les enjeux de la transcription (Vol. 371), pp. 127–156. Perpignan: Presses Universitaires de Perpignan.
Mohri, M., Rostamizadeh, A., & Talwalkar, A.
(2012) Foundations of machine learning. MIT press.
Ochs, E.
(1979) Transcription as a theory. In B. B. Schieffelin (Ed.), Developmental Pragmatics (pp. 42–72). Academic Press.
Pelachaud, C., & Glas, N.
(2015a) Definitions of Engagement in Human-Agent Interaction. Paper presented at the International Workshop on Engagement in Human Computer Interaction (ENHANCE), Xi’an, China.
Pelachaud, C., & Glas, N.
(2015b) Topic transition strategies for an information-giving agent. Paper presented at the 15th European Workshop on natural Language Generation, Brighton, UK.
Pelikan, H. R. M., & Broth, M.
(2016) Why That Nao?: How Humans Adapt to a Conventional Humanoid Robot in Taking Turns-at-Talk. Paper presented at the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, California, USA.
Pitsch, K., Kuzuoka, H., Suzuki, Y., Süssenbach, L., Luff, P., & Heath, C.
(2009) “The first five seconds”: Contingent stepwise entry into an interaction as a means to secure sustained engagement in Human-Robot-Interaction. Paper presented at the IEEE International Symposium on Robot and Human Interactive Communication ROMAN 2009, Toyama, Japan.
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S.
(2018) Voice Interfaces in Everyday Life. Montreal QC, Canada: Association for Computing Machinery.
Rollet, N.
(2010) All the things you are. Activité multimodale, frontières et musiques improvisées en répétition In N. Andrieux-Reix (Ed.), Frontières. Du linguistique au sémiotique (pp. 279–302). Limoges: Lambert-Lucas.
Rollet, N.
(2013) “D’accord”. Approche conversationnelle et multimodale d’une forme située dans les appels au Samu-Centre 15. L’Information grammaticale, 1391.
Rollet, N., Jain, V., Licoppe, C., & Devillers, L.
(2017) Towards Interactional Symbiosis: Epistemic Balance and Co-presence in a Quantified Self Experiment. In L. Gamberini, A. Spagnolli, G. Jacucci, B. Blankertz, & J. Freeman (Eds.), Symbiotic Interaction: 5th International Workshop, Symbiotic 2016, Padua, Italy, September 29–30, 2016, Revised Selected Papers (pp. 143–154). Cham: Springer International Publishing.
Sacks, H.
(1984) On doing ‘being ordinary’. In J. M. Atkinson & J. Heritage (Eds.), Structures of Social Action (pp. 413–429). Cambridge: CUP.
Sacks, H.
(1992) Lectures on conversation (Jefferson, G. ed.). Oxford: Blackwell.
Sacks, H., & Schegloff, E.
(1973) Opening up closings. Semiotica, 81, 289–326.
Sacks, H., Schegloff, E. A., & Jefferson, G.
(1974) A symplest systematics for the organization of turn-taking for conversation. Language, 501, 696–731.
Sadazuka, K., Kuno, Y., Kawashima, M., & Yamazaki, K.
(2007) Museum Guide Robot with Effective Head Gestures. Paper presented at the International Conference on Control, Automation and Systems, Seoul, Korea.
Schegloff, E.
(1987) Between micro and macro: contexts and other connections. In B. Giesen, J. C. Alexander, R. Münch, & N. J. Smelser (Eds.), The Micro-macro link (pp. 207–234). Los Angeles: University of California Press.
Schegloff, E.
(2002) Accounts of conduct in interaction. Interruption, overlap, and turn-taking. In T. J. H. (Ed.), Handbook of sociological theory (pp. 287–321). New York: Kluwer Academic/Plenum publishers.
Schegloff, E.
(2007) Sequence organization in interaction. A Primer in Conversation Analysis. (Vol. 11): Cambridge University Press.
Sidner, C. L., & Dzikovska, M.
(200216–16Oct. 2002) Human-robot interaction: engagement between humans and robots for hosting activities. Paper presented at the 4th IEEE International Conference on Multimodal Interfaces, Pittsburgh, USA.
Sidner, C. L., Lee, C., Kidd, C. D., & Rich, C.
(2005) Explorations in engagement for humans and robots. Artificial Intelligence, 1661, 140–164.
Stivers, T.
(2008) Stance, Alignment, and Affiliation During Storytelling: When Nodding Is a Token of Affiliation. Research on Language & Social Interaction, 44(1), 31–57.
Stivers, T.
(2015) Coding Social Interaction: A Heretical Approach in Conversation Analysis?Research on Language and Social Interaction, 48(1), 1–19.
Šabanović, S., & Chang, W.-L.
(2016) Socializing robots: constructing robotic sociality in the design and use of the assistive robot PARO. AI and Society, 31(4), 537–551.
Straub, I.
(2016) “It looks like a human!” The interrelation of social presence, interaction and agency ascription: a case study about the effects of an android robot on social agency ascription. AI and Society, 31(4), 553–571.
Suchman, L.
(2007) Human-Machine Reconfigurations. Plans and situated actions, 2nd edition. New York: Cambridge University Press.
Yu, Z., Scherer, S., Devault, D., Gratch, J., Stratou, G., Morency, L., & Cassell, J.
(2013) Multimodal Prediction of Psychological Disorder: Learning Verbal and Nonverbal Commonality in Adjacency Pairs. Paper presented at the 17th Workshop Series on the Semantics and Pragmatics of Dialogue, Amsterdam, Netherland.
Zimmerman, D.
(2006) How closing matters in emergency telephone calls. Paper presented at the Annual meeting of the American Sociology Association, Montréal, Canada.
Cited by
Cited by 1 other publications
Kopp, Stefan & Nicole Krämer
2021. Revisiting Human-Agent Communication: The Importance of Joint Co-construction and Understanding Mental States. Frontiers in Psychology 12
This list is based on CrossRef data as of 15 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.