Cover not available
Article published In:
Interpreting
Vol. 21:2 (2019) ► pp.245–269

Holistic assessment of consecutive interpretation

How interpreter trainers rate student performances

| Hankuk University of Foreign Studies
The article aims to show how interpreter trainers holistically grade student performances. For this purpose, experimental rating sessions were held for four undergraduate interpreter trainers. The raters were asked to think aloud their quality judgments while holistically assessing six recordings of consecutive interpretation. Their concurrent verbal reports, along with reflective reports, interview transcripts, and video recordings of computer screen activity, were collected and analysed in detail. Findings revealed various facets of interpreting performance assessment, including what procedures the raters followed, what aspects of the performance they focused on, what criteria they depended on for their judgment decisions, and why two ratings of the same performance were divergent. This article also presents a tentative model for holistic rating of consecutive interpretation.
Keywords: raters, quality judgment, rating process, think-aloud protocol (TAP), coding
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Procedural dimension of interpreting performance assessment
    • 2.2Thinking aloud as a data elicitation method
  • 3.Method
    • 3.1Participants
    • 3.2Task materials
    • 3.3Procedures
    • 3.4Coding schemes
    • 3.5Data analysis
  • 4.Results
    • 4.1Lengths of the verbal protocols
    • 4.2Grades
    • 4.3Rating styles and methods
      • 4.3.1R1
      • 4.3.2R2
      • 4.3.3R3
      • 4.3.4R4
    • 4.4Distribution of positive and negative comments by protocol code
      • 4.4.1R1
      • 4.4.2R2
      • 4.4.3R3
      • 4.4.4R4
    • 4.5A tentative model of the holistic rating process
  • 5.Concluding remarks
  • Acknowledgements
  • Notes
  • References
Published online: 11 November 2019
DOI logo
https://doi.org/10.1075/intp.00029.lee
References (57)
References
Bartłomiejczyk, M. (2007). Interpreting quality as perceived by trainee interpreters: Self-evaluation. The Interpreter and Translator Trainer 1 (2), 247–267. DOI logoGoogle Scholar
Bejar, I. I. (2012). Rater cognition: Implications for validity. Educational Measurement: Issues and Practice 31 (3), 2–9. DOI logoGoogle Scholar
Bowles, M. A. (2010). The think-aloud controversy in second language research. London: Routledge. DOI logoGoogle Scholar
Bühler, H. (1986). Linguistic (semantic) and extralinguistic (pragmatic) criteria for the evaluation of conference interpretation and interpreters. Multilingua 5 (4), 231–235.Google Scholar
Chabasse, C. & Kader, S. (2014). Putting interpreting admissions exams to the test. Interpreting 16 (1), 19–33. DOI logoGoogle Scholar
Chiaro, D. & Nocella, G. (2004). Interpreters’ perception of linguistic and non-linguistic factors affecting quality: A survey through the World Wide Web. Meta 49 (2), 279–293. DOI logoGoogle Scholar
Choi, J. Y. (2013). Assessing the impact of text length on consecutive interpreting. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 85–96.Google Scholar
Cifuentes-Férez, P. & Rojo, A. (2015). Thinking for translating: A think-aloud protocol on the translation of manner-of-motion verbs. Target 27 (2), 273–300. DOI logoGoogle Scholar
Coffey, A. & Atkinson, P. (1996). Making sense of qualitative data: Complementary research strategies. Thousand Oaks, CA: Sage.Google Scholar
Cohen, A. D. (2000). Exploring strategies in test-taking: Fine-tuning verbal reports from respondents. In G. Ekbatani & H. Pierson (Eds.), Learner-directed assessment in ESL. Mahwah, NJ: Lawrence Erlbaum, 127–150.Google Scholar
De Gregoris, G. (2014). The limits of expectations vs. assessment questionnaire-based surveys on simultaneous interpreting quality: The need for a gestaltic model of perception. Rivista internazionale di tecnica della traduzione 161, 57–87.Google Scholar
DeRember, M. L. (1998). Writing assessment: Raters’ elaboration of the rating task. Assessing Writing 5 (1), 7–29. DOI logoGoogle Scholar
Englund Dimitrova, B. & Tiselius, E. (2014). Retrospection in interpreting and translation: Explaining the process? MonTI Special Issue 11, 177–200. DOI logoGoogle Scholar
Ericsson, K. A. (2000/2001). Expertise in interpreting: An expert-performance perspective. Interpreting 5 (2), 187–220. DOI logoGoogle Scholar
Ericsson, K. A. & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press. DOI logoGoogle Scholar
(1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture and Activity 5 (3), 178–186. DOI logoGoogle Scholar
Eyckmans, J., Anckaert, P. & Segers, W. (2016). Translation and interpretation skills. In D. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment. Berlin: Walter de Gruyter, 219–235. DOI logoGoogle Scholar
Garzone, G. (2003). Reliability of quality criteria evaluation in survey research. In A. Collados Aís, M. M. Fernández Sánchez & D. Gile (Eds.), La evaluación de la calidad en interpretación: Investigación. Granada: Comares.Google Scholar
Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge: Cambridge University Press.Google Scholar
Iglesias Fernández, E. (2013). Unpacking delivery criteria in interpreting quality assessment. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 51–66.Google Scholar
Ivanova, A. (2000). The use of retrospection in research on simultaneous interpreting. In S. Tirkkonen-Condit & R. Jääskeläinen (Eds.), Tapping and mapping the processes of translation and interpreting: Outlooks on empirical research. Amsterdam: John Benjamins, 27–52. DOI logoGoogle Scholar
Jakobsen, A. L. (2003). Effects of think aloud on translation speed, revision, and segmentation. In F. Alves (Ed.), Triangulating translation: Perspectives in process-oriented research. Amsterdam: John Benjamins, 69–95. DOI logoGoogle Scholar
Jarvella, R. J., Jensen, A., Jensen, E. H. & Anderson, M. S. (2002). Towards characterizing translator expertise, knowledge and know-how: Some findings using TAPs and experimental methods. In A. Riccardi (Ed.), Translation studies: Perspectives on an emerging discipline. Cambridge: Cambridge University Press, 172–197.Google Scholar
Jourdenais, R. (2001). Cognition, instruction and protocol analysis. In P. Robinson (Ed.), Cognition and second language instruction. Cambridge: Cambridge University Press, 354–375. DOI logoGoogle Scholar
Künzli, A. (2007). Translation revision: A study of the performance of ten professional translators revising a legal text. In Y. Gambier, M. Shlesinger & R. Stolze (Eds.), Doubts and directions in translation studies. Amsterdam: John Benjamins, 115–126. DOI logoGoogle Scholar
Lee, J. (2009). Toward more reliable assessment of interpreting performance. In S. Hale, U. Ozolins & L. Stern (Eds.), The Critical Link 5: Quality in interpreting – a shared responsibility. Amsterdam: John Benjamins, 171–185. DOI logoGoogle Scholar
Lee, S.-B. (2014). An interpreting self-efficacy (ISE) scale for undergraduate students majoring in consecutive interpreting: Construction and preliminary validation. The Interpreter and Translator Trainer 8 (2), 183–203. DOI logoGoogle Scholar
Li, D. (2004). Trustworthiness of think-aloud protocols in the study of translation processes. International Journal of Applied Linguistics 14 (3), 301–313. DOI logoGoogle Scholar
Liu, M. (2013). Design and analysis of Taiwan’s interpretation certification examination. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 163–178.Google Scholar
(2015). Assessment. In F. Pöchhacker (Ed.), Routledge encyclopedia of interpreting studies. London: Routledge, 20–22.Google Scholar
Lörscher, W. (2005). The translation process: Methods and problems of its investigation. Meta 50 (2), 597–608. DOI logoGoogle Scholar
Miles, M. B., Huberman, A. M. & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
Naumenko, O. (2015). Improving performance assessment score validation practices: An instructional module on generalizability theory. Working Papers on Language and Diversity in Education I (1), 1–17.Google Scholar
O’Hagan, S. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang.Google Scholar
Pöchhacker, F. (2001). Quality assessment in conference and community interpreting. Meta 46 (2), 410–425. DOI logoGoogle Scholar
(2011). Assessing aptitude for interpreting: The SynCloze test. Interpreting 13 (1), 106–120. DOI logoGoogle Scholar
(2016). Introducing interpreting studies (2nd ed.). London: Routledge. DOI logoGoogle Scholar
Pressley, M. & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Rallis, S. F. & Rossman, G. B. (2003). Mixed methods in evaluation contexts: A pragmatic framework. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage, 491–512.Google Scholar
Russell, D. & Winston, B. (2014). Tapping into the interpreting process: Using participant reports to inform the interpreting process in educational settings. Translation & Interpreting 6 (1), 102–127.Google Scholar
Saldaña, J. (2016). The coding manual for qualitative researchers (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
Saldanha, G. & O’Brien, S. (2013). Research methodologies in translation studies. London: Routledge.Google Scholar
Sawyer, D. B. (2004). Fundamental aspects of interpreter education: Curriculum and assessment. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Sun, S. (2011). Think-aloud-based translation process research: Some methodological considerations. Meta 56 (4), 928–951. DOI logoGoogle Scholar
Tiselius, E. (2009). Revisiting Carroll’s scales. In C. Angelelli & H. E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies: A call for dialogue between research and practice. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Van Someren, M. W., Barnard, Y. F. & Sandberg, J. A. C. (1994). The think-aloud method: A practical guide to modeling cognitive processes. London: Academic Press.Google Scholar
Vik-Tuovinen, G. V. (2002). Retrospection as a method of studying the process of simultaneous interpreting. In G. Garzone & M. Viezzi (Eds.), Interpreting in the 21st century: Challenges and opportunities. Amsterdam: John Benjamins, 63–72. DOI logoGoogle Scholar
Wallace, M. (2013). Rethinking bifurcated testing models in the court interpreter certification process. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 67–83.Google Scholar
Wang, J., Napier, J., Goswell, D. & Carmichael, A. (2015). The design and application of rubrics to assess signed language interpreting performance. The Interpreter and Translator Trainer 9 (1), 83–103. DOI logoGoogle Scholar
Wigglesworth, G. (2005). Current approaches to researching second language learner processes. Annual Review of Applied Linguistics 251, 90–111. DOI logoGoogle Scholar
Wu, F. S. (2013). How do we assess students in the interpreting examinations? In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt: Peter Lang, 15–33.Google Scholar
Zwischenberger, C. (2010). Quality criteria in simultaneous interpreting: An international vs. a national view. The Interpreters’ Newsletter 151, 127–142.Google Scholar
Cited by (10)

Cited by ten other publications

Han, Chao, Binghan Zheng, Mingqing Xie & Shirong Chen
2024. Raters’ scoring process in assessment of interpreting: an empirical study based on eye tracking and retrospective verbalisation. The Interpreter and Translator Trainer 18:3 ► pp. 400 ff. DOI logo
Lu, Rong, Muhammad Alif Redzuan Abdullah & Lay Hoon Ang
2023. Into-A or Into-B, That is a Question: A Systematic Literature Review of Directionality and Performance in Consecutive Interpreting. SAGE Open 13:4 DOI logo
Lu, Xiaolei & Chao Han
2023. Automatic assessment of spoken-language interpreting based on machine-translation evaluation metrics. Interpreting. International Journal of Research and Practice in Interpreting 25:1 ► pp. 109 ff. DOI logo
Moratto, Riccardo & Zhimiao Yang
2023. Probing the cognitive load of consecutive interpreters. Translation and Interpreting Studies DOI logo
Chen, Jing, Huabo Yang & Chao Han
2022. Holistic versus analytic scoring of spoken-language interpreting: a multi-perspectival comparative analysis. The Interpreter and Translator Trainer 16:4 ► pp. 558 ff. DOI logo
Yang, Yuan & Xiangdong Li
2022. Which theories are taught to students and how they are taught: A content analysis of interpreting textbooks . Círculo de Lingüística Aplicada a la Comunicación 92 ► pp. 167 ff. DOI logo
Han, Chao, Rui Xiao & Wei Su
2021. Assessing the fidelity of consecutive interpreting. Interpreting. International Journal of Research and Practice in Interpreting 23:2 ► pp. 245 ff. DOI logo
Wang, Weiwei
2021. Introducing China’s Standards of English Language Ability (CSE)—Interpreting Competence Scales. In Testing and Assessment of Interpreting [New Frontiers in Translation Studies, ], ► pp. 15 ff. DOI logo
Abdel Latif, Muhammad M. M.
2020. Translation and Interpreting Assessment Research. In Translator and Interpreter Education Research [New Frontiers in Translation Studies, ], ► pp. 61 ff. DOI logo
Wang, Weiwei, Yi Xu, Binhua Wang & Lei Mu
2020. Developing Interpreting Competence Scales in China. Frontiers in Psychology 11 DOI logo

This list is based on CrossRef data as of 12 september 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.