Written Discourse Completion Tasks in the Foreign Language Examination (YDS)

As the most prestigious and popular standardized achievement test to certify examinees’ proficiency of the English language at the national level, Foreign Language Examination (YDS) has been mostly taken by academic staff, undergraduate and graduate students, state employees, and military personnel for years in Turkey. The current study set out to address multiple-choice written discourse completion tasks (WDCTs) in YDS from 2015 to 2019 in the light of pragmatic knowledge, seemingly presumed to be the elephant in the room by test-developers in written language placement tests and its components, such as sociolinguistics, sociocultural, psychological, contextual, and rhetorical meanings to disclose whether they can represent functional language use in high context communications. Findings reveal that a huge gap subsists between real-life situations and given tasks in the test. At the end of the fine-grained inspection, implications and future research suggestions are stated to ensure validity and involve speaking skill more directly in YDS by emphasizing what to touch and expressly centre on the speech rather than solely assessing examinees’ knowledge for recognition and appealing to multiple-choice items with the analysis of international, large-scale language tests, such as TOEFL and IELTS.

Yabancı Dil Sınavındaki (YDS) Yazılı Söylem Tamamlama Araçlarının Eleştirel Analizi: Uluslararası Testlerin Gelişime Katkıları

Askeri personel, devlet memurları, lisans ve lisansüstü öğrenciler ile akademik personelin çoğunlukla katılım gösterdiği Yabancı Dil Sınavı (YDS), yerel bağlamda adayların İngiliz dili hakimiyetini belgelemek amacıyla Türkiye’de yıllardır uygulanan en popüler ve en prestijli standardize edilmiş başarı testidir. Bu çalışma, 2015-2019 yılları arasında YDS’de adaylara yöneltilen çoktan seçmeli, yazılı söylem tamamlama sorularının genellikle yazılı dil seviye belirleme sınavlarında göz ardı edilen edimsel bilgi ve bileşenleri olan sosyal dilbilim, sosyokültürel, psikolojik, bağlamsal ve söz bilimsel anlamlar bakımından yüksek bağlamlı, etkileşimli durumların ne kadar işlevsel dil kullanımını yansıttığını açığa kavuşturmak amacıyla tasarlanmıştır. Bulgular gerçek dil kullanımı ile testte verilen sorular arasında belirgin bir farkı işaret etmektedir. Çalışmanın sonunda, yapı geçerliliğini sağlamak, sınavda gerçeğe daha yakın konuşmaları dahil ederken adayın sadece bilgiyi tanımasını ölçmek ve çoktan seçmeli sorulardan sonuca varmasını sağlamak yerine hangi noktalara değinilmesi gerektiğini uluslararası geçerliği olan sınavların detaylı incelemesi sonucunda listeleyen önerilere, gelecek araştırmalara yön vermesi için yer verilmiştir.

PDF

___

Akın, G. (2016). Evaluation of national foreign language test in Turkey. Asian Journal of Educational Research, 4(3), 11–21.

Akpınar, K. D., & Çakıldere, B. (2013). Washback effects of high-stakes language tests of Turkey (KPDS and UDS) on productive and receptive skills of academic personnel. Journal of Language and Linguistic Studies, 9(2), 81–94.

Austin, J. L. (1962). How to do things with words. Oxford, UK: Clarendon Press.

Bachman, L. (2002). Alternative interpretations of alternative assessments: some validity issues in educational performance assessments. Educational Measurement: Issues and Practice, 21(3), 5-19.

Bachman, L. F. & Palmer, A. S. (1996). Language testing in practice. Oxford, UK: Oxford University Press.

Bateson, G. (1935). Culture contact and schismogenesis. Man, 35, 178–183. https://doi.org/10.2307/2789408

Beebe, L., & Cummings, M. (1996). Natural speech act data versus written questionnaire data: How data collection method affects speech act performance. In S. M. Gass & J. Neu (Eds.), Speech acts across cultures: Challenges to communication in a second language (pp. 65-86). Berlin: Walter de Gruyter.

Beebe, L., & Waring, H. Z. (2002). The pragmatics in the interlanguage pragmatics research agenda: The case of tone. Paper presented at the American Association for Applied Linguistics Conference, Salt Lake City, Utah.

Brown, J. D. (2001). Six types of pragmatics tests in two different contexts. In K. R. Rose & G. Kasper (Eds.), Pragmatics in language teaching (pp. 301-325). Cambridge, UK: Cambridge University Press.

Brown, J. D., Hudson, T., Norris, J., & Bonk, W. (2002). An investigation of second language task-based performance assessments. Honolulu, HI: Second Language Teaching & Curriculum Center, University of Hawaii

Brown, P., & Levinson, S. (1987). Politeness: Some universals in language usage. Cambridge, UK: Cambridge University Press.

Canale, M. (1983). On some dimensions of language proficiency. In J. Oller (Ed.), Issues in language testing (pp. 333-42). Rowley, MA: Newbury House.

Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, I, 1-47.

Carroll, J. B. (1968). The psychology of language testing. In A. Davies (Ed.), Language testing symposium: A psycholinguistic approach (pp. 46-69). London: Oxford University Press.

Celce-Murcia, M., Dörnyei, Z., & Thurrell, S. (1995). Communicative competence: A pedagogically motivated model with content specifications. Issues in Applied Linguistics, 6(2), 5-35.

Clennell, C. (1999). Promoting pragmatics awareness and spoken discourse skills with EAP classes. ELT Journal, 53(2).

Chapelle, C. (1998). Construct definition and validity inquiry in SLA research. In, L. Bachman & A. Cohen (Eds.), Interfaces between second language acquisition and language testing research (pp. 32-70). Cambridge, UK: Cambridge University Press.

Chomsky, N. (1957). Syntactic structures. The Hague: Mouton de Gruyter.

Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge, MA: MIT Press.

Clark, J. (1978). Direct testing of speaking proficiency: Theory and application. Princeton, NJ: Educational Testing Service.

Cohen, A. D. (2019). Considerations in assessing pragmatic appropriateness in spoken language. Language Teaching, 1–20. https://doi.org/10.1017/S0261444819000156.

Çetin, K., & Kılıçkaya, F. (2020). An inquiry into Turkish academicians’ willingness to communicate in English. Journal of Narrative and Language Studies, 7(13), 289-308.

Demir, D., & Genç, A. (2016). The analysis of foreign language proficiency exam for the state employees based on the CEFR. Hacettepe University Journal of Education, 31(1), 53–65. https://doi.org/10.16986/HUJE.2015014090.

Dewaele, J. M., & Dewaele, L. (2020). Are foreign language learners’ enjoyment and anxiety specific to the teacher? An investigation into the dynamics of learners’ classroom emotions. Studies in Second Language Learning and Teaching, 10(1), 45-65. http://dx.doi.org/10.14746/ssllt.2020.10.1.3

Dinçer, B. H. (2019). The validity of translation items in English language proficiency exams in Turkey. (Unpublished master’s thesis). Hacettepe University, Ankara: Turkey.

Douglas, D. (2005). Testing languages for specific purposes. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (pp. 857-868). Mahwah, NJ: Lawrence Erlbaum.

ETS (2019a). TOEFL iBT test integrated speaking rubrics (Scoring standards). Princeton, NJ: Educational Testing Service.

ETS (2019b). TOEFL iBT test independent speaking rubrics (Scoring standards). Princeton, NJ: Educational Testing Service.

Farhady, H. (1983). New directions for ESL proficiency testing. In J. Oller (Ed.), Issues in language testing (pp. 253-69). Rowley, MA: Newbury House.

Fulcher, G. (2003). Testing second language speaking. Harlow: Pearson Longman.

Golato, A. (2003). Studying compliment responses: A comparison of DCTs and naturally occurring talk. Applied Linguistics, 24, 90-121.

Grabowski, K. (2009). Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking. (Unpublished doctoral dissertation). Teachers College, Columbia University: New York.

Grice, P. (1975). Logic and conversation. In P. Cole & J. Morgan (Eds.), Syntax and semantics (pp. 41-58). New York: Academic Press.

Gumperz, J. (1982). Discourse strategies. Cambridge, UK: Cambridge University Press.

Gür, Ö. (2012). Testing, assessment and KPDS (foreign language proficiency examination for state employees)–What does this exam assess? Sakarya University Journal of Education, 2(1), 23–32.

Halliday, M. A. K. (1994). An introduction to functional grammar (2nd ed.). London: Edward Arnold.

Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.

Halliday, M. A. K., & Hasan, R. (1989). Language, context, and text: Aspects of language in a socio-semiotic perspective. Oxford, UK: Oxford University Press.

Hudson, T., Detmer, E. & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics. Honolulu, HI: Second Language Teaching and Curriculum Center, University of Hawaii.

Hudson, T., Detmer, E. & Brown, J. D. (1995). Developing prototypic measures of crosscultural pragmatics. Technical Report 7. Honolulu, HI: Second Language Teaching and Curriculum Center, University of Hawaii.

Hutchby, I. & Wooffitt, R. (1998). Conversation analysis: Principles, practice and applications. Cambridge: Polity Press.

Hymes, D. (1967). Models of the interaction of language and social setting. Journal of Social Issues, 23(2), 8-38.

Hymes, D. (1974). Ways of speaking. In R. Bauman & J. Sherzer (Eds.), Explorations in the ethnography of speaking (pp. 433-451). Cambridge, UK: Cambridge University Press.

Jaszczolt, M. (2002). Semantics and pragmatics: Meaning in language and discourse. London: Longman.

Johnson, M. (2001). The art of non-conversation. New Haven, CT: Yale University Press.

Jones, R. (1985). Second language performance testing: An overview. In P. Hauptman, R. LeBlanc, & M. Wesche (Eds.), Second language performance testing (pp. 15-24). Ottawa: University of Ottawa Press.

Kamaşak, R., Sahan, K., & Rose, H. (2021). Academic language-related challenges at an English-medium university. Journal of English for Academic Purposes, 49, 100945. https://doi.org/10.1016/j.jeap.2020.100945

Kasper, G, & Rose, K. (2002). Pragmatic development in a second language. Maiden, MA: Blackwell.

Kıray, G. (2015). Macro-structure analysis of reading comprehension paragraphs of KPDS and YDS exams within years 2003-2013. Hasan Ali Yucel Journal of Education, 12(1), 219–233.

Korsko, P. (2004). The narrative shape of two-party complaints in Portuguese: A discourse analytic study. Unpublished doctoral dissertation, Teachers College, Columbia University, New York, NY.

Kutlu, M. O., Demirogları, G., & Demirogları, B. (2020). Washback effect of higher education institution exam-foreign language test on university candidates’ attitudes. African Educational Research Journal, 8(2), S121-S130.

Külekçi, E. (2016). A concise analysis of the Foreign Language Examination (YDS) in Turkey and its possible washback effects. International Online Journal of Education and Teaching (IOJET), 3(4). 303-315. http://iojet.org/index.php/IOJET/article/view/141/143

Lado, R. (1961). Language testing. New York: McGraw-Hill.

Lazarton, A. (1992). The structural organization of a language interview: A conversational analysis perspective. System, 20, 373-86.

Leech, G. (1983). Principles of pragmatics. London: Longman.

Liu, J. (2006). Measuring interlanguage pragmatic knowledge of EFL learners. Frankfurt am Main: Peter Lang.

Locher, M. A., & Sage L. G. (2010). Introduction to interpersonal pragmatics. In M. A. Locher & L. Sage (Eds.), Interpersonal pragmatics (pp. 1-13). Berlin, New York: De Gruyter Mouton. doi:10.1515/9783110214338.0.1

McCarthy, M. (1991). Discourse analysis for language teachers. Cambridge: Cambridge University Press

McNamara, T., & Roever, C. (2006). Language testing: The social dimension. Maiden, MA: Blackwell.

Ochs, E. (1996). Linguistic resources for socializing humanity. In J. Gumperz & S. Levinson (Eds.), Rethinking linguistic relativity (pp. 407-437). Cambridge, UK: Cambridge University Press.

Oller, J. (1979). Language tests at schools. London: Longman.

OSYM -Öğrenci Seçme ve Yerleştirme Merkezi (2016). YDS kılavuzu. Retrieved on Monday, March 02, 2020 from http://dokuman.osym.gov.tr/pdfdokuman/2016/GENEL/EsdegerlikYonergesi26022016.pdf.

OSYM (2015). 2015 Yabancı dil bilgisi seviye tespit sınavı (YDS). Ankara: OSYM.

OSYM (2016). 2016 Yabancı dil bilgisi seviye tespit sınavı (YDS). Ankara: OSYM.

OSYM (2017). 2017 Yabancı dil bilgisi seviye tespit sınavı (YDS). Ankara: OSYM.

OSYM (2018). 2018 Yabancı dil bilgisi seviye tespit sınavı (YDS). Ankara: OSYM.

Özmen, K. S. (2011). Washback effects of the inter-university foreign language examination on foreign language competences of candidate academics. Novitas-ROYAL, 5(2), 215-228.

Özer, Z., & Altay, İ.F. (2021). Examining the level of enjoyment and anxiety among Turkish EFL students. Journal of Language and Linguistic Studies, 17(Special Issue), 663-671.

Polat, M. (2019). Washback effect of Turkish foreign language proficiency exam YDS: A scale development study. Journal of Language and Linguistic Studies, 15(4), 1252-1268.

Polat, M. (2020). Washback effects of YDS and YOKDIL on foreign language teaching. Kastamonu Education Journal, 28(5), 2188-2202. doi: 10.24106/kefdergi.709027

Purpura, J. (2004). Assessing grammar. Cambridge, UK: Cambridge University Press.

Purpura, J. (2014). Cognition and language assessment. In A. J. Kunnan (Ed.), The companion to language assessment (pp. 1452-1476). Oxford, UK: Wiley-Blackwell.

Purpura, J. (2016). Second and foreign language assessment. The Modern Language Journal, 100, 190-208.

Purpura, J. (2017). Assessing meaning. In E. Shohamy, L. G. Or, & S. May (Eds.), Language testing and assessment, encyclopedia of language and education (3rd ed.) (pp. 33-61). Springer International Publishing AG.

Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23, 229-256.

Roever, C. (2018). Testing of L2 pragmatics and possibilities for measuring implicit knowledge (Unpublished manuscript). Department of Linguistics & Applied Linguistics, The University of Melbourne, Australia.

Sarıtepeci, M., Duran, A., & Ermiş, U. F. (2019). A new trend in preparing for foreign language exam (YDS) in Turkey: Case of WhatsApp in mobile learning. Education and Information Technologies, 24(5), 2677-2699. https://doi.org/10.1007/s10639-019-09893-4

Suesattayawong, O. (2006). Exploring trouble in speaking English in Rajvithi hospital. (Unpublished doctoral dissertation). Srinakharinwirot University, Bangkok: Thailand.

Schegloff, E. (1987). Between micro and macro: Contexts and other connections. In J. Alexander, B, Giesen, R. Munch, & N. Smelser (Eds.), The micro-macro link (pp. 207-234). Berkeley, CA: University of California Press.

Searle, J. (1969). The philosophy of language. New York: Oxford University Press.

Spolsky, B. (1968). Language testing—the problem of validation. TESOL Quarterly, (2)2, 88-94.

Tadayon, F., & Ravand, H. (2016). Using grounded theory to validate Bachman and Palmer’s (1996) strategic competence in EFL graph-writing. Lang Test Asia 6, 8. https://doi.org/10.1186/s40468-016-0031-y

Taguchi, N. (2018). Data collection and analysis in developmental L2 pragmatics research: Discourse completion test, role-play, and naturalistic recording. In A. Gudmestad & A. Edmonds (Eds.), Critical reflections on data in second language acquisition (pp. 7–32). Amsterdam: John Benjamins.

Thomas, J. (1983). Cross-cultural pragmatic failure. Applied Linguistics, 4, 91-112.

Thomas, J. (1995). Meaning in interaction: An introduction to pragmatics. London: Longman.

Timpe-Laughlin, V., Green, A., & Oh, S. (2021). Raising pragmatic awareness: A think aloud study. System, 98, 102470. https://doi.org/10.1016/j.system.2021.102470

Yamashita, S. (1996). Six measures of JSL pragmatics. Technical Report 14. Honolulu, HI: Second Language Teaching and Curriculum Center, University of Hawaii.

Yamashita, S. (2001). Using pictures for research in pragmatics. Eliciting pragmatic strategies by picture response tasks. In T. Hudson & J. D. Brown (Eds.), A focus on language test development (pp. 35-56). Honolulu, HI: University of Hawaii, National Foreign Language Resource Center.

Youn, S. J. & Bi, N. Z. (2019). Investigating test-takers’ strategy use in task-based L2 pragmatic speaking assessment. Intercultural Pragmatics, 16, 185–218.

Yule, G. (1996). Pragmatics. Oxford, UK: Oxford University Press.

Zeybek, G. (2015). Foreign Language Examination (FLE) test takers’ foreign language learning attitude, motivation and test anxiety in Turkey. (Unpublished master’s thesis). Süleyman Demirel University, Isparta, Turkey.