Measuring Heritage Language Learners’ Proficiency for Research Purposes: An Argument-based Validity Study of the Korean C-test

Abstract

Heritage language learners (HLLs) have increasingly become a focus of interest in applied linguistics research (Kagan & Dillon, 2012), but the lack of consistent conceptualization of HL proficiency has hindered the systematic accumulation of research knowledge about HLLs (Son, 2017). Shortcut proficiency measures may be a way to address these shortcomings (Norris, 2018). C-tests have been found to be particularly promising in providing a quick measurement of language learners’ global proficiency (Eckes & Grotjahn, 2006), yet validation studies on this shortcut measure have focused on Foreign Language Learners (FLLs).

To address these critical gaps, this study developed a validity argument (Kane 2006, 2011, 2013) to evaluate the use of an innovative Korean C-test (Son et al., 2018) to assess Korean HLLs and FLLs for applied linguistics research purposes. Ninety-three Korean language learners, 41 HLLs and 52 FLLs, were assessed using five instruments: the Korean C-test, an Elicited Imitation Test (EIT, Kim et al., 2016), ACTFL Oral Proficiency Interview-computer (OPIc), ACTFL Writing Proficiency test (WPT), and a self-assessment questionnaire. The resulting data were then examined for five inferences—theoretical grounds, evaluation, generalization, explanation, and extrapolation—in terms of warrants, assumptions, backing, and rebuttals.

C-test items accurately and reliably distributed Korean learners into a wide range of proficiency levels (IRT person separation index=5.50 and Cronbach’s α=.94). Although C-test performance was closely related to the literacy-based WPT proficiency indicator (ρ=.87), it also correlated strongly with the oracy-based OPIc and EIT proficiency measures (ρ=.81 or higher). Multiple regression analyses showed that both speaking and writing proficiency could explain 80% of the C-test score variance with 𝑅²=.80, 𝐹(2,90)=181.83, 𝑝=.000. Nevertheless, writing was more important than speaking proficiency in predicting C-test performance for HLLs, whereas both language skill predictors were equally important for FLLs. Furthermore, a hierarchical cluster analysis provided a bottom-up categorization of learners into HLLs and FLLs and questioned the expectation that HLLs are always linguistically different to FLLs. Overall, the evidence supported the use of the Korean C-test to assess both HLLs and FLLs across a range of proficiency levels for applied linguistic research purposes. The study concludes with suggestions for future research.