385B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... Berlinda Mandasari UDK [808:37.091.26]:811.111'243(594) Faculty of Letters, Universitas Negeri Malang DOI: 10.4312/vestnik.16.385-402 Faculty of Arts and Education, Universitas Izvirni znanstveni članek Teknokrat Indonesia, Indonesia berlinda@teknokrat.ac.id Maulida Rahmah Faculty of Letters, Universitas Negeri Malang Faculty of Humanity, Universitas Sari Mulia Indonesia maulidarahmah.m.pd@gmail.com Nur Mukminatien Faculty of Letters, Universitas Negeri Malang Indonesia nur.mukminatien.fs@um.ac.id DEVELOPING AN ASSESSMENT INSTRUMENT MODEL FOR INDONESIAN CULTURE-BASED ORAL PRESENTATIONS FOR UNDERGRADUATE EFL STUDENTS 1 INTRODUCTION The main goal of 21st-century language learning is communicative competence, particu- larly particularly oral skills, which are crucial for academic and career success (Fareen, 2018; López et al., 2019: 207; Werdiningsih & Mukminatien, 2023: 422). Effective oral communication involves the smooth, clear expression of ideas and grammatical accu- racy (AlSaleem, 2018; Mamo & Yigzaw, 2015). Developing students’ oral competence prepares them for professional tasks like cooperation, negotiation, and problem-solving. Integrating effective teaching methods and supportive environments is essential for en- hancing students’ oral communication skills (Cavanagh et al., 2019: 441). Learning a language is closely tied to understanding its culture, making culture-based teaching essential. A language reflects the cultural values of its speakers, and in today’s globalized world teaching English requires not only linguistic competence but also intercul- tural communicative competence (Cheng, 2012: 164; Wahyuni et al., 2014: 84). This is in line with the findings of Darong and Menggo (2021: 253), who revealed that when someone 386 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES learns a language, they inherently acquire some knowledge about the people who speak that language. To achieve this, learners of the target language need to produce language with both linguistic and cultural competency. Marhamah et al. (2017: 492) suggest that incorporating target culture into teaching is seen positively, as it enhances students’ cultural awareness and understanding of nonverbal cues, which are crucial for effective communi- cation. While the target culture is important, using a more familiar local culture can boost student engagement and reduce awkwardness (Alakrash et al., 2021: 549). Therefore, local culture-based materials should be considered when teaching speaking. Darong and Menggo (2021: 256) highlight that incorporating culture into English teaching can enhance students’ appreciation of their culture. In Indonesia, Seftika et al. (2017: 19) found that teaching English through a local cultural context improves students’ understanding of Indonesian culture and increases their awareness of cultural diversity, aiding in effective social communication. Speaking practice in English also helps stu- dents transfer their knowledge of local culture. In addition, Rahman et al. (2022: 67) found that integrating local culture into speaking lessons significantly boosts students’ motivation and emotions. It reduces anxiety, creates positive emotional experiences, and fosters a more enjoyable learning environment and speaking sessions with local cultural content. Overall, familiarity with the local context is beneficial for English learners. As teaching English speaking skills using local culture has been proven beneficial, the question arises as to what constitutes effective ways of assessing the student’s oral communication performance in this context. In the language education, assessment is considered a crucial process to identify whether students have achieved the specified learning aims, emphasizing communication strategies, encouraging communication, managing unpredictable language output, and being appropriate to the focal situation (Richards, 2006). Assessment can help achieve the intended educational purposes, pro- viding essential feedback on students’ understanding and progress (Fitriyah et al., 2022: 126; Slamet & Mukminatien, 2024: 189). However, each learning objective needs to be assessed in order to ascertain the success of the learning process (Khotimah et al., 2024: 96; Masitoh et al., 2023: 926), as otherwise the intended learning aims may not be achieved. While assessment helps to map the students’ learning progress, some of the speak- ing instruments that are currently available are not well-suited for assessing students’ culture-based oral performance. Some studies suggest assessing speaking by focusing on pronunciation, fluency, organization and structure, language use, and gestures (Werdin- ingsih & Mukminatien, 2023: 425), thus neglecting the value of the various aspects of in- tercultural competence embedded in the culture-based content knowledge as assessment indicators. In addition, the learning activities used in this context must be able to develop students’ creativity and critical thinking, as these are important skills for their future careers. The absence of speaking instruments that cover these skills may be perceived negatively by students, as it suggests that being creative and critical thinkers has nothing 387B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... to do with their learning process. Therefore, it is crucial to develop more comprehensive assessment instruments that address these gaps. The present study uncovered the problem faced by Indonesian private university lec- turers in teaching English for tourism conversation courses to carry out classroom-based assessments. Due to a limited understanding of this issue, the procedures used to carry out classroom-based assessments of Indonesian culture-based oral presentations do not meet the criteria of effective classroom-based assessment, and several changes are need- ed. First, to reduce ambiguities, it is essential to validate speaking prompts with expert input, ensuring they are perceived as high-quality, readable, and relevant to the learning objectives. Second, relying solely on linguistic competence is inadequate for Indonesian culture-based oral presentations, and it is important to incorporate the dimension of inter- cultural communicative competence and acknowledge students’ efforts in critical think- ing and creativity (Santoso et al., 2021: 4). To address this issue, this study envisions the development of Indonesian culture-based oral presentation instruments, as guided by two main research questions: 1. What is an effective speaking prompt? 2. How are the speaking assessment instrument indicators best developed and validated? 2 METHOD 2.1 Research Design This research was done for development purposes, adopting a research and development (R&D) design. The final products of this study are speaking prompt and assessment in- dicators on Indonesian culture-based oral presentation. Implementing the ADDIE mod- el adopted from Morrison (2010) throughout a descriptive study allows researchers to systematically document each aspect of the instructional design, while learning about best practices and areas for improvement. This systematic method not only improves comprehension of useful educational practices but also encourages continual develop- ment in terms of instructional design efforts by documenting comprehensive descriptions of procedures and outcomes at each stage. The stages of development include analysis, design, development, implementation, and evaluation. a) Analysis First, researchers examined the curriculum, focusing on the syllabus of English for tourism conversation courses at one private university in Indonesia to identify suita- ble speaking activities and assessment models aligned with learning objectives. Sec- ond, the researchers conducted library research to create a standardized Indonesian culture-based speaking assessment model, considering the needs of both teachers and students. 388 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES b) Design The design step is done by designing speaking outlines containing speaking prompt and assessment indicators. First, the researchers formulate the speaking prompt. The specifications of the prompt are as follows. Table 1: Specification of an Effective Speaking Prompt No Specification Description 1 Instruction Students describe the product of cultures such as customs/traditions, cuisine, and social norms, as well as historical facts, individually. 2 Participant Second-semester who enrolled in the English for Tourism Conversation course. 3 Material Each student chooses one of the Indonesian cultures among the listed ones – Javanese, Lampungnese, Bataknese, Balinese, or any other local culture they are interested in. Supporting aids are highly recommended. 4 Setting place and time The presentation takes place at the scheduled time and place, with each presentation having a maximum length of 10 minutes. 5 Assessment indicators Grammatical accuracy, pronunciation, fluency, vocabulary, cultural knowledge, attitude, skill of interpreting and relating information from different cultural contexts, cultural awareness, time management, and appropriate use of audio-visual aids. Table 1 shows that the specifications of an effective speaking prompt comprise five main aspects that aim to produce clear and precise prompts. In terms of instruction, the students need to get clear instructions on what they have to do. Additionally, the prompt should address particular participants to avoid misinterpretation of those who have no interest in the related speaking activities. The specifications also note that students have to insert materials based on Indonesian culture in their presentation, as supported by rel- evant aids either audio or visuals. The prompt should also inform students about the time and place where the presentation will take place, and how long they have to present their materials. Lastly, the students should be informed of the assessment indicators used, so that they are aware of the targeted learning outcomes. In terms of designing speaking assessment indicators, the researchers conducted li- brary research by reviewing relevant previous studies and theories of assessment. After synthesizing these resources (Aprianoto & Haerazi, 2019: 2797; Cheng, 2012: 164; Ne- mati et al., 2014: 8; Wahyuni et al., 2014: 84; Wehmeier, 2020: 535) and taking only relevant data, the researchers divided culture-based speaking assessment indicators into three main aspects: linguistic competence (grammatical accuracy, pronunciation, fluency, and vocabulary), intercultural communicative competence (cultural knowledge, attitude, skill of interpreting and relating information from different cultural contexts, and cultural awareness), and self-management competence (time management and appropriate use of audio-visual aids). 389B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... c) Development The development step involves expert judgments to validate the assessment model. Two prominent lecturers, with doctorate degrees and expert in teaching speaking skills, teaching English as a foreign language, and language assessment, were in- vited to validate the instrument content. They evaluated the quality of the speaking prompt, instruction readability, and the relevancy of the speaking assessment indi- cators for assessing students’ linguistic, intercultural, and self-management com- petence in relation to the learning objectives. The experts also tested the assess- ment indicators on volunteer students. Their feedback was then used to refine the assessment. d) Implementation The implementation step involves testing the speaking prompt and the assessment indicators for students. Four second-semester English education student volunteers read the instructions to ensure clarity. With a similar understanding to the experts, they were given one week to prepare their oral presentations. After one week, the four students made their presentations on Indonesian culture and were assessed by the two experts using the developed indicators. The scoring results from the experts were then analysed for validity and reliability. e) Evaluation The evaluation step followed the completion of the other four steps. Here, the re- searchers analysed the instrument’s strengths and weaknesses based on the valid- ity and reliability results. Once deemed appropriate, they created an assessment model with a valid speaking prompt for culture-based presentations and adaptable culture-based speaking assessment indicators for broader culture-based language learning. 2.2 Research Instrument The products of this study include a speaking prompt and culture-based speaking as- sessment indicators. To ensure the validity of the speaking prompt, researchers distrib- uted close-ended questionnaires to measure its quality, readability, and relevancy to the learning objectives. The experts could add support for their responses to the question- naire. The second product is the culture-based speaking assessment indicators. As the main goal is to help students achieve their communicative competence, the research- ers developed indicators that cover linguistic competence and speaking skills in terms of fluency, grammatical accuracy, pronunciation, and vocabulary (Cheng, 2012: 164; Wahyuni et al., 2014: 84); intercultural communicative competence in which students present knowledge of cultures, attitudes, and cultural awareness, as well as the skills needed to interpret and relate information from many cultural backgrounds (Aprianoto & Haerazi, 2019: 2797; Nemati et al., 2014: 8); and self-management competence in 390 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES which students can control the length of the presentation and choose appropriate au- dio-visual aids to support it (Wehmeier et al., 2020: 535). Once the students can meet the standards set for the various indicators of assessment when conducting an oral presentation, their communicative competence is achieved. 2.3 Data Analysis Technique The researchers used content validity to analyse the questionnaire data on the speak- ing prompt and assessment indicators, obtaining expert feedback through a close-end- ed questionnaire. Additionally, the researchers used statistical analysis – calculating Cronbach’s Alpha using SPSS – to assess the reliability of the speaking assessment indicators across dimensions and as a whole. Testing reliability ensures consistent and impartial measurement over time and across different parts of the instrument, as proposed by Shweta and Chaturvedi (2015: 21). Furthermore, they recommended using the interpretation criteria of inter-rater reliability proposed by Landis and Koch (1977), cited in Shweta and Chaturvedi (2015: 22), and as presented in the following table. Table 2: Interpretation Criteria of Inter-Rater Reliability Coefficient Coefficient Interpretation <0.00 Poor 0.00 - 0.20 Slight 0.20 - 0.40 Fair 0.41 - 0.60 Moderate 0.61 - 0.80 Substantial 0.81 - 1.00 Almost Perfect According to Gelişli and Beisenbayeva (2017: 282), an instrument is considered highly reliable and acceptable if the coefficient of consistency is more than 0.7. The researchers evaluated the reliability of the indicators right after the instrument was used by the two raters to assess the oral presentation of the four second-semester student vol- unteers in the English education program, using a five-point scoring scale ranging from 1 to 5, with 1 meaning “poor” and 5 “excellent”. 391B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... 3 FINDINGS 3.1 Content Validity of Speaking Prompt As part of their attempt to develop an assessment of Indonesian culture-based oral pres- entation, the researchers developed the specifications of an effective speaking prompt. The quality, readability, and relevancy of the prompt were examined by two experts on teaching speaking and teaching English as a foreign language (E1), and language assess- ment (E2). The results of their assessment are shown in the following table. Table 3: Validation Results for the Speaking Prompt Aspects Guided Questions E1 E2 Quality Does the speaking prompt have good clarity and quality? Yes Yes Readability Does the speaking prompt address the specific participant? Yes Yes Does the speaking prompt state the time and place for conducting the oral presentation? Yes Yes Relevancy Do the materials stated in the speaking prompt represent to the related cultural aspects? Yes Yes Do the assessment indicators stated in the speaking prompt cover all aspects of the students’ communicative competence in culture- based oral presentations? Yes Yes Referring to the findings in Table 3, the experts unanimously agreed that the speaking prompt meets the criteria of quality, readability, and relevancy. Both ex- perts found the speaking prompt to be clear and of high quality. They also agreed that the instructions were readable and easily understood by people who share a similar perception of what is required during the oral presentation. There were no ambigu- ities in the language used. Furthermore, in terms of relevancy, both experts agreed that the speaking prompt effectively integrated cultural aspects, guiding the students to present selected aspects of Indonesian culture, including customs/traditions, cui- sine, and social norms. In addition, they affirmed that the assessment indicators are relevant for evaluating students’ communicative competence in culture-based oral presentations. 3.2 Content Validity of Culture-Based Speaking Assessment Indicators The second assessment instrument used culture-based speaking assessment indicators. Similar to validating the speaking prompt, the researchers brought in experts to validate the content of the indicators, with the results presented below. 392 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES Table 4: Validation Results of Culture-Based Speaking Assessment Indicators Aspect Guided Question E1 E2 Linguistic Competence 1. Is assessing students’ grammatical accuracy, fluency, vocabulary, and pronunciation relevant to assessing students’ oral presentation? Yes Yes 2. Should content be included in linguistic competence? No No Intercultural Communicative Competence 3. Is assessing students’ knowledge of cultures, attitudes, skills, and cultural awareness relevant to assessing students’ oral presentation? Yes Yes Self-Management Competence 4. Does assessing students’ time management in controlling their oral presentation and managing their content delivery with regard to a given length contribute to students’ critical thinking? Yes Yes 5. Does assessing aids in supporting students’ oral presentation contribute to critical thinking? Yes Yes As shown in Table 4, the experts agreed that the assessment indicators covering the three competencies are relevant for evaluating students’ culture-based oral pres- entations. However, one specific question was deemed irrelevant, leading to redundan- cy in the indicators, as the dimension of content should belong under intercultural com- municative competence rather than linguistic competence. This makes sense because the content of the presentation reflects cultural products. As such, discussing content relates to students’ knowledge of Indonesian culture, which they present. Consequent- ly, both experts recommended removing the content dimension from the linguistic competence aspect. 3.3 Reliability of Assessment Indicators of Indonesian Culture-Based Oral Presentation Measuring the reliability of the speaking assessment indicators was done through in- ter-rater reliability. Two experts scored the oral performance of four second-semester student volunteers as a pilot study using scoring rubrics developed from assessment indi- cators. The results of scoring the students’ oral performance were calculated and analysed using Cronbach’s alpha. The following table presents the inter-rater reliability of each dimension of competence. 393B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... Table 5: Coefficient of Inter-Rater Reliability in Each Dimension Aspect Dimension Coefficient Criteria Linguistic Competence Fluency 1.0 Almost perfect Grammatical accuracy .80 Substantial Vocabulary .74 Substantial Pronunciation 1.0 Almost perfect Intercultural Communicative Competence Knowledge .72 Substantial Skill .84 Almost perfect Attitude .93 Almost perfect Cultural awareness .72 Substantial Self-Management Competence Time management .93 Almost perfect Appropriate use of audio and visual aids 1.0 Almost perfect The table above shows the results for the inter-rater reliability coefficient, show- ing various coefficients of reliability ranging from substantial to almost perfect. The “substantial” level includes grammatical accuracy, vocabulary, knowledge, and cul- tural awareness, while the “almost perfect” level includes fluency, pronunciation, skill, attitude, time management, and appropriate use of audio and visual aids. The results indicate that each dimension of the assessment indicators is applicable to assess stu- dents’ oral presentation. In order to find the overall reliability of the assessment indicators, the researchers calculated the consistency using Cronbach’s alpha, with the result presented in the fol- lowing figure. Figure 1: The Coefficient of Reliability of All Indicators As shown in Figure 1, the reliability coefficient of all indicators is 0.99, which im- plies that all the all indicators for assessing students’ oral presentations are reliable and applicable. To some extent, the indicators can be employed to assess various kinds of speaking activities, particularly those that involve linguistic, intercultural, and self-man- agement competence. The following table shows the assessment indicators used for the students’ culture-based oral presentations. 394 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES Table 6: The Assessment Indicators of Indonesian Culture-Based Oral Presentation Competence Dimension Indicator Linguistic Competence Pronunciation Students are able to pronounce words correctly. Fluency Students are able to perform at a smooth and continuous pace, with virtually no repetition or hesitancy during a presentation. Grammatical accuracy Students are able to use a variety of structures with only occasional grammatical errors. Vocabulary Students possess an extensive vocabulary range, and make precise and accurate word choices that are highly appropriate to enhance effective communication. Intercultural Communicative Competence Knowledge of the culture Students are able to present the cultural products (customs, cuisine, social norms) of a selected culture and deliver the culture-based materials in a capable manner. Attitude Students are able to communicate politely, exhibit curiosity, perform open-mindedly, suspend scepticism and beliefs about both their own and other cultures, avoid making assumptions about which cultures are right or wrong, and respect cultural differences and similarities. Skill in interpreting and relating information Students are able to recognize cultural viewpoints in a new cultural setting, provide an explanation of the new cultural setting they encounter, recognize the potential for cultural misunderstandings, and resolve them. Cultural awareness Students are able to share cultural products and practices appropriately, as well as modify their values and actions while engaging with other cultures. Self-Management Competence Time management Students are able to manage their presentations so they meet the required length of time. Appropriate use of audio and visual aids Students are able to use aids that are very clear and visually/ audibly appealing and support the materials on specific cultures being presented. 4 DISCUSSION The outcome of this study was a culture-based speaking assessment instrument model. More specifically, two kinds of instruments were developed as the final products – the speak- ing prompt and culture-based speaking assessment indicators. The validity of the speaking prompt and assessment indicators was ensured by examining the content validity, which was done with the help of two experts. Almanasreh et al. (2019: 215) note that content validity can offer a proof of how well-suited and representational an instrument’s components are for the intended construct. A pair of experts evaluated each component of the instruments and assigned scores based on their representation and relevance to the content domain. Three as- pects of the speaking prompt – quality, readability, and relevancy – were analyzed, along with the indicators for linguistic, intercultural, and self-management competence. Two validation 395B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... stages were conducted: the development stage and the expert-judgment stage, ensuring the in- struments’ high quality. The results indicated that the speaking prompt was appropriate for the learning objectives and was deemed valid. Similarly, the competencies assessed – linguistic, intercultural, and self-management – were found to be relevant for evaluating the students’ speaking performance in oral presentations. This is in line with what has been suggested by previous studies (Almanasreh et al., 2019: 216; Koller et al., 2017: 2), in which a two-step procedure (development and judgment quantification) is recommended to assess each instru- ment’s content validity. This can subsequently assess the instrument’s representativeness and relevance to the subject matter. By carrying out content validation systematically and thor- oughly, useful information can be obtained for researchers and practitioners. Since both instruments were valid, the following step was undertaken to ensure their reliability. Four English education student volunteers were invited to read the in- structions and conduct the culture-based oral presentation. Their performances were assessed by two experts using scoring rubrics based on culture-based assessment in- dicators. The scoring results were calculated and analyzed, revealing a Cronbach’s alpha reliability coefficient of 0.99, indicating that they are reliable, applicable, and acceptable to measure the students’ oral presentations. Aprianoto and Haerazi (2019: 2803) state that an instrument is highly reliable if Cronbach’s alpha is higher than 0.7. A clear speaking prompt/instruction for students’ culture-based oral presentations contributes significantly to their success in achieving communicative competence. It in- forms students about the target audience, the task requirements, time allocation, location, necessary materials, and how the related assessment indicators can help them prepare effectively. Moreover, teachers may experience challenges if the prompt is vague, hin- dering their ability to enhance students’ skills. As such, clear instructions are recommend- ed to enhance students’ cognitive strategies when generating speech and improve their awareness of the learning process (Sabnani, 2024: 75). More emphasis should also be given to engaging students in meaning-making based on the instructions they have been given, to better prepare them for achieving effective communication. The speaking assessment indicators used in this study encompass three aspects of com- petence: linguistic, intercultural, and self-management. Regarding linguistic competence, an analysis of the literature and expert judgments identified four key dimensions: fluency, grammatical accuracy, pronunciation, and vocabulary. These dimensions are considered crucial for improving students’ content delivery during speeches. The rules of word creation and vocabulary, pronunciation, and sentence construction make up linguistic competence. Rahnama et al. (2016: 1) also added fluency to the students’ linguistics competence, which refers to the student’s ability to produce the spoken language without undue pausing or hes- itation. Competence in all four dimensions will enhance students’ awareness of the literal meaning of an utterance, as well as their knowledge and ability to utilize language correctly in communicative contexts within a particular speech community. 396 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES In terms of intercultural communicative competence, four dimensions were construct- ed: knowledge, attitude, skills, and cultural awareness. Student’s knowledge of cultures reflects their ability to describe the product of cultures such as customs/traditions, cuisine, and social norms as well as historical facts in a capable manner. In line with this, Byram and Morgan (1994) emphasize that factual knowledge about a society, including its customs, geography, and history, is essential for students to effectively communicate across cultural boundaries. Additionally, students’ attitudes toward cultures should demonstrate respect for both similarities and differences, avoiding judgmental actions regarding the correctness of various cultural practices. Intercultural communicative competence, on a broader scale, in- volves fostering a positive attitude towards language learning and appreciation for individ- uals from diverse cultures and nations (Candel-Mora, 2015: 28). More practically, students’ ability to orally present cultures involves identifying cultural perspectives in new cultural contexts, explaining these contexts, and recognizing and addressing potential cultural mis- understandings. Ilie (2019: 265) suggested that the essential skills needed to process any information that needs to be learned include observation, listening, analysis, evaluation, interpretation, and relation. Critical introspection is also crucial for developing intercultural communicative competence. Lastly, the dimension of cultural awareness demonstrates stu- dents’ ability to adapt their beliefs and behaviors appropriately in the exchange of culture and traditions. It also involves understanding how people from different cultures think and act and recognizing the differences in values, customs, and behaviors between one’s culture and the host culture (Ilie, 2019: 267-268). In terms of self-management competence, two dimensions are involved; time man- agement and appropriate audio-visual aids. As the speaking prompt requires students to deliver an oral presentation of a given length and bring aids to support their presentation, the student’s ability to manage their time and select appropriate aids is considered cru- cial. The deliberate planning and structuring of study schedules includes choosing the best times and durations for students to complete their assignments (Alario-Hoyos et al., 2017: 120). Moreover, critical and creative thinking requires mental processes through- out project implementation, including effective time management skills from students (Santoso et al., 2021: 4). Derakhshan et al. (2015: 523) also stated that the success of project-based learning is influenced by several factors, including time management, as students must be able to meet their learning objectives within the time series. Addition- ally appropriate audio and visual aids to support oral presentations reflects the students’ creativity and curiosity, which, in turn, enhances their performance. 5 CONCLUSION This study developed an assessment model for Indonesian culture-based oral presenta- tion by undergraduate EFL students taking English for tourism conversation courses. 397B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... Two instruments were developed; a speaking prompt and a set of culture-based speaking assessment indicators. The instruments were validated by experts using content validity, and the results showed that both instruments were valid and reliable. Constructive feed- back from the two experts was essential to minimizing irrelevancy among the indicators. To ensure the reliability of the assessment indicators, inter-rater reliability was measured using Cronbach’s alpha after the indicators had been used to assess four student volun- teers. The resulting coefficient of internal consistency was 0.99. In short, these assessment instrument indicators were reliable and applicable for assessing students’ culture-based speaking practice. The instruments were developed purposively to meet the need for classroom-based assessment instruments, particularly in culture-based speaking activities. On the one hand, the specifications of the speaking prompt instrument cannot be merely adopted by teachers or lecturers, as each institution may have a different policy in instructing students with particular speaking activities. On the other hand, the culture-based speaking instru- ment indicators can be adopted broadly by teachers and lecturers who deal with teaching culture-based speaking. Teachers and lecturers may then develop various scoring rubrics that are suitable for each indicator. The ADDIE model used in this study is considered an effective and adaptable procedure for such instruments. Therefore, it is recommended that the model be applied to develop a classroom-based assessment instrument for other areas of English language learning. 6 ACKNOWLEDGEMENT The authors would like to thank the Centre for Higher Education Funding (BPPT) and Indonesia Endowment Fund for Education Agency (LPDP) from the Ministry of Finance, Republic Indonesia, for granting the scholarship and providing funding for this rese- arch. The authors also would like to extend their gratitude to Universitas Negeri Malang, Universitas Teknokrat Indonesia, and Universitas Sari Mulia for their constant support. Finally, the authors wish to express their sincere gratitude to Dr. Afrianto and Intan Ha- mzah for their insightful feedback throughout the article review process and their in- valuable technical assistance during the research. Their contributions have significantly enriched the quality of this work. 398 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES BIBLIOGRAPHY ALAKRASH, Hussein/Bushra EDAM/Elaf BUSTAN/Mustofa ARMNAZI/Ayman ENAYAT/Taif BUSTAN (2021) Developing English language skills and confidence using local culture-based materials in EFL curriculum. Linguistica Antverpiensia 1/1, 548–564. ALARIO-HOYOS, Carlos/Iria ESTÉVEZ-AYRES/Mar PÉREZ-SANAGUSTÍN/Car- los Delgado KLOOS/Carmen FERNÁNDEZ-PANADERO (2017) Understanding learners’ motivation and learning strategies in MOOCs. The International Review of Research in Open and Distributed Learning 18/3. https://doi.org/10.19173/irrodl. v18i3.2996. ALMANASREH, Enas/Rebekah MOLES/Timothy F. CHEN (2019) Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy 15/2, 214–221. https://doi.org/10.1016/j.sapharm.2018.03.066. ALSALEEM, Basmah Issa (2018) The effect of Facebook activities on enhancing oral communication skills for EFL learners. International Education Studies 11/5, 144– 153. https://doi.org/10.5539/ies.v11n5p144. APRIANOTO/HAERAZI (2019) Development and assessment of an interculture-based Instrument model in the teaching of speaking skills. Universal Journal of Education- al Research 7/12, 2796–2805. https://doi.org/10.13189/ujer.2019.071230. BYRAM, Michael/Carol MORGAN (1994) Teaching and learning language and culture. Bristol, Blue Ridge Summit: Multilingual Matters. https://doi. org/10.21832/9781800418158. CANDEL-MORA, Miguel Ángel (2015) Attitudes towards Intercultural communicative competence of English for specific purposes students. Procedia – Social and Behav- ioral Sciences 178, 26–31. https://doi.org/10.1016/j.sbspro.2015.03.141. CAVANAGH, Thomas M./Christopher LEEDS/Janet M. PETERS (2019) Increasing oral communication self-efficacy improves oral communication and general academic performance. Business and Professional Communication Quarterly 82/4, 440–457. https://doi.org/10.1177/2329490619853242. CHENG, Ching-Mei (2012) The influence of collage EFL teachers’ understandings of intercultural competence on their self-reported pedagogical practice in Taiwan. Eng- lish Teaching: Practice and Critique 11, 164–182. DARONG, Hieronimus Canggung/Sebastianus MENGGO (2021) Repositioning culture in teaching target language: Local culture or target culture? Premise: Journal of English Education 10/2, 250–266. https://doi.org/10.24127/pj.v10i2.4106. DERAKHSHAN, Ali/Farhanaz TAHERY/Nasrin MIRARAB (2015) Helping adult and young learners to communicate in speaking classes with confidence. Mediterra- nean Journal of Social Sciences 6/2, 520–525. https://doi.org/10.5901/mjss.2015. v6n2p520. 399B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... FAREEN, Jabbar Al Muzzamil (2018) Job focus: Revisiting students’ communicative needs and industrial demands. Journal of Language and Education 4/4, 42–53. https://doi.org/10.17323/2411-7390-2018-4-4-42-53. FITRIYAH, Ima/Fitriatul MASITOH/Utami WIDIATI (2022) Classroom-based langua- ge assessment literacy and professional development need between novice and expe- rienced EFL teachers. Indonesian Journal of Applied Linguistics 12/1, 124–134. https://doi.org/10.17509/ijal.v12i1.46539. GELIŞLI, Yucel/Lyazzat BEISENBAYEVA (2017) Scientific inquiry competency per- ception scale (The Case of Kazak Post-Graduate students) reliability and validity study. International Journal of Instruction 10/1, 273–288. https://doi.org/10.12973/ iji.2017.10117a. ILIE, Oana-Antonia (2019) The intercultural competence. developing effective intercul- tural communication skills. International Conference Knowledge-Based Organiza- tion, 264–268. https://doi.org/10.2478/kbo-2019-0092. KHOTIMAH, Khusnul/Deisyi Anna BATUNAN/Nur MUKMINATIEN (2024) Teach- ers’ made writing assessment instrument: Pedagogical insight from process to product. Studies in English Language and Education 11/1, 95–115. https://doi. org/10.24815/siele.v11i1.26774. KOLLER, Ingrid/Michael R. LEVENSON/Judith GLÜCK (2017) What do you think you are measuring? A mixed-methods procedure for assessing the content valid- ity of test items and theory-based scaling. Frontiers in Psychology 8. https://doi. org/10.3389/fpsyg.2017.00126. LANDIS, J. Richard/Gary G. KOCH (1977) The measurement of observer agreement for categorical data. Biometrics 33, 159–174. LÓPEZ, Ángel Rodriguez/Jaime E. SOUTO/Maria Lusia Arroyo NOBLEJAS (2019) Improving teaching capacity to increase student achievement: The key role of com- munication competences in Higher Education. Studies in Educational Evaluation 60, 205–213. https://doi.org/10.1016/j.stueduc.2018.10.002. MAMO, Betegiorgis/Abiy YIGZAW (2015) Fidelity of peer mediation and its role in im- proving students’ oral English communication skills. Science, Technology and Arts Research Journal 4/1, 203–214. https://doi.org/10.4314/star.v4i1.33. MARHAMAH, Riadhul/Bukhari DAUD/Iskandar Abdul SAMAD (2017) Integrating target language culture into teaching-learning EFL. English Education Journal 8/4, 488–502. MASITOH, Fitriatul/Ima FITRIYAH/Nur MUKMINATIEN (2023) Developing reading assessment instrument for intermediate EFL learners: AZWAR model. English Re- view: Journal of English Education 11/3, 925–934. https://doi.org/10.25134/erjee. v11i3.7461. MORRISON, Garry R. (2010) Designing effective instruction (6th Edition). John Wiley & Sons. 400 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES NEMATI, Azadeh/Amir MARZBAN/Ensieh MALEKI (2014) The role of intercultural competence in foreign language teaching. International Journal of Language and Linguistics (Special Issue: Innovations in Foreign Language Teaching) 2/6–1, 8–12. RAHMAN, Abdul/Syahban Mada ALI/BADRIAH (2022) Integrating of local cul- ture-based instruction in teaching speaking classroom at the third semester students of Universitas Tomakaka Mamuju. Inspiring: English Education Journal 5/1, 64– 72. https://doi.org/10.35905/inspiring.v5i1.2728. RAHNAMA, Mobina/Neda Fatehi RAD/Hamid BAGHERI (2016) Developing EFL learners’ speaking ability, accuracy, and fluency. ELT Voices 6/1, 1–7. RICHARDS, Jack C. (2006) Communicative language teaching today. Cambridge Uni- versity Press. SABNANI, Robbie Lee (2024) Strategy instruction and enhancement of young Learners’ speaking skills. Language Teaching Research 41, 67–80. SANTOSO, Agus Muji/P. R. PRIMANDIRI/S. ZUBAIDAH/M. AMIN (2021) The development of students’ worksheets using project-based learning (PjBL) in improving higher order thinking skills (HOTs) and time management skills of students. Journal of Physics: Conference Series 1806/1. https://doi. org/10.1088/1742-6596/1806/1/012173. SARABADANI, Javad/Majid SANATKHANI/Maryam AMIRCHAGHMAGHI/Majid Fahresti SANI (2015) Quantification of the content validity of written tests at Mash- had Dental school in 2011–2012. Future of Medical Education Journal 5/4, 30–42. SEFTIKA, Seftika/Miftahul JANAH/Wuri SYAPUTRI (2017) Improving students’ un- derstanding of Indonesian culture through English learning based local culture con- text. ELT-Lectura 4/2, 19–24. SHWETA/Ram BAJPAI/Himanshu CHATURVEDI (2015) Evaluation of inter-rater agreement and inter-rater reliability for observational data: an overview of concepts and methods. Journal of the Indian Academy of Applied Psychology 41/3, 20–27. SLAMET, Joko/Nur MUKMINATIEN (2024) Developing an online formative assess- ment instrument for listening skill through LMS. LEARN Journal: Language Edu- cation and Acquisition Research Network 17/1, 188–211. WAHYUNI, Dwi/Diemroh IHSAN/Rita HAYATI (2014) Linguistic competence and speaking performance of English education study program students of Sriwijaya University. The Journal of English Literacy Education: The Teaching and Learning of English as a Foreign Language 1/2, 83–92. WEHMEIER, Peter M./Theresa FOX/Johanna M. DOERR/Nadja SCHNIERER/Matthi- as BENDER/Urs M. NATER (2020) Development and validation of a brief meas- ure of self-management competence: The self-management self-test (SMST). Ther- apeutic Innovation & Regulatory Science 54/3, 534–543. https://doi.org/10.1007/ s43441-019-00085-3. 401B. Mandasari, M. Rahmah, N. Mukminatien: DEVELOPING AN ASSESSMENT INSTRUMENT ... WERDININGSIH, Indah/Nur MUKMINATIEN (2023) Testing public speaking pro- ficiency: Designing an assessment instrument for university students. Internation- al Social Sciences and Humanities 2/2, 422–431. https://doi.org/10.32528/issh. v2i2.261. POVZETEK RAZVOJ OCENJEVALNEGA MODELA ZA USTNE PREDSTAVITVE DODIPLOMSKIH ŠTUDENTOV O INDONEZIJSKI KULTURI PRI POUKU ANGLEŠČINE KOT TUJEGA JEZIKA Ker povpraševanje po visoko usposobljenih diplomantih z dobrimi sposobnostmi ustnega spora- zumevanja narašča, je v visokošolskem izobraževanju poseben poudarek na učenju angleškega jezika. Poučevanje jezika z navezovanjem na elemente lokalnih kultur lahko poveča študentovo samozavest, zavzetost in kulturno ozaveščenost. Pomanjkanje učinkovitih ocenjevalnih orodij, s pomočjo katerih bi lahko pri pouku ovrednotili študentovo sporazumevalno sposobnost, pa ovira doseganje učnih ciljev. Namen naše raziskave je bil razviti ocenjevalni model za ustno spora- zumevanje, ki temelji na napotkih oz. spodbudah za ustno sporazumevanje in kazalnikih za ocen- jevanje ustnega sporazumevanja o različnih kulturah. Ocenjevalna orodja, katerih ustreznost so preverili strokovnjaki, smo zasnovali posebej za potrebe ocenjevanja kulturno obarvanih govornih dejavnosti pri pouku angleščine. Izhodišče naše raziskave je bil predmet angleške konverzacije za potrebe turizma na zasebni unverzi v Indoneziji. Na podlagi opisa predmeta in njegovih učnih ciljev smo se lotili raziskave in razvoja v okviru modela ADDIE, ki obsega analizo, načrtovanje, razvoj, izvedbo in vrednotenje. Podatki iz relevantne literature in mnenja strokovnjakov, ki so ovrednotili ocenjevalno orodje, so pokazali, da je uporaba napotka oz. spodbude za ustno spo- razumevanje utemeljena in pomembna za doseganje učnih ciljev ter da je uporaba kazalnikov za ocenjevanje ustnega sporazumevanja o različnih kulturah utemljena in ustrezna pri ocenjevan- ju kulturno obarvanih govornih dejavnosti. Govorne dejavnosti naj bi pripomogle k razvoju treh kompetenc: jezikovne in medkulturne kompetence ter kompetence samoupravljanja. Predlagamo, da se v prispevku predstavljeni model uporabi kot podlaga za oblikovanje ocenjevalnih orodij tudi na drugih področjih razvoja jezikovnih kompetenc pri pouku angleškega jezika, ne le na področju govornega sporazumevanja. Vsestranskost predstavljenega modela ponuja možnosti za nadaljnje raziskave in za uporabo modela v drugih izobraževalnih okoljih. Ključne besede: ocenjevanje, orodje, govorjenje, sposobnost ustnega sporazumevanja 402 VESTNIK ZA TUJE JEZIKE/JOURNAL FOR FOREIGN LANGUAGES ABSTRACT DEVELOPING AN ASSESSMENT INSTRUMENT MODEL FOR INDONESIAN CULTURE-BASED ORAL PRESENTATIONS FOR UNDERGRADUATE EFL STUDENTS As demand grows for highly qualified graduates with strong oral communication skills, prioritizing English-speaking proficiency in higher education has become essential. Incorporating local cul- tures into language teaching can boost students’ confidence, engagement, and cultural awareness. However, the absence of effective classroom-based assessment instruments to measure students’ communication competence hinders the achievement of learning objectives. This study aims to de- velop a speaking assessment model in terms of a valid speaking prompt and culture-based speaking assessment indicators. Experts verified the instruments based on content validity. The instruments were designed specifically to meet the needs of classroom-based assessment instruments in cul- ture-based speaking activities. Developed from the course description and learning objectives of English for tourism conversation course at a private university in Indonesia, this study employed research and development (R&D) by adopting the ADDIE model covering the stages of analysis, design, development, implementation, and evaluation. After gathering data by reviewing related literature and involving experts to validate the instruments, this study obtained the following con- clusions. First, the speaking prompt is valid and relevant to the learning objectives; second, the cul- ture-based speaking assessment indicators are valid and relevant to assess culture-based speaking activities. Three competencies are expected to be achieved from the speaking activities – linguistic, intercultural communication, and self-management. Therefore, it is proposed that the model be used to create a classroom-based assessment instrument not only for speaking skills but also for other aspects of English language development. This versatility highlights the potential for further research and the application of the model to various other educational contexts. Keywords: assessment, instrument, speaking, oral communication competence