Abstract

Conclusion. It is expected that the results of this study can contribute to maintaining the quality of the distance learning environment in higher education in Indonesia through the development of distance learning environment instruments for the Open University as a form of reflection by Open University students on teaching materials and their evaluation for the Open University academic community.

Keywords: Instrument development, learning environment, distance education, Indonesia, Open University

Adams, D., & Farmer, G. L. (2022). Graded Response Modeling of the DESSA’s Self- and Social Awareness Subscales in a Special Needs Sample. Youth, 2(2), 98–112. https://doi.org/10.3390/youth2020008

Al-Jarf, R. (2021). Efl Speaking Practice in Distance Leaning During the Coronavirus Pandemic 2020-2021. International Journal of Research -GRANTHAALAYAH, 9(7), 179–196. https://doi.org/10.29121/granthaalayah.v9.i7.2021.4094

Al-mawee, W., Morgan, K., & Gharaibeh, T. (2021). International Journal of Educational Research Open Student ’ s perspective on distance learning during COVID-19 pandemic : A case study of Western Michigan University , United States. International Journal of Educational Research Open, 2(September), 100080. https://doi.org/10.1016/j.ijedro.2021.100080

Anderson, T., & Dron, J. (2011). International Review of Research in Open and Distributed Learning. International Review of Research in Open and Distributed Learning, 12(3), 80–97.

Aristeidou, M., & Cross, S. (2021). Disrupted distance learning: the impact of Covid-19 on study habits of distance learning university students. Open Learning, 36(3), 263–282. https://doi.org/10.1080/02680513.2021.1973400

Baker, F. B. (2001). The Basics of Item Response Theory; ERIC Clearinghouse on Assessment and Evaluation. College Park. https://doi.org/10.1007/978-3-319-54205-8_1

Baker, F. B., & Kim, S.-H. (2017). Test Calibration BT  - The Basics of Item Response Theory Using R (F. B. Baker & S.-H. Kim (Eds.); pp. 105–125). Springer International Publishing. https://doi.org/10.1007/978-3-319-54205-8_7

Banerjee, M., Capozzoli, M., McSweeney, L., & Sinha, D. (1999). Beyond kappa: A review of interrater agreement measures. Canadian Journal of Statistics, 27(1), 3–23. https://doi.org/10.2307/3315487

Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. In Psychological Bulletin (Vol. 88, Issue 3, pp. 588–606). American Psychological Association. https://doi.org/10.1037/0033-2909.88.3.588

Borisova, O. V., Vasbieva, D. G., Malykh, N. I., Vasnev, S. A., & Bírová, J. (2006). International electronic journal of mathematics education. International Electronic Journal of Mathematics Education, 11(5), 1175–1184.

Bork-Hüffer, T., Kulcar, V., Brielmair, F., Markl, A., Immer, D. M., Juen, B., Walter, M. H., & Kaufmann, K. (2021). University students’ perception, evaluation and spaces of distance learning during the covid-19 pandemic in austria: What can we learn for post-pandemic educational futures? Sustainability (Switzerland), 13(14). https://doi.org/10.3390/su13147595

Chang, C. W., Lee, J. H., Wang, C. Y., & Chen, G. D. (2010). Improving the authentic learning experience by integrating robots into the mixed-reality environment. Computers and Education, 55(4), 1572–1578. https://doi.org/10.1016/j.compedu.2010.06.023

Chin, K. Y., Lee, K. F., & Chen, Y. L. (2018). Using an interactive ubiquitous learning system to enhance authentic learning experiences in a cultural heritage course. Interactive Learning Environments, 26(4), 444–459. https://doi.org/10.1080/10494820.2017.1341939

Clayton, J. F. (2007). Development and Validation of an Instrument for Assessing Online Learning Environments in Tertiary Education: The Online Learning Environment Survey (OLLES) [Curtin University of Technology]. https://doi.org/10.1002/(SICI)1098-237X(199704)81:2<161::AID-SCE3>3.0.CO;2-D

Creswell, J. W. (2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. In Educational Research.

Czerniewicz, L., Agherdien, N., Badenhorst, J., Belluigi, D., Chambers, T., Chili, M., de Villiers, M., Felix, A., Gachago, D., Gokhale, C., Ivala, E., Kramm, N., Madiba, M., Mistri, G., Mgqwashu, E., Pallitt, N., Prinsloo, P., Solomon, K., Strydom, S., … Wissing, G. (2020). A Wake-Up Call: Equity, Inequality and Covid-19 Emergency Remote Teaching and Learning. Postdigital Science and Education, 2(3), 946–967. https://doi.org/10.1007/s42438-020-00187-4

Daroedono, E., Siagian, F. E., Alfarabi, M., Cing, J. M., Arodes, E. S., Sirait, R. H., Suryowati, T., Sunarti, L. S., Ahmad, L. N., Wiyanto, M., Kurniaty, L., & Hutabarat, R. S. O. (2020). The impact of COVID-19 on medical education: our students perception on the practice of long distance learning. International Journal of Community Medicine and Public Health, 7(7), 2790–2796.

de la Peña, D., Lizcano, D., & Martínez-Álvarez, I. (2021). Learning through play: Gamification model in university-level distance learning. Entertainment Computing, 39(April). https://doi.org/10.1016/j.entcom.2021.100430

Demetriou, L., Keramioti, L., & Hadjicharalambous, D. (2021). Examining the Relationship Between Distance Learning Processes and University Students’ Anxiety in Times of Covid19. European Journal of Social Sciences Studies, 6(2). https://doi.org/10.46827/ejsss.v6i2.1012

Derakhshandeh, Z., & Esmaeili, B. (2020). Active-Learning in the Online Environment. Journal of Educational Multimedia and Hypermedia, 29(4), 299–311.

Dorman, J. P. (1999). The Development and Validation of an Instrument to Assess Institutional-Level Environment in Universities. Learning Environments Research, 1, 333–352. https://doi.org/10.1023/A:1008883508462

Falk, C. F. (2020). The Monotonic Polynomial Graded Response Model: Implementation and a Comparative Study. Applied Psychological Measurement, 44(6), 465–481. https://doi.org/10.1177/0146621620909897

Fernández-Pascual, M. D., Ferrer-Cascales, R., Reig-Ferrer, A., Albaladejo-Blázquez, N., & Walker, S. L. (2015). Validation of a Spanish version of the Distance Education Learning Environments Survey (DELES) in Spain. Learning Environments Research, 18(2), 179–196. https://doi.org/10.1007/s10984-015-9179-0

Ferrando, P. J., & Lorenzo-Seva, U. (2018). Assessing the Quality and Appropriateness of Factor Solutions and Factor Score Estimates in Exploratory Item Factor Analysis. Educational and Psychological Measurement, 78(5), 762–780. https://doi.org/10.1177/0013164417719308

Fleiss, J. R. (1971). Measuring nominal scale agreement. Psychological Bulletin, 76(5), 378–382.

Friedman, E. M., & Mare, R. D. (2019). The Schooling of Offspring and the Survival of Parents Author ( s ): Esther M . Friedman and Robert D . Mare Published by : Springer on behalf of the Population Association of America Stable URL : https://www.jstor.org/stable/42920056. 51(4), 1271–1293.

Geert Hofstede, & Michael H. Bond. (1984). Hofstede´s Culture Dimensions. An independent Validation Using Rokeach´s Value Survey. Journal of Cross-Cultural Psychology, 15(4), 417–433.

Gupte, T., Watts, F. M., Schmidt-McCormack, J. A., Zaimi, I., Gere, A. R., & Shultz, G. V. (2021). Students’ Meaningful Learning Experiences from Participating in Organic Chemistry Writing-To-Learn Activities. Chemistry Education Research and Practice, 22(2), 396–414. https://doi.org/10.1039/x0xx00000x

Guttman, L. (1954). Some necessary conditions for common-factor analysis. Psychometrika, 19(2), 149–161. https://doi.org/10.1007/BF02289162

Herrington, J., Reeves, T. C., & Oliver, R. (2010). A Guide to Authentic e-Learning. Routledge.

Hinkel, E. (2005). Handbook of Research in Second Language Teaching and Learning. In Handbook of Research in Second Language Teaching and Learning. https://doi.org/10.4324/9781410612700

Hlib, P., Zatonatska, T., & Liutyi, I. (2019). Utilization of Information Technologies in Higher Education. 2019 IEEE International Conference on Advanced Trends in Information Theory, ATIT 2019 - Proceedings, 349–354. https://doi.org/10.1109/ATIT49449.2019.9030449

Hodges, C., Moor, S., Lockee, B., Trust, T., & Bond, A. (2022). The Difference between Emergency Remote Teaching and Online Learning. Frontiers in Education, 7. https://doi.org/10.3389/feduc.2022.921332

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

Informatika, K. K. dan. (2013). Indonesia Miliki Kekayaan dan Keanekaragaman Budaya. Kominfo. https://kominfo.go.id/index.php/content/detail/3415/Kominfo+%3A+Pengguna+Intern et+di+Indonesia+63+Juta+Orang/0/berita_satker

Ismaili, Y. (2020). Evaluation of students’ attitude toward distance learning during the pandemic (Covid-19): a case study of ELTE university. On the Horizon, 29(1), 17–30. https://doi.org/10.1108/OTH-09-2020-0032

Jack R. Fraenkel, Jessen, K. R., & Mirsky, R. (2008). How to Design Research in Education and Evaluate. In Glia.

Joyner, D. A., Wang, Q., Thakare, S., Jing, S., Goel, A., & MacIntyre, B. (2020). The Synchronicity Paradox in Online Education. L@S 2020 - Proceedings of the 7th ACM Conference on Learning @ Scale, August 2020, 15–24. https://doi.org/10.1145/3386527.3405922

Junior High School Directorate Manager. (2022). Kenali 9 Aspek dalam Survei Lingkungan Belajar ANBK. Direktorat SMP. https://ditsmp.kemdikbud.go.id/kenali-9-aspek-dalam-survei-lingkungan-belajar-anbk/

Kirschner, P., Strijbos, J. W., Kreijns, K., & Beers, P. J. (2004). Designing electronic collaborative learning environments. Educational Technology Research and Development, 52(3), 47–66. https://doi.org/10.1007/BF02504675

Koes, A. (2021). Pendidikan RI Ketinggalan 128 Tahun, Mengejarnya Jangan Tunggu Pemerintah. Gatra.

LaHuis, D. M., Clark, P., & O’Brien, E. (2011). An Examination of Item Response Theory Item Fit Indices for the Graded Response Model. Organizational Research Methods, 14(1), 10–23. https://doi.org/10.1177/1094428109350930

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

Lazorak, O., Belkina, O. V., & Yaroslavova, E. (2021). Changes in Student Autonomy via E-Learning Courses. International Journal of Emerging Technologies in Learning, 16(17), 209–225. https://doi.org/10.3991/ijet.v16i17.23863

LeBuffe, P. A., Shapiro, V. B., & Robitaille, J. L. (2018). The Devereux Student Strengths Assessment (DESSA) comprehensive system: Screening, assessing, planning, and monitoring. Journal of Applied Developmental Psychology, 55, 62–70. https://doi.org/https://doi.org/10.1016/j.appdev.2017.05.002

Ludwig-Hardman, S., & Dunlap, J. C. (2003). Learner support services for online students: Scaffolding for success. International Review of Research in Open and Distance Learning, 4(1), 45–61. https://doi.org/10.19173/irrodl.v4i1.131

Mäkelä, T., & Helfenstein, S. (2016). Developing a conceptual framework for participatory design of psychosocial and physical learning environments. Learning Environments Research, 19(3), 411–440. https://doi.org/10.1007/s10984-016-9214-9

Masalimova, A. R., Ryazanova, E. L., Tararina, L. I., Sokolova, E. G., Ikrennikova, Y. B., Efimushkina, S. V., & Shulga, T. I. (2021). Distance learning hybrid format for university students in post-pandemic perspective: Collaborative technologies aspect. Cypriot Journal of Educational Sciences, 16(1), 389–395. https://doi.org/10.18844/cjes.v16i1.5536

Mateucci, M., & Stracqualursi, L. (2006). STUDENT ASSESSMENT VIA GRADED RESPONSE MODEL. Statistica, 66(4), 435–447.

Maydeu-Olivares, A. (2013). Goodness-of-Fit Assessment of Item Response Theory Models. Measurement: Interdisciplinary Research and Perspectives, 11(3), 71–101. https://doi.org/10.1080/15366367.2013.831680

Minister of Education, Culture, Research and Technology Regulation No. 53 of 2023 concerning Quality Assurance of Higher Education, 5 1 (2023).

Regulation of the Minister of Education and Culture of the Republic of Indonesia Number 109 of 2013 Concerning the Implementation of Distance Education in Higher Education, Minister of Education and Culture of the Republic of Indonesia Number 109 of 2013 1 (2013).

Mokken, R. J. (2011). A Theory and Procedure of Scale Analysis: With Applications in Political Research. De Gruyter.

Moore, J. L., Dickson-Deane, C., & Galyen, K. (2011). E-Learning, Online Learning, and Distance Learning Environments: Are they the same? The Internet and Higher Education, 14(2), 129–135. https://doi.org/https://doi.org/10.1016/j.iheduc.2010.10.001

Moos, R. H. (1979). Evaluating Educational Environments. Jossey-Bass Publishers.

Nestulya, S., & Shara, S. (2023). Distance learning as a relevant educational technology in higher education institutions. Scientific Bulletin of Mukachevo State University Series “Pedagogy and Psychology”, 9(1), 39–46. https://doi.org/10.52534/msu-pp1.2023.39

Newby, M. (1998). A Study of The Effectiveness of Computer Laboratory Classes as Learning Environments. Curtin University of Technology.

Nguyen, T. H., Han, H. R., Kim, M. T., & Chan, K. S. (2014). An Introduction to Item Response Theory for Patient-reported Outcome Measurement. Patient, 7(1), 23–35. https://doi.org/10.1007/s40271-013-0041-0

Ningtyas, L. D., Putrawan, I. M., & Rahayu, W. (2020). Student ’ s Environmental Personality and Sensitivity ’ s Construct Validity Analyzed by Multitrait-Multimethod. Palarch’s Journal Of Archaeology Of Egypt/Egyptology, 17(9), 3.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (Third edit). McGraw-Hill, Inc.

Onwuegbuzie, A. J., Bustamante, R. M., & Nelson, J. A. (2009). Mixed Research as a Tool for Developing Quantitative Instruments. Journal of Mixed Methods Research, 4(1), 56–78. https://doi.org/10.1177/1558689809355805

Özkök, A., Walker, S. L., & Büyüköztürk, Ş. (2009). Reliability and validity of a Turkish version of the DELES. Learning Environments Research, 12(3), 175–190. https://doi.org/10.1007/s10984-009-9060-0

Pearson, J., & Trinidad, S. (2005). OLES: An instrument for refining the design of e-learning environments. Journal of Computer Assisted Learning, 21(6), 396–404. https://doi.org/10.1111/j.1365-2729.2005.00146.x

Polianovskyi, H., Zatonatska, T., Dluhopolskyi, O., & Liutyi, I. (2021). Digital and Technological Support of Distance Learning at Universities under COVID-19 (Case of Ukraine). Revista Romaneasca Pentru Educatie Multidimensionala, 13(4), 595–613. https://doi.org/10.18662/rrem/13.4/500

Polman, J., Hornstra, L., & Volman, M. (2021). The meaning of meaningful learning in mathematics in upper-primary education. Learning Environments Research, 24(3), 469–486. https://doi.org/10.1007/s10984-020-09337-8

Putra, N. S. (2021). Pendidikan di Indonesia Yang Belum Merata. Kabar Pendidikan, 1.

Rahayu, W., Putra, M. D. K., Iriyadi, D., Rahmawati, Y., & Koul, R. B. (2020). A Rasch and factor analysis of an Indonesian version of the Student Perception of Opportunity Competence Development (SPOCD) questionnaire. Cogent Education, 7(1). https://doi.org/10.1080/2331186X.2020.1721633

Rahmawati, Y., Ridwan, A., Cahyana, U., & Wuryaningsih, T. (2020). The integration of ethnopedagogy in science learning to improve student engagement and cultural awareness. Universal Journal of Educational Research, 8(2), 662–671. https://doi.org/10.13189/ujer.2020.080239

Rahmawati, Y., Ridwan, A., & Nurbaity. (2017). Should we learn culture in chemistry classroom? Integration ethnochemistry in culturally responsive teaching. AIP Conference Proceedings, 1868. https://doi.org/10.1063/1.4995108

Recker, J., & Rosemann, M. (2010). A Measurement Instrument for Process Modeling Research. Scandinavian Journal of Information Systems, 22(2), 3–30.

Reise, S. P., & Waller, N. G. (2009). Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5, 27–48. https://doi.org/10.1146/annurev.clinpsy.032408.153553

Richards, S., Cutler, D. M., & Meara, E. R. (2008). The Gap Gets Bigger: Changes in Mortality and Life Expectancy, By Education, 1981-2000. Health Affairs, 27(2), 350–360.

Riyadi, Madani, F., Apriliani, L., & Siva, A. N. (2023). Peningkatan Kompetensi Guru melalui Penyusunan Instrumen Penilaian Berbasis HOTS. Sarwahita, 19, 568–582. https://doi.org/10.21009/sarwahita.19k.6

Samejima, F. (2016). Graded response models. Handbook of Item Response Theory, 1, 95–107. https://doi.org/10.1201/9781315374512

Sheffield, R., Koul, R., Blackley, S., Fitriani, E., Rahmawati, Y., & Resek, D. (2018). Transnational examination of STEM education. International Journal of Innovation in Science and Mathematics Education, 26(8), 67–80.

Snipes, M., & Taylor, D. C. (2014). Model selection and Akaike Information Criteria: An example from wine ratings and prices. Wine Economics and Policy, 3(1), 3–9. https://doi.org/https://doi.org/10.1016/j.wep.2014.03.001

Spencer, R. (2003). Handbook of Distance Education. In The International Review of Research in Open and Distributed Learning (Vol. 4, Issue 2). https://doi.org/10.19173/irrodl.v4i2.143

Tabachnick, B. G., & Fidell, L. S. (1989). Using Multivariate Statistics. In Harper-Collins Publisher, Inc. Harper-Collins Publisher, Inc.

Teddlie, C., & Tashakkori, A. (2009). Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. SAGE Publications.

Terbuka, U. (2023). Visi dan Misi. Universitas Terbuka. http://www.jpkk.edu.my/jpkk/index.php/maklumat-kami/visi-dan-misi.html

Tong, F., Tang, S., Irby, B. J., Lara-Alecio, R., & Guerrero, C. (2020). The determination of appropriate coefficient indices for inter-rater reliability: Using classroom observation instruments as fidelity measures in large-scale randomized research. International Journal of Educational Research, 99(June 2019), 101514. https://doi.org/10.1016/j.ijer.2019.101514

Treagust, D. F., & Fraser, B. J. (1986). Validation and Application of the College and University Classroom Environment Inventory (CUCEI). American Educational Research Association, 1–25.

Tsarapkina, J., Lyapunzova, E., Anisimova, A., & Mironov, A. (2021). Overview of European Universities with Open Distance Learning. E3S Web of Conferences, 258, 1–7. https://doi.org/10.1051/e3sconf/202125807066

Universitas Merdeka Surabaya. (2021). Universitas Merdeka Surabaya. https://lms.unmerbaya.ac.id/?lang=id

Universitas Siber Asia. (2023). PJJ MANAJEMEN - Universitas Siber Asia. Universitas Siber Asia. https://unsia.ac.id/pjj-manajemen/

Universitas Siber Muhammadiyah. (2021). Halaman Depan - Universitas Siber Muhammadiyah. Universitas Siber Muhammadiyah. https://sibermu.ac.id/

Universitas Terbuka. (2023). Making Higher Education Open to All. Universitas Terbuka. https://www.ut.ac.id/

Utomo Priyambodo. (2022). Pendidikan di Indonesia Tertinggal dari Negara-Negara Tetangga. National Geographic.

Virtič, M. P., Dolenc, K., & Šorgo, A. (2021). Changes in online distance learning behaviour of university students during the coronavirus disease 2019 outbreak, and development of the model of forced distance online learning preferences. European Journal of Educational Research, 10(1), 393–411. https://doi.org/10.12973/EU-JER.10.1.393

Walker, S. L., & Fraser, B. J. (2005). Development and validation of an instrument for assessing distance education learning environments in higher education: The Distance Education Learning Environments Survey (DELES). Phenomenology and the Cognitive Sciences, 4(3), 289–308. https://doi.org/10.1007/s10984-005-1568-3

Wasson, B., & Kirschner, P. A. (2020). Learning Design: European Approaches. TechTrends, 64(6), 815–827. https://doi.org/10.1007/s11528-020-00498-0

Williams, K. (2024). Sample Size Calculator with Slovin’s Formula for More Precise Surveys. Survey Sparrow. https://surveysparrow.com/blog/sample-size-calculator/

Williamson, B., Eynon, R., & Potter, J. (2020). Pandemic politics, pedagogies and practices: digital technologies and distance education during the coronavirus emergency. Learning, Media and Technology, 45(2), 107–114. https://doi.org/10.1080/17439884.2020.1761641

Wulandari, R. (2021, July 17). Mengapa Kualitas Pendidikan Indoensia Masih Tertinggal? Kompasiana.Com.

Xu, J., Paek, I., & Xia, Y. (2017). Investigating the Behaviors of M2 and RMSEA2 in Fitting a Unidimensional Model to Multidimensional Data. Applied Psychological Measurement, 41(8), 632–644. https://doi.org/10.1177/0146621617710464

Yu, Z. (2022). Sustaining Student Roles, Digital Literacy, Learning Achievements, and Motivation in Online Learning Environments during the COVID-19 Pandemic. Sustainability (Switzerland), 14(8). https://doi.org/10.3390/su14084388