ICT and Internet Usage among Kosovar Students: The Impact of Trends on Achievement in the PISA Scales

Arif Shala and Albulene Grajcevci

2023 VOL. 10, No. 1

Abstract: The current analysis aims to research the impact of ICT usage on students’ achievement in the 2018 PISA assessment, specifically achievement in mathematics, science and reading scales. In this assessment, a total of 5,058 randomly selected students from 224 schools across Kosovo took part in the study. In addition, to the subscales assessing reading, mathematics and science achievement, participants also completed a variety of questionnaires developed by OECD to assess perceptions, socioeconomic factors and classroom/school contexts, among others. Current study analysis reveals the link between achievement and ICT usage among students, such as using computers for research, reading news, learning something new, and reading emails, among others. In a developing country where the achievement gap between students is increasing rapidly it is crucial to understand how access to resources hinders equity. Results indicate that access to the internet and computers at home links to higher achievement. However, Kosovar students tend to use technology and the internet mainly to chat with friends, and significantly less for learning something new or for doing homework. Gender differences are evident, with female students reporting a lower online presence. In conclusion, Kosovar students use computers and the internet mainly for entertainment and not for learning, which suggests that achievement cannot be attributed to ICT access and/or use.

Keywords: ICT, internet, PISA performance, Kosovo.

Introduction

Digital skills are becoming more and more important for learning in the current century, with online learning being the most important upcoming trend in knowledge delivery (Finn, 2002). This new form of learning has been accredited for its power to transform education (Dziuban et al., 2005; Garrison & Kanuka, 2004; Graham & Robison, 2007), as well as its unlimited implementation across versatile settings and domains (Bonk et al., 2006; Picciano & Seaman, 2009).

Research on the impact of ICTs in teaching, learning and performance has not been scarce as a wide body of literature exists. The initial studies generally focused on the use of computers but during the years the focus has shifted to the impact of activities such as usage of the internet, blogs, wikis and online platforms, to name just a few. As the focus of the studies was mixed, so were the results of the studies. Studies have come to both conclusions, that ICTs impact learning and performance, as well as the application of ICT in schools does not influence learning and performance in students (Au-Yong-Oliveira et al, 2018; Ausbusson et al., 2014; Sherman et al., 2010).

While research is far from conclusive, the clear advantage of technology is that it enables students to access educational materials and resources online at any time that is suitable for them. The first issue raised here is that technology can also hinder learning by attracting attention to online games or chats, thus distracting learning. Similarly, the use of ICT in learning can also reduce significantly the creativity of learners, since the application of ICT uses a set of acceptable responses/behaviours (Halverson et al., 2012; Livingson & Condie, 2006).

Considering existing evidence, the question is to what extent, if at all, ICT use impacts achievement in domains such as reading, math and science. The study of Fuchs and Woessman (2004) reported that there is a positive correlation between performance in PISA and the availability of computers in school settings. The study of Coates et al. (2004), reported that across three universities students performed 15% higher when they had a face-to-face interaction compared to those who learned only online. Similarly, the study of Rouse et al. (2004) discovered that software programmes designed to enhance language and reading literacy skills, did not lead to language acquisition or the improvement of reading literacy skills as expected. Another study that researched the impact of modes of learning on students who attended statistics and managerial economics classes, discovered that students who had a traditional learning setting performed 14.1% higher compared to students who learned only online. The mode of learning on the other hand, did not significantly affect performance and learning in the managerial economics class (Anstine & Skidmore, 2005).

Education systems across world are expected to prepare students for a globalised society (Livingston, 2001), at the heart of which lies the potential of information and communication technologies to shape learning and achievement (21st Century Literacy Summit, 2002) at a time when research evidence is yet inconclusive. Many supporters of such technologies list the ways in which such development leads to more effective teaching and learning (McGuinness, 1999). Evidently, research studies maintain that such advances make learners more active as well as enhance their thinking skills (Livingston et al., 2004). ICT will also enable teachers to create powerful teaching strategies as well as enhance their professional development. Current studies report that students and teachers are gaining ever increasing access to technologies including but not limited to the internet, laptops, mobile phones and interactive whiteboards (Condie et al., 2005;  et al., 2002).

The issue with ICT application in schools is that research consistently reports mixed implications (Fraillon et al., 2014; Escueta et al., 2017). The findings from PISA also follow this line of reasoning — using ICT in the classroom does not lead to higher performance in PISA scales. The one remaining argument is that using ICTs in the classroom will ultimately impact the time the teacher has at his/her disposal, what has to be taught as well as what methodologies are used to teach, and it is these variables that undeniably shape achievement in students (Scherff & Piazza, 2008; Schmidt & Maier, 2009).

Researching the impact of ICT in learning demands researchers not only understand how often and what types of ICT students used but also what strategies are used in teaching and learning. Students’ digital skills are increased by including competency-based learning in curricula (European Commission, 2013).  In order to address this need, education systems are not anymore teaching ICT skills in one subject, rather, they are expanding digital literacy learning over many subjects and years (European Commission, 2013). This makes ICT skills the skills that are needed in many subjects but the learning of such skills is a topic on its own. Considering the important impact of ICT skills on learning, PISA 2021 assessment will measure ICT literacy as a sole domain. The idea is to assess ICT literacy, more specifically what competence should students have and how these competencies should be assessed. PISA defines ICT literacy as “the interest, attitude and ability of individuals to appropriately use digital technology and communication tools to access, manage, integrate and evaluate information, construct new knowledge and communicate with others in order to participate effectively in society" (Lennon et al., 2003, p. 8).

Research studies link student achievement with the quality of tasks assigned by the teacher (e.g., Education Queensland, 2002; National Council of Mathematics, 2000; Newmann et al., 1996; Snape & Fox-Turnbull, 2013). Evidently there is a growing number of research studies that explore the application of learning technologies by teachers (e.g., see Ertmer et al., 2012; Mueller et al., 2008). Teachers on a daily basis decide what technologies to use based on how these technologies foster student learning (Halverson et. al, 2012; Livingston & Condie, 2006). Teachers make such choices based on their beliefs, self-efficacy and attitude towards learning technologies (Ertmer et al., 2012; Mueller et al., 2008) as well as their competence in IT (Voogt & Knezek, 2008). Nevertheless, Somekh (2008) argues that the responsibility for using technology in teaching and learning remains with the schools. Schools ought to encourage teachers to use technology innovation in their teaching (Livingston & Condie, 2010) and what decides the application of technologies in learning are the perceptions of how useful as well as user friendly such technologies are (Davis, 1989). The application of technologies in teaching has been reported to result in higher satisfaction, and motivation as well as efficiency but also to increase interaction and flexibility (British Educational Communications and Technology Agency, 2008; Kennewell et al., 2008; Smith et al., 2005). The use of technologies in some studies has been linked to higher attention, concentration and reflection as well as more collaboration (Kennewell & Beauchamp, 2007). There are also researchers that maintain that learning technologies do nothing more than reinforce traditional instruction (Higgins 2005, Schuck & Kearney, 2008). Nonetheless, the majority of researchers maintain that technology has changed teaching and learning approaches across schools (Livingston & Condie, 2010).

ICTs in education have been evaluated by PISA as well. Firstly, students in most countries have access to ICT, they have the internet at home and school and these practices have only expanded since the 2015 PISA. In the PISA assessment of 2003, 57% of students participating in the PISA assessment had the internet at home, while this number increased to 92% in the 2015 assessment. By 2015 roughly 91% of students participating in the assessment had the internet in school (Lorenceau et al., 2019).

Considering such developments, it is no surprise that students nowadays spend more time online while they are at home as well as while they are in school. As the internet becomes more readily available, the use of the internet by students is increasing as well. An example of such an enormous increase is the country of Costa Rica where students in 2012 spent 19 hours online, but in 2015 spent 37 hours online, a considerable increase of 18 hours per week. On average, in OECD countries in 2015, students reported spending 2.7 hours more online compared to 2012. The increasing trend of internet usage is not showing any signs of decrease (Lorenceau et al., 2019). According to PISA data students in Taipei spent 7.7 hours more online in 2015 than they did in 2012; Belgian students spent 7.3 hours more; and students in Austria and Spain spent 6.3 hours more online in 2015 compared to 2012 (Lorenceau et al., 2019).

On average, data from the 2012 and 2015 assessments indicate that across all OECD countries the hours students spent online have increased from 21 to 29 hours per week, an overall increase of 8 hours per week (OECD, 2018). The issue with increased time spent online is that, while students gain access to more opportunities, they also become more exposed towards possible risks (Hooft Graafland, 2018). According to Hooft Graafland, 2018), when students go online they can gain access to online classes, as well as learn more about health, identity and any other issue that may be of relevance to them. According to OECD (2018) students who spend more time online are the disadvantaged students, and such exposure can lead to students not being content with their lives, while also exposing them to dangerous practices such as violence and online bullying to name some (Lorenceau et al., 2019).

The study of Angrist and Lavy (2002) on the impact of computers in elementary and middle school students’ learning indicated that using computers in learning and teaching did not lead to higher scores. In fact, 4th grade students had higher mathematics scores when they attended schools that did not use computers, as compared to students who did attend schools that used computers in teaching and learning. On a similar note, Goolsbee and Guryan (2002) reported that students’ achievement was not increased in schools which invested in internet connections, instead an increase in the use of the internet by students was reported.  In the Netherlands, investments in software and computers have not resulted in enhanced student achievement; instead, they have negatively impacted the learning of language and mathematics (Leuven et. Al, 2004). The study of Benjerjee et al. (2004) conducted in poor urban areas in India discovered that the use of mathematics software in schools did increase mathematics skills, while it did not have a positive impact on other skills or subjects. Similarly, the study of Machin et al. (2006) on the impact of ICT in student performance in the years 1999-2003 reported that the use of ICTs had a positive impact on student performance.

The study of Spiezia (2014) reported that students who used computers more performed better in PISA, this evidently was true across all countries. The differences were larger and significant between students who used a computer every day and those who did so only once or twice a week. In countries such as Norway and Czech Republic, students who used a computer once or twice a week had 40 points more than students who used a computer less than that. 

Furthermore, students who use the computer every day in the case of Chile and the Czech Republic performed 70 to 51 points higher than students who used the computer once or twice a week. In countries such as Norway, Portugal, Croatia and Hungary the difference in performance was 40 points, while in Austria, Belgium, Italy, the Netherlands, Spain and Lithuania the difference was 35 points.

The Objective and Hypotheses

While there may be considerable research studies conducted abroad, no research study exists in the link between ICT usage and Kosovar students. In order to address this research gap, the current study researches the ICT usage trends among Kosovar students, such as using computers for chatting, research information, reading news, learning something new, and reading emails, among others. Furthermore, the study explores gender differences in usage trends, and to explore if gender plays a role in such trends. Finally, the study researches the impact on performance of gender, internet access and access to computers to complete homework. Considering the existing literature, the current study addresses the impact of ICT skills and resources in the performance of Kosovar students in PISA. The study researches the trends among the online presence of Kosovar students in order to understand their impact on achievement.

H1. Students who have a computer they can use for schoolwork will outperform students who do not.

H2. There is a significant positive correlation between having access to the internet and performance in PISA.

H3. There are gender differences in using computers and the internet.

H4. Students will use computers and the internet mainly for activities other than learning.

Methods

Respondents

Current research makes use of the PISA Data Explorer available on the website of the Organization for Economic Co-operation and development (OECD PISA Explorer, 2018). The online dataset provides information on all variables for all countries participating in the PISA assessment. In the case of this publication only the data for Kosovo were analysed. In 2018, 5,058 students from 224 schools participated in the assessment, representing 25,739 students at the age of 15 in Kosovo.

Procedure

During 2018 across 224 schools in Kosovo, students were randomly selected to participate in the PISA study. Instead of the paper and pencil version of the PISA 2015, PISA 2018 was a computer-based assessment for Kosovar students. Once the data have become available, OECD provides data sets that can be downloaded by everyone but it also provides a data explorer that enables researchers to directly conduct analysis on the OECD website on their selected variables (OECD PISA Explorer, 2018). For the present analysis the online PISA data explorer was used. The data explorer allows researchers to conduct several statistical procedures such as T-tests and Anova and obtain p-values. The analyses were done in the PISA data explorer which enables researchers to select variables and conduct tests on their relation to PISA results. The study used the following variables: internet access, access to computers, using computers and the internet to read emails, chat online, read news, learn something new and to search for practical tips. The hypotheses of this study are built on the existing literature. To that end we expect to find a significant positive correlation between student performance in reading literacy, mathematics and science on the one hand, and computer usage trends on the other.

Instrument

Current research makes use of the PISA Data Explorer available on the website of the Organization for Economic Co-operation and development (OECD PISA Explorer, 2018). The online dataset provides information on all variables for all countries participating in the PISA assessment. In the case of this publication only the data for Kosovo were analysed. The subscale used in this study is the home possessions subscale as well as the internet presence and usage subscales.

Results

In terms of gender differences, female students outperformed boys in the reading scale and the science scale. The difference in performance in the reading literacy scale was 26 points, a difference that was significant p < 0.00. In addition, female students outperformed male students by six points in science, a difference which was also significant p < 0.01. The only scale in which male students performed higher than female students was the mathematics scale — four points higher — but this difference was not significant p > 0.05. Students who had a link to the internet at home, performed higher in all achievement scales compared to students who had no link to the internet. The difference was five points in reading literacy, 44 points in mathematics and 36 points in science. In addition, students who had access to a computer at home to use for schoolwork, outperformed students who did not have a computer by five points in reading literacy, 16 points in mathematics and, finally, by 16 points in science.

Table 1: PISA Performance of Kosovar Students According to Gender and Home Possessions

When asked how often they read e-mails 2,152 students which is roughly half of the participants, responded that they did not know what e-mails were or they had never read e-mails. A large number of participants, more specifically 1,262 students reported that they read e-mails several times a month, 812 said that they did so several times a week and, finally, 438 students reported that they read e-mails several times a day. More than half of the participants reported chatting online several times a day (N = 2,707). Students also reported using chat several times a week, more specifically 823 students did so. Considering this number, 76% of the participants used chatting online every day or several times a week. This is a huge contrast to reading e-mails online, when only 27% did so every day or several times a week. Online chat was used several times a month by 514 students, never or almost never by 445, and only 129 reported not knowing what it was. Similar to using the internet for chatting versus checking e-mails, 46% of the participants did not know what e-mails were or never checked them. In contrast, only 12% of the participants did not know what online chats were or never used them.

Table 2: Frequency of Internet Presents of Kosovar Students

When asked if they read online news, 1,060 students reported that they read news many times a day, 1,336 reported that they read news several times a week, 1,042 participants read the news several times a month, 976 never or almost never read the news and, finally, 192 reported that they did not know what this was. When asked about how often they searched online for something new 1,071 students reported doing so every day, 1,663 searched online several times a week, 1,092 searched online several times a month. In contrast 623 reported that they never or almost never searched online for new things and, finally, 195 students did not know what searching online was. To elaborate, 17% of Kosovar students never searched online for new information. Only 657 students reported that they participated in online groups or forums several times a day, and a similar number, 361 students, did not know what this was. Finally, the majority of students searched online for practical information several times a week or several times a month, which was 54% of all participants, only 827 searched online several times a day and, finally, 24% never searched online for the new information.

Table 3: Frequency of Internet Presence of Kosovar Students According to Gender

In terms of gender differences, male students reported that they read e-mails more often than female students, the same trend is visible across all measures, with male students reporting more online presence compared to female students. Understanding gender differences is important primarily because gender differences are evident in PISA studies (Sälzer, 2021), and exploring how girls and boys differ in their use of ICT resources could explain these achievement differences.

Discussion and Conclusion

Recently researchers and policy makers have increased their attention to the impact of Information and Communication Technologies (ICT) on achievement and performance. To that end, a review of current understanding in the domain reveals that policy makers are rather enthusiastic about the impact of ICT to learning and achievement, while researchers on the other hand have not been able to find enough evidence to support the enthusiasm of policy makers (Spiezia, 2014). The current analysis also reveals that in a developing country’s context while students who had access to computers and internet did perform higher that those who did not have access, students mainly used such resources for entertainment and not learning.  The results obtained from the PISA dataset reveal that Kosovar students, tended to use computers and the internet mainly for chatting, the number of students who used such resources decreased in cases of reading email, doing homework or learning something new. Interestingly enough, the trend remained the same for female students as well, however they reported lower usage of computers and the internet for all variables. While female students reported lower internet presence, in the meantime female students outperformed male students in the reading literacy scale as well as the science scale. In both cases the differences were significant.

Changes and advancements in technology have shaped the manner of not only how we interact with one another and communicate, but also how we evolve and learn (Gonçalves et al., 2016; Moreira et al., 2016). Having access to the internet has made individuals connect to one another, become technologically fluent and has raised their global awareness (Jordan, 2016). The changes in technology and young learners, have made the use of ICT in education necessary, while this has also contributed to the issue of knowledge management. The use of technology in teaching, it is argued, makes students more focused than when they are in class (Fonseca et al., 2014), while also making them feel challenged. In the meantime there is also evidence from 35 countries that there is a negative correlation between the use of digital devices and student achievement in PISA scales (Sälzer, 2021). In Kosovo, however there is a lack of education software and tools in the national language and this may have contributed to students not being able to use ICT resources for learning.

Access to such technologies has led to the belief that ICTs contribute to education, as governments have developed online learning platforms, mostly in mathematics and science, in order to promote student learning and enhance the effectiveness of learning. These platforms include not only texts but also simulations and animations. Such platforms go one step further by enabling not only teacher-based assessment but also self-assessment, which allows students to check their learning (Livingston & Condie, 2010). The study by Condie et al. (2005) indicates that, due to the inclusion of ICT in schools, Scottish students were able to use the internet and use e-mail as well as write.

While the presence of technologies is growing across education systems, the understanding of teachers on how to use this technology to increase learning should also grow. Enabling students to learn independently online and then incorporate this learning into classroom learning is the new challenge teachers face (Oliveira et al., 2018). Education technologies have the power to reshape the relationship between teachers and students, with the latter becoming knowledge creators as opposed to passive recipients of knowledge and information from teachers (Livingston & Condie, 2004)

This change in roles will require students to take responsibility for their learning, while teachers will have to adjust from their role as the sole providers of knowledge to leaders of the learning process (Oliveira et al., 2018). Evidently, teachers need to learn how to restructure the relationship with students, how to develop online learning environments that enable students to learn on their own as well as reflect on their learning, and teachers will also need to teach students how to connect different forms of learning as well as previous knowledge to the newly acquired one. Failure to adopt any of these roles leads to teachers withdrawing from the experience (Halverson et. al, 2012; Livingson & Condie, 2006).

In conclusion, large achievement gaps among categories of students in Kosovo may provide a picture of a largely inequitable education system. In this regard the common understanding is that such differences may be due to a lack of resources such as computers and the internet at school and at home. In a first review of results this may well become a main argument, since students who have access to computers and the internet at home had higher achievement in all PISA scales, namely reading literacy, mathematics and science. Students who had access to the internet at home and who had a computer to do their homework on, were the ones who also had higher achievements in all PISA scales.  This may easily contribute to policy makers emphasising the need to equip schools with computers in an attempt to drive achievement of all students and in the meantime ensure equitability. However, a deeper analysis of PISA data reveals that it is important to understand two issues: first how such resources are used by learners and, secondly, in what ways can such resources be used to ensure learning. In regard to the issue of how ICT resources are used by students in Kosovo, the analysis reveals that computers and the internet are mainly used for leisure and learning. This then raises the issue of how to ensure that such resources are truly used for learning, and in the case of Kosovo this may then link to the development of education tools and software in the national language and training teachers to actually use these resources in their teaching. Evidently, in developing countries, equity in education is not secured by making resources available, instead it largely depends on changing perceptions and practices.

References

Angrist, J., & Lavy, V. (2002). New evidence on classroom computers and pupil learning. Economic Journal, 112, 735-765.

Anstine, J., & Skidmore, M. (2005). A small sample study of traditional and online courses with sample selection adjustment. Journal of Economic Education, 36, 107-127.

Au-Yong-Oliveira, M., Gonçalves, R., Martins, J., & Branco, F. (2018). The social impact of technology on millennials and consequences for higher education and leadership. Telematics and Informatics, 35, 956-963.

British Educational Communications and Technology Agency (2008). Harnessing technology school survey 2007: Analysis and key findings. http://dera.ioe .ac.uk/1552/

Bonk, C.J., Kim, K. J., & Zeng, T. (2006). Future directions of blended learning in higher education and workplace learning settings. In C. J. Bonk & C. R. Graham (Eds.), The handbook of blended learning: Global perspectives, local designs (pp. 550–567). Pfeiffer.

Coates, D., Humphreys, B., Kane, J., & Vachris, M.A. (2004). ‘No significant distance’ between face-to-face and online instruction: Evidence from principles of economics. Economics of Education Review, 23(6), 533-546.

Condie, R., Simpson, M., Payne, F., & Gray, D. (2002). The impact of information and communication technology initiatives in Scottish schools (Insight Series No. 2). SEED.

Condie, R., Munro, R., & Muir, D. (2005). The impact of ICT initiatives in Scottish schools: Phase3. SEED.

Davis, F.D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-339.

Dziuban, C., Moskal, P., & Hartman, J. (2005). Higher education, blended learning and the generations: Knowledge is power—no more. In J. Bourne & J. C. Moore (Eds.), Elements of quality online education: Engaging communities (pp. 85–100). Sloan Consortium.

European Commission (2013). Survey of schools: ICT in education — Benchmarking access, use and attitudes to technology in Europe’s schools. http://dx.doi.org/10.2759/94499.

Ertmer, P.A., Ottenbreit-Leftwich, A.T., Sadik, O., Sendurur, E., & Sendurur, P. (2012). Teacher beliefs and technology integration practices: A critical relationship. Computers & Education, 59(2), 423-435.

Escueta, M., Quan, V., Nickow, A.J., & Oreopoulos, P. (2017). Education technology: An evidence-based review. National Bureau of Economic Research. http://dx.doi.org/10.3386/w23744

Fonseca, D., Martí, N., Redondo, E., Navarro, I., & Sánchez, A. (2014). Relationship between student profile, tool use, participation, and academic performance with the use of Augmented Reality technology for visualized architecture models. Computers in Human Behavior, 31, 434-445.

Fuchs T., & Woessmann, L.  (2004). Computers and student learning: Bivariate and multivariate evidence on the availability and use of computers at home and at school. CESIFO Working Paper No. 1321. CESIFO.

Fraillon, J., Ainley, J., Schulz, W., Friedman, T., & Gebhardt, E. (2014). Preparing for life in a digital age. Springer International Publishing, Cham. http://dx.doi.org/10.1007/978-3-319-14222-7

Garrison, D.R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7, 95-105. doi:10.1016/j.iheduc.2004.02.001

Goolsbee, A., & Guryan, J. (2002). The impact of internet subsidies in public schools. NBER Working Paper, 9090.

Gonçalves, R., Martins, J., & Rocha, Á. (2016). Internet e redes sociais como instrumentos potenciadores de negócio. RISTI – Revista Ibérica de Sistemas e Tecnologias de Informação 09–11.

Graham, C.R., & Robison, R. (2007). Realizing the transformational potential of blended learning: Comparing cases of transforming blends and enhancing blends in higher education. In A.G. Picciano & C.D. Dziuban (Eds.), Blended learning: Research perspectives. The Sloan Consortium.

Halverson, L.R., Graham, C., Spring, K.J., & Drysdale, S. (2012). An analysis of high impact scholarship and publication trends in blended learning. Distance Education, 33, 381-413, DOI:10.1080/01587919.2012.723166

Hooft Graafland, J. (2018). New technologies and 21st century children: Recent trends and outcomes.  OECD Education Working Papers, No. 179. OECD. https://dx.doi.org/10.1787/e071a505-en.

Jordan, P. (2016). How to define Millennials? Gen C Traveller.

Kennewell, S., & Beauchamp, G. (2007). The features of interactive whiteboards and their influence on learning. Learning, Media and Technology, 32(3), 227–241.

Lennon, M., Kirsch, I., Von Davier, M., Wagner, M., & Yamamoto, K. (2003). Feasibility Study for the PISA ICT Literacy Assessment: Report to Network, Educational Testing Service. https://eric.ed.gov/?id=ED504154

Livingston, K.,  & Condie, R. (2010). The impact of an online learning programme on teaching and learning strategies. Theory Into Practice, 45, 150-158. DOI:10.1207/ s15430421tip4502_7

Livingston, K. (2001). Disadvantaged teenagers and technology. Russell Sage Foundation.

Livingston, K., Soden, R., & Kirkwood, M. (2004). Post-16 pedagogy and thinking skills: An evaluation. Learning and Skills Research Centre.

Lorenceau, A., Marec, C., & Mostafa, T. (2019). Upgrading the ICT questionnaire items in PISA 2021. OECD Education Working Paper No. 202. http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=EDU/WKP(2019)10&docLanguage=En

Martins, J., Gonçalves, R., Oliveira,T., Cota, M., & Branco, F. (2016). Understanding the determinants of social network sites adoption at firm level: A mixed methodology approach. Electronic Commercial Research Applications 18, 10-26.

McGuinness, C. (1999). From thinking skills to thinking classrooms: A review and evaluation of approaches for developing pupils’ thinking (Research Brief No. 115). Her Majesty’s Stationery Office, Department for Education and Employment.

Moreira, F., Ferreira, M.J., Santos, C.P., Durão, N. (2016). Evolution and use of mobile devices in higher education: A case study in Portuguese higher education institutions between 2009/2010 and 2014/2015. Telematics and Informatics.

Mueller, J., Wood, E., Willoughby, T., Ross, C., & Specht, J. (2008). Identifying discriminating variables between teachers who fully integrate computers and teachers with limited integration. Computers & Education, 51(4), 1523–1537.

National Council of Mathematics (2000). Principles and standards for school mathematics. Author.

Newmann, F., Marks, H., & Gamoran, A. (1996). Authentic pedagogy and student performance. American Journal of Education, 104(4), 280–312.

OECD (2018). PISA 2021 Mathematics Framework (First Draft), EDU/PISA/GB(2018)4. 45th meeting of the PISA Governing Board, Directorate for Education and Skills, Programme for International Student Assessment. https://www.upc.smm.lt/naujienos/smm/penkiolikmeciumatematinis-rastingumas/GB-2018-4-PISA-2021-Mathematics-Framework-First-Draft.pdf

OECD (2018). PISA in Focus #83. https://www.oecdilibrary.org/docserver/1e912a10en.pdf?expires=1548775159&id=id&accname=ocid84004878&checksum=CD46091CFF6D EAE7985BF0545A3B5A88

OECD (2010). Inspired by technology, driven by pedagogy: A systemic approach to technology-based school innovations (Centre for Educational Research and Innovation).

OECD Publishing. http://www.oecd-ilibrary.org/education/ inspired-by-technology-driven-by-pedagogy_9789264094437-en

Picciano, A.G., & Seaman, J. (2009). K–12 online learning: A 2008 follow-up of the survey of U.S. school district administrators. Sloan Consortium. http://www.sloanconsortium.org/publications/survey/pdf/k-12_online_learning_2008.pdf

Rouse, C., Krueger, A., & Markman, L. (2004). Putting computerised instruction to the test: A randomized evaluation of a `scientifically-based' reading programme. NBER Working Paper, 10315.

Sälzer, C. (2021). Lesen im 21. Jahrhundert: Lesekompetenzen in einer digitalen Welt. Deutschlandspezifische Ergebnisse des PISA-Berichts „21st-century readers“.

Shala, A., Grajcevci, A., & Latifi, F. (2021). Does socioeconomic status influence achievement? An analysis of the performance of Kosovar students on the 2015 and 2018 PISA Assessment. Journal of Elementary Education, 14(4), 393-408.

Scherff, L., & Piazza, C. (2008). Why now, more than ever, we need to talk about opportunity to learn. International Literacy Association Wiley. http://dx.doi.org/10.2307/40058135.

Schmidt, W., & Maier, A. (2009). Opportunity to learn. In G. Sykes et al. (Eds.), Handbook of education policy research. Routledge.

Sherman, T., M., Sanders, M., & Kwon, H. (2010).Teaching in middle school technology education: A review of recent practices. International Journal of Technology and Design Education, 20, 367-379.

Smith, H.J., Higgins, S., Wall, K., & Miller, J. (2005). Interactive whiteboards: Boon or bandwagon? A critical review of the literature. Journal of Computer Assisted Learning, 21, 91–101.

Snape, P., & Fox-Turnbull, W. (2013). Perspectives of authenticity: Implementation in technology education. International Journal of Technology and Design Education, 23(1), 51-68.

Somekh, B. (2008). Factors affecting teachers’ pedagogical adoption of ICT. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 449–460). Springer Science+Business Media.

Spiezia, V. (2010). Does computer use increase educational achievements? Student-level evidence from PISA. OECD Journal: Economic Studies, 2010, 1-22.

Voogt, J., & Knezek, G. (2008). IT in primary and secondary education: Emerging issues. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 5-22). Springer Science+Business Media.

Authors:

Arif Shala teaches at AAB College, Prishtina, Kosovo. He holds a PhD in Education from Ludwig  Maximilian  University of Munich, Germany. His expertise is in educational  science, with a specific focus on 21st-century skills, student performance, learning in the new era and digital learning. Email: arif.shala@universitetiaab.com

Albulene Grajcevci teaches at the University Isa Boletini Mitrovice. She holds degrees from Ludwig Maximilian University of Munich, Germany and the University of Prishtina, Kosovo. Her expertise is in educational psychology, with a special focus on achievement goals, achievement emotions, organisational learning and computer enhanced learning. Email: albulene.grajcevci@umib.net

Cite this paper as: Shala, A., & Grajcevci, A. (2023). ICT and internet usage among Kosovar students: The impact of trends on achievement in the PISA scales. Journal of Learning for Development, 10(1), 122-133.