UNIT 1
Learning objectives
Understand the foundational concepts of Educational Technology (EdTech) and its role in transforming K-12 education by enhancing teaching methods and improving learning outcomes.
Analyze the significance of research, evidence, and impact assessments in evaluating the effectiveness and educational value of EdTech solutions.
Explore various evaluation methodologies such as randomized controlled trials, quasi-experimental designs, mixed-methods research, and case studies to assess EdTech innovations.
Develop the ability to identify and navigate challenges like equity, accessibility, and contextual appropriateness when implementing EdTech policies and strategies.
Introduction
In today’s digital age, educational technology (EdTech) has the potential to transform K-12 education by enhancing teaching methods, engaging students, and improving learning outcomes. Policy-makers need to understand how to effectively evaluate these technologies is crucial to making informed decisions that can drive educational improvements and allocate resources effectively. This unit will provide you with a foundational overview of EdTech, the significance of research and evidence in evaluating its impact, and the various approaches available for assessing educational technologies.
This terminology is taken from the Glossary developed by the International Centre for EdTech Impact researchers and collaborators, available from: https://www.foreduimpact.org/glossary
Glossary
-
Educational Technology (EdTech) refers to the use of technology to facilitate learning and improve educational processes. This encompasses a wide range of tools and applications, including online learning platforms, digital textbooks, interactive software, and educational apps designed to support teachers and engage students.
The Reimagining Education Through Technology is a landscape of over 300 EdTech innovations around the world, presented using an intuitive framework that classifies the Edtech solutions into nine teaching and learning interactions. The landscape was conducted by Central Square Foundation in October 2020 using publicly available information. While the landscape of EdTech solutions today would feature many newer products, the framework continues to be a key resource for understanding the different teaching and learning interactions being powered by technology.
Access the report and framework here
Access the database of EdTech innovations here
-
Research involves systematic investigation aimed at discovering and interpreting facts to increase knowledge about a specific area. In the context of EdTech, research helps us understand how different technologies influence student learning and engagement.
Impact refers to the mesurable effects that educational technology has on students, educators, and the overall educational system. This includes improved academic performance, increased student motivation, and enhanced teaching effectiveness.
Evidence is the data and findings derived from rigorous research. In EdTech evaluations, evidence helps determine whether a technology effectively achieves its intended outcomes.
-
Several approaches can be used to evaluate EdTech:
Experimental Design: This involves randomized controlled trials (RCTs) to compare the effectiveness of an EdTech intervention against a control group. This method is considered the gold standard for establishing causality.
Chimple is a game-based app for FLN, and provides teachers with a console to remotely assign homework and additional practice material in the form of simple & engaging activities. Chimple partnered with the Bharti Airtel Foundation to implement its teacher-led programme in a few districts in Haryana, with students in Grades 1 and 2 covering 34 schools. A study (RCT) was conducted to evaluate the effectiveness of a teacher-led at-home learning technology program. In this program, students accessed educational technology (EdTech) on their parents' smartphones, supplementing their in-school learning to enhance foundational literacy and numeracy (FLN) skills. Results indicated that 10 mins of Chimple usage per day over a nine-month intervention period was equivalent to one additional year of schooling.
Access the report here
Quasi-Experimental Design: When randomization isn’t feasible, quasi-experimental designs can be employed, such as using matched groups or pre-test/post-test comparisons to evaluate impact.
TheLiftEd EdTech Accelerator supporting the NIPUN Bharat mission supports eight high-quality EdTech solutions for two years. The Accelerator’s evidence generation agenda includes a learning outcome evaluation to assess the impact of EdTech partner interventions on student learning outcomes. The primary research questions of the study include: is the usage of the solution leading to an improvement in learning outcomes, what level of dosage leads to a given impact and at what cost is this impact achieved. The Research methodology is a quasi-experimental design (QED), having individual treatment groups and comparable control groups for three (Ei Mindspark, ThinkZone, Top Parent) EdTech solutions. The study adopts a difference-in-differences (D-i-D) approach to estimate the treatment-on-treated (ToT) effect of the EdTech solution. A modified version of the EGRA & EGMA tools, contextualised to India generated by Educational Initiatives (Ei) and the Central Square Foundation (CSF) administered through Ei’s “Science of Learning (SoL) app. Students in grades 2 and 3 across a few districts (based on the partner's existing deployment / user base) of Rajasthan, Odisha and Uttar Pradesh would be assessed as a part of the evaluation spanning over August 2024 to March 2025.
Mixed-Methods Research: This approach combines quantitative and qualitative methods to provide a more comprehensive understanding of how EdTech is used and its effects on learning.
TheLiftEd EdTech Accelerator supporting the NIPUN Bharat mission supports eight high-quality EdTech solutions for two years. The Accelerator’s evidence generation agenda includes (i) Learning Outcomes Evaluation – to assess the impact on student learning outcomes; (ii) Impact of Acceleration study – to capture the effectiveness of the strategies implemented within the Accelerator; and (iii) Insights on User Experience Study – a qualitative analysis that gathers feedback from end users on key aspects of the EdTech programme lifecycle, including acquisition, onboarding, engagement and retention. These evaluations being conducted under the supervision of a Principal Investigator aim to provide actionable insights to inform future interventions and improvements.
Learning outcomes evaluation has been described above in the quasi experimental design section.
The Impact of Acceleration study aims to extract actionable insights into the programme’s impact, focusing on three core objectives of i) )Nature of support provided - analysing the structure, relevance, intensity and alignment of the support offered with partner expectations and needs. ii) Organisational and operational outcomes: Identifying outcomes at both strategic and operational levels, including contributions to organisational direction, scaling initiatives, product improvements and the development of partnerships. iii) Best practices and learnings: highlighting Accelerator-wide best practices and partner-specific insights to refine and enhance future support packages for EdTech partners.
The User Experience (UX) study systematically explores the stages of EdTech programme lifecycle to identify best practices, challenges and contextual factors influencing outcomes. By analysing what works (or doesn’t) and understanding the reasons behind these results, the study aims to offer actionable insights for the EdTech ecosystem. The study employs qualitative methods, including in-depth interviews (IDIs) and focus group discussions (FGDs), to capture perspectives from a range of stakeholders—children, parents, teachers, and community resource persons—depending on the engagement model.
Case Studies: In-depth explorations of specific implementations can offer valuable insights into the contextual factors influencing EdTech effectiveness.
A device distribution program in Almora district of Himachal Pradesh (600 students in Grade 4 and Grade 8 spread across 83 government schools) was undertaken to identify best practices for implementation of a programme where dedicated devices are given. The project identified and consolidated the various levers required to create a device distribution model that would provide insights about EdTech usage at home using dedicated devices. The findings have been distilled into two parts; part I is an implementation playbook that provides insights into procurement, provisioning, distribution and maintenance of devices. Part II describes the programme and its components - implementation cycles, programme groups; which was deployed to encourage the use of EdTech on dedicated devices.
Access the report here - Part I, Part II
Surveys and Interviews: Collecting feedback from students, teachers, and administrators can provide qualitative data on the perceived effectiveness and usability of EdTech tools.
The Bharat Survey for Edtech is a household-level biennial survey that provides ground-up data and reliable insights from the low-income segment in India on their access to technology, attitudes and behaviours towards children’s education and adoption of technology for learning, and children’s use and experience with technology for learning. The first round of BaSE was conducted in 2022 across 6 states in India, covering both rural and urban settlements. It revealed interesting insights around children’s access to devices at a household level and their caregivers' sentiment with respect to using devices for education. These insights can be used by policy makers, EdTech innovators, practitioners, researchers and funders to encourage data-backed dialogue and decision making.
Read the BaSE report survey here.
Fact Box: Types of Research and Their Relevance for EdTech Evaluations
-
Relevance: RCTs provide robust evidence of causality by comparing outcomes between treatment and control groups. They are critical for understanding whether an EdTech tool directly improves learning outcomes.
-
Relevance: These studies are useful when RCTs are impractical. They help identify the effects of EdTech in real-world settings while controlling for other variables.
-
Relevance: Combining quantitative and qualitative data offers a holistic view of EdTech implementation and impact, providing insights into user experiences alongside statistical outcomes.
-
Relevance: Tracking the same students over time can help assess the long-term effects of EdTech on learning, engagement, and skill development.
-
Relevance: These provide detailed examinations of specific EdTech implementations, helping to identify best practices and challenges faced by educators and students.
A pupil-level RCT can be used to establish proof-of-concept for a new EdTech solution by randomising children within one school to engage with a new EdTech compared to standard instructional practice. This enables a rigorous test of an EdTech solution whilst controlling for external school-level factors influencing results. Pitchford (2015) adopted this approach to demonstrate proof-of-concept that a personalised learning EdTech – onecourse – significantly improved foundational numeracy compared to standard classroom instruction within one primary school in Malawi. The program is now being scaled nationwide.
Examples:
A quasi-experimental study was adopted by Pitchford and colleagues (2019) to assess if an EdTech solution enhanced foundational skills compared to standard practice in primary schools in Malawi. The EdTech solution was implemented in seven ‘intervention’ schools chosen by the Government of Malawi. To compare to standard practice, seven ‘control’ schools were recruited that did not have the Edtech solution, but were matched to the intervention schools in key demographics, including number of 1st- and 2nd-grade pupils, number of 1st- and 2nd-grade teachers, overall school size and geographical location, in an attempt to control for these variables influencing results.
A mixed-methods study was adopted by Outhwaite and colleagues (2019) to evaluate the impact of the implementation of an EdTech solution on learning outcomes. A two-phased approach was used that combined qualitative data with quantitative analyses to examine the causal relationships between intervention implementation and outcomes in the context of a maths app intervention previously evaluated in a RCT across 11 UK schools. Phase I identified four implementation themes from class observations which were then rated so they could be entered into a regression analysis in Phase II, to examine if the implementation themes predicted learning outcomes.
How policy makers in Malawi used research evidence to inform decisions to adopt EdTech to support foundational learning in all state primary schools
Case study:
In 1994, Malawi introduced free primary education, significantly increasing access to schooling. However, the country, one of the world’s poorest, faces a high learning poverty rate of 87%, meaning that the majority of children in grade 4 are unable to read. This learning gap is exacerbated by overcrowded classrooms, limited resources, and a shortage of trained teachers, all of which hinder foundational education in the country.
To address these challenges, the Government of Malawi, in partnership with the non-profit organizations Imagine Worldwide and onebillion, has introduced an innovative EdTech designed to improve literacy and numeracy skills. This EdTech uses personalized, adaptive software – onecourse – on tablets, offering children in grades 1-4 engaging activities, games, and stories tailored to their individual learning levels.
In 2013, a randomized control trial demonstrated the effectiveness of this technology. Children using the software for just 30 minutes a day for 12 weeks showed significant improvements in numeracy compared to traditional classroom methods. Further research confirmed that the EdTech program boosted literacy and numeracy skills for both boys and girls and for children with special educational needs, and was effective in diverse contexts, including bilingual and refugee settings.
Rigorous, ongoing studies have shown that the EdTech not only improves academic outcomes but also increases children’s excitement about learning and enhances school attendance and learners’ confidence. The program has even helped mitigate learning loss during COVID-19 school closures. The program is now being implemented nationwide, with plans to serve 3.8 million children annually across all 6,000 state primary schools in Malawi.
Informed by strong research evidence, the Government of Malawi officially launched the nationwide rollout of the program in 2022. The program exemplifies how EdTech can help eliminate learning poverty and improve educational outcomes for marginalized children globally. The hope is that other governments will follow Malawi’s lead in making foundational education a reality for all children.
Further information about this case study can be found here.
