UNIT 2
Introduction
In Unit 2, we dive deeper into the mechanisms of evaluating educational technology, focusing on the critical role of teacher involvement and the importance of collaborative approaches to ensure that edtech is not user-friendly, evidence-based and business sustainable. This unit thus equips policy-makers with practical insights into designing, testing, and scaling EdTech solutions within K-12 systems.
Learning objectives
Understand the importance of testbeds and teacher involvement in the testing and evaluation of EdTech solutions to ensure practicality, usability, and alignment with educational needs.
Differentiate between co-design and randomized controlled trials (RCTs) as approaches to developing and evaluating EdTech solutions, and determine their appropriate applications.
Apply foundational concepts like logic models and theories of change to plan and evaluate EdTech initiatives effectively, connecting inputs and activities to long-term outcomes.
Recognize the distinction between outputs and outcomes, and learn how to measure immediate results and broader impacts of EdTech interventions.
Testbeds and Involvement of Teachers
Testbeds refer to controlled environments or schools where EdTech products are tested in real-world classroom settings. They offer a structured way to evaluate technology’s performance and impact while incorporating valuable feedback from teachers and students.
Why Are Testbeds Important?
They allow for practical testing of technology in a real learning environment.
Teachers play a critical role by providing feedback on how EdTech fits within existing curricula and pedagogical approaches.
Testbeds help identify potential issues early, ensuring the technology is tailored to the actual needs of students and educators before full-scale implementation.
Involvement of Teachers in EdTech Testing
Teachers are at the frontline of EdTech implementation. Their direct involvement ensures the tools are both practical and effective in enhancing learning outcomes.
Collaborating with teachers during the testing phase helps in understanding classroom dynamics, improving the user interface, and aligning content with curriculum requirements.
Teacher feedback ensures the technology is not only innovative but also addresses real educational challenges and improves student engagement.
Example
The Global EdTech Testbed Network, as highlighted on the GlobalEdTech Testbed Network website, is an initiative that partners with schools to rigorously test EdTech solutions. Teachers are key constituents in this process, ensuring that products are fine-tuned to meet classroom needs and achieve measurable educational outcomes.
Chimple is a game-based app for FLN, and provides teachers with a console to remotely assign homework and additional practice material in the form of simple & engaging activities.
Chimple partnered with the Bharti Airtel Foundation to implement its teacher-led programme in a few districts in Haryana, with students in Grades 1 and 2 covering 34 schools. Prior to the RCT, various programme aspects were tested and iterated - model discovery with iterative program design, uncovering strategies of teacher engagement, behavioural nudges, monitoring of student and teacher data, product enhancement based on feedback from the user studies led to the resultant stabilization of the model which was then evaluated. Access the RCT report here.
Sandbox approach molly@edtechhub.org
Sandboxes support implementers to test and iterate interventions throughout implementation. They provide a space for partners - governments, startups, NGOs, grassroots organisations, researchers and implementers - to test and grow ideas in conditions of uncertainty.
We define boundaries in real-world environments that will offer us a mini version of the whole education system in a country. This allows us to test specific components of a model and use what is learnt to define the next step while limiting implications and damage to the broader system.
We can test and trial all parts of the system - technology products, pedagogical approaches, and even ways of funding, or policy to understand what works, when, and in what combination. For example, we might test a new intervention in a small number of schools first, improving its cost per user through experimental sprints, increasing its impact by supporting the organisation with pedagogical expertise or even trialling new ways of procurement, at a small scale, before rolling it out. The idea is that we increase our scale in line with confidence that it works.
Use an iterative, experimental approach in real time, applying the level of rigour required (no more, no less) to allow decision-makers to test and iterate their model in low-cost ways and scale only that which has evidence of cost-effective impact
We bring together different actors, connecting theory and practice, private and public sector, various funders and innovators with governments, and entrepreneurs with academics and the people impacted by the work.
Co-Design Versus RCT Approaches
Co-Design involves a collaborative process where developers, researchers, and educators work together to design EdTech tools that directly meet the needs of students and teachers. In contrast, Randomized Controlled Trials (RCTs) use scientific experimentation to determine the effectiveness of an EdTech tool by comparing it with alternative approaches, such as for example traditional teaching methods.
Co-Design:
Collaborative Approach: Developers work closely with teachers and students to ensure the product meets educational needs and fits well into teaching practices.
Flexibility: The tool evolves through ongoing feedback, allowing it to be tailored during the development process.
Teacher Empowerment: Teachers are co-creators, ensuring the EdTech product aligns with classroom realities.
RCT Approach:
Scientific Rigor: RCTs offer a high standard of evidence by using control and intervention groups to test the effectiveness of EdTech tools.
Limited Flexibility: The design of the tool is often fixed before testing begins, limiting the possibility for adaptation based on real-time feedback.
Proven Impact: If successful, RCTs can offer clear, quantifiable evidence that a technology improves student learning outcomes.
Example
A pupil-level RCT can be used to establish proof-of-concept for a new EdTech solution by randomising children within one school to engage with a new EdTech compared to standard instructional practice. This enables a rigorous test of an EdTech solution whilst controlling for external school-level factors influencing results. Pitchford (2015) adopted this approach to demonstrate proof-of-concept that a personalised learning EdTech – onecourse – significantly improved foundational numeracy compared to standard classroom instruction within one primary school in Malawi. The program is now being scaled nationwide.
Which approach to use?
Co-design is ideal for the early stages of product development, ensuring the EdTech tool is aligned with user needs.
RCTs are useful for later stages, providing solid evidence of a product's effectiveness once it is fully developed.
The Basics of Logic Model, Theory of Change, and the Difference Between Outputs and Outcomes
Logic Model
A Logic Model is a visual representation that outlines the inputs, activities, outputs, and outcomes of an EdTech project. It helps stakeholders understand how the project is expected to work and achieve its goals.
Inputs: Resources needed to implement the project (e.g., funding, technology, training).
Activities:
Actions taken to develop and deliver the product (e.g., app development, teacher training).
Outputs: Immediate results from the activities (e.g., number of students using the app).
Outcomes: Long-term impact on learning (e.g., improved reading skills).
Theory of Change
A Theory of Change goes beyond the logic model by explaining how and why a desired change is expected to happen in a particular context. It outlines the assumptions, conditions, and pathways through which an EdTech intervention leads to improved educational outcomes.
It helps in identifying the causal mechanisms behind how an EdTech product improves learning.
It ensures all stakeholders understand the big picture and the rationale behind each stage of the project.
Outputs Versus Outcomes
Outputs: These are immediate, measurable products of the intervention (e.g., how many students are using the technology or how many lessons were completed).
Outcomes: These are the broader, longer-term changes (e.g., improvements in student achievement or engagement with the subject matter). Outcomes reflect the true impact of the EdTech solution.
Fact Box: Key Takeaways
-
Testbeds offer a practical approach to testing EdTech tools in real classroom settings, with valuable feedback from teachers and students.
-
Co-design fosters collaboration and flexibility, ensuring that EdTech products are directly aligned with the needs of schools, while RCTs provide rigorous evidence of effectiveness.
-
Logic Models and Theory of Change are essential tools for planning and evaluating EdTech projects, helping to clarify the pathway from inputs to long-term outcomes.
-
Understanding the difference between outputs and outcomes is crucial for measuring both immediate results and long-term impact in EdTech evaluations.
