The AI Analyzer

Advancing trustworthy, transparent, and scalable evaluation of EdTech evidence

The AI Evidence Analyzer is a specialized tool that systematically evaluates academic research and rates the quality of evidence using the EVER (Evidence Evaluation and Rating) framework developed by Professor Natalia Kucirkova and research teams at Harvard University and the Leibniz Institute for Research and Information.

Built with advanced natural language processing and machine-learning models trained specifically on research data, the Analyzer extracts relevant information from academic articles, classifies study characteristics, and produces structured, transparent, and replicable ratings. This enables researchers, investors, educators, and policymakers to make informed, evidence-based decisions with confidence and clarity.

The Analyzer and its companion chatbot were developed by the International Centre for EdTech Impact for Eduevidence.org, with support from researchers at the University of Oxford and funding from the Norwegian government through Innovation Norway

10_MARKETPLACE_SHOT_5_0862_02_QC.jpeg
02_CIRCLE_SHOT_3_0307_04_QC.jpeg
04_CIRCLE_SHOT_4_0422_03_QC.jpeg
03_CIRCLE_SHOT_5_0351_04_QC.jpg

What the Analyzer Evaluates

Using the EVER framework, the tool reviews research across multiple dimensions of evidence quality. Each article is analyzed and scored using a structured set of criteria, ensuring consistency across studies and enabling clear comparisons for users .

Sections of the evaluation include:

- Study characteristics

- Methods and design quality

- Outcome measurement

- Contextual relevance

- Reporting transparency

- Overall evidence rating

Users receive a clear breakdown of criteria → results, including examples, structured summaries, and justification text generated from the article.

What are the

Chatbot advantages

  • Transparent – Evaluation results are visible to users.

  • Tunable – Retrieval can prioritize high-maturity or externally validated studies.

  • Traceable – Claims are always backed by both paper text and structured judgments.

Use cases

  • For Companies. Evidence strength summaries for specific EdTech solutions Benchmarking against competitors Insights for product improvement and validation.

  • For Policymakers. Context-specific evidence landscapes Support for funding, procurement, and implementation decisions Country- or sector-specific policy recommendations

  • For Educators & School Leaders. Clarity on what works for specific outcomes (e.g., literacy, math) Comparisons of tools within their context Reliable summaries of research quality

  • For Researchers & Funders. Systematic evaluation of study quality Identification of gaps and opportunities Support for evidence synthesis and meta-analysis.

How It Works: The Analysis Pipeline

  • The Analyzer follows a multi-stage pipeline to evaluate research articles AI EduEvidence analyser:

    Document ingestion & text extraction

    AI-based classification of study characteristics

    Application of EVER evidence-rating criteria

    Generation of structured results and explanations

    Document annotation showing where evidence was identified

    Human validation & feedback to strengthen model performance over time

    Human oversight is a core principle: expert feedback continuously improves the system, and the tool is intentionally not built on generic LLMs that are not trained for research evidence quality assessment

  • Guiding Principles

    The Analyzer is built around three pillars of trustworthy evidence evaluation:

    Transparency – Full visibility into how judgments are made, including annotated texts and reasoning.

    Consistency – Standardized evidence criteria ensure reliable comparisons across studies, solutions, and contexts.

    Traceability – Every finding is linked to source text and structured evaluative logic.

  • EVER Criteria: Example Output

    A typical evaluation includes:

    A structured summary of strengths and limitations

    Study metadata and contextual information

    Examples of extracted text supporting the rating

    Item-level results across each EVER category

  • Item-Level View and Document Annotations

    The Analyzer presents detailed, line-by-line scoring for each component of the EVER framework, accompanied by AI-generated justifications and in-document highlights showing the evidence used to make each judgment. This gives users a transparent, audit-ready evaluation of every study

  • Integrated Chatbot for Evidence Queries

    The Analyzer includes a research-aware chatbot that retrieves both raw papers and evaluated summaries, enabling users to ask targeted questions such as:

    “What evidence supports EdTech Company X’s impact on math learning outcomes?”
    Example response:
    “Company X has moderate-quality evidence supporting its impact on math outcomes. Two studies rated as high-quality found significant improvements in standardized math tests (Evidence Quality: 4/5), while another medium-quality study showed more limited improvements (Evidence Quality: 3/5).”

Want access to the analyzer?

Get in touch!