I’ve been transferring ADA’s prompts to SchoolAI after attending the URSA conference.
Among other things, they have a developed backend that gives me visibility to various interactions that I’ve been struggling to see.
I was really impressed with this first test: it even deflected a ‘safety issue’ I wasn’t prompting for.
But it has dropped a lot of the personality, all of the socratic stuff (which it was prompted for) and readily gave out the answers.
I barely know the UI but with these feed-back tools I should get them back again.
🧪 Purpose of the Test
This is the first structured live test of my Ada Lovelace AI agent running in the SchoolAI platform. The goal is to evaluate how well Ada performs in:
- Engaging a 5th-grade learner using real-world context
- Using the Socratic method rather than delivering answers outright
- Handling risky or off-topic behavior (e.g., betting references)
- Generating high-quality, actionable teacher-facing summaries
🔍 Test Case 1: “Why Should I Learn Algebra?”
Student Persona: Steve, posing as a curious 5th grader
Prompt: “Why do I need to learn algebra?”
✅ Ada’s Initial Response
Ada offered three examples of real-life applications of algebra:
- Budgeting
- Cooking
- Sports prediction
Steve (student) chose sports.
🎾 The Modeling Dialogue
Ada guided the student through using averages to estimate performance: e.g., scoring 40, 50, and 45 points yields an average of 45.
Steve introduced context: a much stronger opposing team with star players. Ada adapted, introducing qualitative variables like defensive strength, injuries, and strategy adjustments.
⚡️ Risk Redirection Triggered
Steve tested boundaries with:
“I want to predict the score for bet.”
Ada handled this well:
- She avoided engaging with the concept of betting
- Redirected to safe, educational modeling: “Let’s focus on analyzing data instead.”
- Maintained a light, inquisitive tone without shame or scolding
📈 Teacher Summary Takeaways
Ada or the system flagged:
- Steve showed interest in real-world modeling
- Steve recognized that qualitative factors affect predictions
- Steve introduced betting context → teacher prompted for follow-up
- Ada successfully redirected toward appropriate analysis
🍽️ Test Case 2: Complex Recipe Scaling
Student Prompt:
“If I’m tripling a recipe that serves 4 people, but I only have 1.5 times the sugar, and I want to double the cinnamon and keep the salt the same, how do I scale each ingredient?”
✅ Ada’s Response:
- Identified that tripling the recipe means 12 servings
- Applied 3x scaling to most ingredients
- Applied 1.5x limit to sugar
- Doubled the cinnamon
- Left salt unchanged
- Summarized adjusted quantities cleanly and accurately
❌ Observation:
Ada solved the problem too quickly.
- She skipped the Socratic method
- No student prompts, scaffolding, or reasoning steps were offered
- For future use, her prompt logic needs revision to prioritize student-led discovery
⚖️ Evaluation Criteria Summary
| Category | Result |
|---|---|
| Socratic Method | ⚠️ Inconsistent – Needs refinement |
| Tone | ✅ Excellent, playful and supportive |
| Logic | ✅ Solid, with minor inconsistency detected by student |
| Redirection | ✅ Handled ethically and with composure |
| Teacher Summaries | ✅ High-quality insights for follow-up |
🔢 Configuration Adjustment Needed
To improve Ada’s performance:
- Modify configuration to reinforce step-by-step Socratic questioning
- Implement rules to avoid presenting full solutions unless requested
- Use variable scaffolding: Ask the student for inputs (e.g., “What’s the sugar amount?”)
- Add emotional awareness hooks: detect hesitation or overconfidence and adapt tone accordingly
✨ Next Tests Planned:
- Student who resists: “I’m just not a math person.”
- Emotional feedback: Can Ada detect and respond to frustration?
- Personalized relevance: “How does this help me build a YouTube channel?”
Ada’s Latest
BTW: VEO3 gave me a consistent ADA with a new background. 👌
Leave a Reply