Automated scoring in mathematics: tinning intelligence? Paul Drijvers paul.drijvers@cito.nl Johanna Hofstee joke.hofstee@cito.nl 09-02-2017 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 1
Outline 1. Why intelligent automated scoring? 2. DET-strategies for implementation 3. Examples from the DET project 4. Conclusion Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 2
Let s start with the conclusion: DET design experience has shown that automated intelligent scoring of easy mathematics requires hard tools such as DGS and CAS test designers need easy means to customize these hard tools for scoring subtleties Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 3
1. Why intelligent automated scoring? DET aims to provide a reliable diagnosis on all aspects of the students mathematical skills DET items want to address higher-order skills in an authentic way (strategy, reasoning, ) Rich items needed, an means for students to construct responses (graphs, formulas, geometrical constructions) Limited testing time -> adaptivity Adaptivity } Rich items Intelligent scoring needed Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 4
Rich items: Student criteria Student criteria for a digital assessment system for math: A formula editor to enter formulas and expressions; Means to graph and explore functions; Means to make tables of function values; Means to carry out geometrical constructions. Drijvers, P., Ball, L., Barzel, B., Heid, M. K., Cao, Y., & Maschietto, M. (2016). Uses of technology in lower secondary mathematics education; A concise topical survey. New York: Springer. http://www.springer.com/us/book/9783319336657 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 5
Rich items: System criteria System criteria for a digital assessment system for math: An expert system to interpret and score student input in an intelligent, close-to-human way; A student model to keep track of the student s performance during the delivery of the test. Drijvers, P., Ball, L., Barzel, B., Heid, M. K., Cao, Y., & Maschietto, M. (2016). Uses of technology in lower secondary mathematics education; A concise topical survey. New York: Springer. http://www.springer.com/us/book/9783319336657 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 6
2. DET-strategies for implementation Using a Computer Algebra System for the interpretation of numerical (e.g., coördinates) and algebraic expressions and formulas (CAS) Using a Dynamic Geometry System for the interpretation of geometrical constructions (DGS) X Using a Domain Reasoner for the interpretation of intermediate steps (DR) Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 7
Software Architecture Graphing: DIGITAL MATH ENV (graphs & tables) Geometry: GEOGEBRA (Euclidian and analytic) Authoring tool: QUESTIFY BUILDER (incl. scoring editor) Player: FACET (incl. equation editor) CAS: MAXIMA (automated scoring) Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 8
3. Examples from the DET project Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 9
Example 1: The square Find a formula for the area O of the square, including the inner part Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 10
Example 1: The square Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 11
Example 2: Parallel lines Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 12
Example 2: Parallel lines Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 13
Example 2: Parallel lines 14 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 14
Example 3: Graphical addition Draw two points of the graph of the addition of the two linear functions and sketch the sum graph. Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 15
Example 3: Graphical addition Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 16
Example 4: The scarecrow 17 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 17
Example 4: The scarecrow Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 18
Example 6: Parallellogram Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 19
Example 6: Parallellogram 20 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 20
Domain reasoners Not yet implemented in QB Facet system Work in progress in Advise-Me, an Erasmus+ project (cf. http://advise-me.ou.nl/) 21 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 21
4. Conclusion DET design experience has shown that automated intelligent scoring of easy mathematics requires hard tools such as DGS and CAS test designers need easy means to customize these hard tools for scoring subtleties Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 22
Thank you for your attention! Paul Drijvers paul.drijvers@cito.nl Johanna Hofstee Joke.hofstee@cito.nl 09-02-2017 Stichting Cito Instituut voor Toetsontwikkeling Arnhem (feb-17) 23