Meagher-Lundberg, P. (2000). Comparison variables useful to teachers in analysing assessment results. (Tech. Rep. No. 1). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises views of focus groups of teachers about the types of comparisons they wish to make when making use of assessment results. The main types were: school decile, authority; and geographical location; gender; ethnicity, age group, Year level, and language background of the student; and the curriculum level(s) the student was at.
Zwiegelaar, J. (2000). Evaluation of the first trials of reading and writing
assessments. (Tech. Rep. No. 2). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises feedback responses of the teachers to the first asTTle trial literacy assessment papers conducted in October 2000. Feedback was generally positive across both reading and writing and from both teachers and students
Langstaff, J. (2000). Evaluation of the second trial of reading and writing assessment papers. (Tech. Rep. No. 3). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises the feedback responses of the teachers and students to the asTTle trial literacy assessment papers conducted in October 2000. Feedback was generally positive across both reading and writing and from both teachers and students.
Limbrick, L., Keenan, J., & Girven, A. (2000). Mapping the English curriculum. (Tech. Rep. No. 4). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This study aimed to map the written language strand of English in the New Zealand Curriculum, with the purpose of establishing profiles of the Years 5, 6, and 7 students working within Levels 2, 3, and 4. A detailed set of Levels 2, 3, and 4 descriptors/ indicators was developed based on focus groups of expert teachers from Years 5, 6, and 7. There was much variability in the expectation of student achievement within each level of the curriculum and the teachers noted the importance of context, purpose of instruction, prior classroom experiences, and the differing processes and strategies that students can use to read and write. The importance of clear specifications of the levels, particularly in expressive writing and poetic writing was noted, as were the future studies to assist NZ teachers to have a better understanding of the functions and processes of English in the New Zealand Curriculum and their purpose. This is a debate that is valuable and well overdue given that English in the New Zealand Curriculum has been gazetted for six years.
Fletcher, R. B. (2000). A review of linear programming and its application to the Assessment Tools for Teaching and Learning (asTTle) Projects. (Tech. Rep. No. 5). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report reviews the international research literature on linear programming as applied to the issues of banking assessment items. It outlines the mathematical procedures needed to obtain feasible solutions to selections made by teachers and constraints imposed by the assessment developers. The various algorithms and heuristic procedures necessary for feasible solutions in an item bank of only 500 items with testlets are discussed and exemplified. The report recommends use of detailed item mapping, limiting the number of ability levels, use of the simultaneous selection of items and sets method, use of the maxim in model, and use of the optimal rounding method in finding solutions.
Glasswell, K., Parr, J., & Aikman, M. (2001). Development of the asTTle writing assessment rubrics for scoring extended writing tasks. (Tech. Rep. No. 6). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report outlines an assessment procedure for extended texts based on "Writing" components of the English Curriculum. It reviews recent research on scoring such texts, explores the notions of genre and purpose, notes difficulties for New Zealand teachers in the current documents available to them, and defends the argument that writing is a social act and the purpose or function of the writing is defined by the context. This means that the form and features of the text may differ according to purpose or that the same form (such as a letter) can serve different purposes. Thus, the scoring rubrics are aligned with the function of the writing, and communication and social purpose are overarching concepts. Rubrics are developed for each of these major functions, each containing three meta-divisions from which to consider the features of text as related to the purpose and context: rhetorical (audience awareness/ purpose); organisational/structural (content inclusion; coherence – sequencing ideas and linking; and language resources for achieving purpose), and conventional: sentences and words (grammatical conventions, spelling, and punctuation).
Irving, S. E. (2001). Report on teacher feedback from the first calibration of reading and writing assessments. (Tech. Rep. No. 7). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises feedback responses of the teachers involved in the first nationally representative calibration of asTTle reading and writing papers in November 2000. Both teacher and student responses are positive.
Parker, A. M. (2001). Teacher feedback from the linking calibration of reading and writing assessments. (Tech. Rep. No. 8). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report compiles the feedback of the teachers who participated in the link calibration of Reading and Writing materials in March 2001. Feedback is generally positive and constructive.
Meagher-Lundberg, P. (2001). Output reporting design: Focus group 1. (Tech. Rep. No. 9). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises findings from the first qualitative focus group of teachers who reacted to initial draft outputs designed to report assessment results to teachers. Information gained from this study was used to redesign and improve the communication of the output reports.
Meagher-Lundberg, P. (2001). Output reporting design: Focus group 2. (Tech. Rep. No. 10). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises findings from the second qualitative focus group of teachers who reacted to second draft outputs designed to report assessment results to teachers. Information gained from this study was used to further redesign and improve the communication of the output reports.
Ell, F. (2001). Mathematics in the New Zealand Curriculum - A concept map of the curriculum document. (Tech. Rep. No. 11). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report analyses the Mathematics in the New Zealand Curriculum document and maps the strands and achievement objectives of Levels 2 to 4 into eight major categories (i.e., understanding number, computing and estimating, time, metric measurement, shape and space, transformation and symmetry, probability, and understanding statistics). Each category has one to three subcategories, giving a total of 13 subdivisions of mathematics. A full discussion and rationale for this repackaging of the curriculum is provided. A second analysis of the curriculum by cognitive process is outlined by curriculum levels. This identifies the achievement objectives at each level by the type of activity required to elicit evidence of attainment.
Meagher-Lundberg, P., & Brown, G. T. L. (2001). Item signature study: Report on the characteristics of reading texts and items from calibration 1. (Tech. Rep. No. 12). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: Items and texts used in the first calibration of asTTle reading assessment were classified in an item signature study to establish their literacy-relevant characteristics. Classification of texts and items is used to inform the reporting of student achievement and to guide development of future assessment materials. A focus group of teachers determined text properties (i.e., curriculum level, form, purpose, genre, considerateness, and essential learning area) and item properties (i.e., cognitive and reading processes, and curriculum processes and objectives) on a two-thirds majority basis. Agreement among teachers was greater for classifying reading texts than assessment items. However, agreement levels were only moderate, ranging mostly between r = .40 and r = .70. Reading texts were found to be largely in Level 2 and 3 of the New Zealand English curriculum and equally divided between literary and information purposes. Assessment items required predominantly surface cognitive processes, retrieval of information comprehension processes, and gaining understanding from text curriculum processes.
Fairhall, U., & Keegan, P. J. (2001). Pängarau curriculum framework and map: Levels 2-4. (Tech. Rep. No. 13). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report details the Level 2-4 asTTle Pängarau Framework and provides a detailed breakdown of curriculum achievement objectives by curriculum level. asTTle is funded by the Ministry of Education to Auckland UniServices at the University of Auckland to research and develop an assessment application for Reading, Writing, Mathematics, Pänui, Pängarau, and Tuhituhi for Years 5-7 (Levels 2-4) for New Zealand schools. We acknowledge this funding, and thank the Ministry of Education for their continued assistance in the development of this project. This report is the first in a series of reports on the Pängarau Curriculum to which a wide range of curriculum experts have responded, whom I would also like to thank for readily providing feedback and analysis.
Hattie, J. A. (2002). Schools like mine: Cluster analysis of New Zealand schools. (Tech. Rep. No. 14). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises a procedure and rationale developed to allow comparison of student achievement in literacy and numeracy by type of school attended. The power of socio-economic status to describe schools is analysed and found inadequate. The clustering procedure uses school size, student ethnicity, region, size of civil jurisdiction, and socio-economic status to group schools for comparison. All schools are placed into one of twenty clusters that are used in the asTTle software as a basis of student achievement comparison.
Brown, G. T. L. (2001). Reporting assessment information to teachers: Report of Project asTTle outputs design. (Tech. Rep. No. 15). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report was prepared to aid the design of Project asTTle outputs for reporting assessment information to teachers. Is sues in graphical communication, user-interface preferences, and communication of statistical information to parents, administrators, and teachers were reviewed. The report overviews the three major output reports that will communicate student achievement in literacy and numeracy in the forthcoming Project asTTle CD-ROM and identifies how the issues raised are addressed.
Meagher-Lundberg, P., & Brown, G. T. L. (2001). Item signature study: Report on the characteristics of reading texts and items from calibration 2. (Tech. Rep. No. 16). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This is the report of the second item signature study conducted with Project asTTle reading assessment items. Six practising teachers determined the cognitive and curriculum characteristics of 222 assessment items and the curriculum and structural characteristics of 33 reading texts. Ratings were conducted in whole group, individual, and pair conditions, with the pair condition providing excellent dependability (f>0.80) and efficiency. The improved inter-rater agreement from the first item signature study (Tech. Rep. 12) can be attributed to revised training and supervision procedures.
Ell, F. (2002). Strategies and thinking about number in children aged 9-11 years (Tech. Rep. No. 17). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report is a review of the international literature on number knowledge, number strategies, and frameworks for classifying children's (aged 9-11) learning of number. Research into children's thinking in these areas suggests a general progression from concrete thinking tied to physical models and counting methods, to abstract thinking using known number facts and relationships. Using information about children's thinking in teacher professional development has resulted in increased effectiveness in teaching. Although the suggested progressions are not developmental stages in a formal sense, they can guide instructional sequences and help teachers to decide ‘where to next' for a child.
van Gardaren, D. (2002). Paper & pencil strategy assessment items for the New Zealand Number Framework: Pilot study. (Tech. Rep. No. 18). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises results of a pilot study, using student focus groups (Years 5 to 7), of paper and pencil strategy assessment items that had been written to elicit information about student use of mental strategies as defined by the New Zealand Number Framework. Significant difficulties were encountered in mapping both items and responses to the Number Framework; however, about half of the items seemed to be effective in providing information from students as to the mental strategy they had used in answering the questions. Recommendations are made to guide future item development.
Parker, A. M., & Brown, G.T.L. (2001). Teacher evaluation of the second calibration of reading and writing assessments: Summary (Tech. Rep. No. 19). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises feedback responses of the teachers (n=1 20) involved in the second nationally representative calibration of asTTle reading and writing papers in May 2001. The majority of teacher and student responses to the reading and writing assessments is positive, though more negative comment is expressed about the writing papers.
Hattie, J. C., Brown, G. T. L., & Keegan, P. J. (embargoed). Narrative requirements document: Project asTTle CD ROM. (Tech. Rep. No. 20). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report provides a narrative description of the functional requirements of the Project asTTle software. It describes the purpose of the software, its implementation context, and details the eight major functional requirements of the software. These requirements have to do with its installation and control, as well as processes for the inputting of data, test creation, test scoring, analysis of student achievement, recording and reporting of student achievement. In addition, the report describes other features required such as linkage to professional instructional resources, and required documentation. These requirements are the basis for the development of software requirement specifications.
Hattie, J. C., & Brown, G. T. L. (2003). Standard setting for asTTle reading: A comparison of methods. (Tech. Rep. No. 21). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report provides an explanation of the standard setting for asTTle reading. asTTle is funded by the Ministry of Education to Auckland Uniservices Ltd. at the University of Auckland to research and develop an assessment application for Reading, Writing, Mathematics, Pänui, Pängarau, and Tuhituhi for Years 5-7 (Levels 2-4) for New Zealand schools. We acknowledge this funding, and thank the Ministry of Education for their continued assistance in the development of this project. Four different standard setting methods were used in setting curriculum level cut scores for asTTle assessments developed to provide performance measures of close reading in the English curriculum. The report contrasts the four methods (examinee-centred, item-centred, test-centred, performance threshold-centred) and describes how the four methods were reconciled. It concludes with recommendations for future standard setting and demonstrates the robustness of the asTTle curriculum level scores.
Hattie, J.A.C., Brown, G.T.L., Keegan, P., Irving, S.E., MacKay, A.J., Sutherland, T., Mooyman, D., & Patel, P. (2003, November). Validation Evidence of asTTle Reading Assessment Results: Norms and Criteria. asTTle Tech. Rep. 22, University of Auckland/Ministry of Education.
Abstract: This report presents an analysis of arguments about assessment that contribute to the validity of asTTle reading comprehension scale and curriculum level scores. These principles relate to the importance of curriculum-linked interpretations, detailed as opposed to general analysis of achievement, the weakness of age or year related normative score interpretation, and methods for establishing legitimate expectations of performance. Furthermore, analysis of a small set of school data shows that the asTTle Surface Cognitive Processing and Finding Information close reading scores provide similar results to generalised measures of reading comprehension and teacher judgements of achievement. However, the richness of asTTle achievement results provide meaningful insight into teachable objectives that students still need to master, notwithstanding overall high norm-referenced ranking on generalised ability measures. It is argued that the current data provide a robust evidence for the validation of the asTTle reading comprehension scores.
Murphy, H., & Keegan, P. J. (2002). Te Reo Mäori literacy curriculum map. Levels 2-4 (Tech. Rep. No. 23). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report details the Level 2-4 asTTle Te Reo Maori Literacy Curriculum Map and provides a detailed breakdown of curriculum achievement objectives by curriculum level. It identifies the rich or big ideas underlying the learning intentions of the Te Reo Maori curriculum. This map was used in the design of Panui and Tuhituhi asTTle tasks for Köeke 2—4 trialled in Maori medium environments during 2001—2002 and provides the curriculum basis for asTTle reports of student performance in asTTle. This report is the first in a series of reports on the Te Reo Maori Curriculum.
Lavery, L., & Brown, G. T. L. (2002). Report on teacher feedback from the first calibration of mathematics assessments (Tech. Rep. No. 24). Auckland, NZ: University of Auckland, Project asTTle.
Abstract This report summarises feedback responses of the teachers (n=243) involved in the first nationally representative calibration of asTTle mathematics papers in November 2001. Feedback was generally positive and from both teachers and students.
Thomas, G., Tagg, A., Holton, D., & Brown, G.T.L. (2002). Numeracy item signature study: A theoretically derived basis. (Tech. Rep. No. 25). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: Items used in the first calibration of the asTTle numeracy assessment were classified in an item signature study to establish their relevant characteristics. A group of expert teachers determined item properties on a two-thirds majority basis. Properties assigned included cognitive processing, curriculum level, achievement objectives, and processes, and considerateness of text and diagrams. Agreement among teachers was very high in all categories, ranging from 99% for text considerateness to 76% for level/position within level, with an overall percentage agreement of 92%.
Glasswell, K., & Brown, G. T. L. (2003). Accuracy in the scoring of writing: Study in large-scale scoring of asTTle writing assessments. (Tech. Rep. No. 26). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report reviews the work of a four-day asTTle writing marking panel in which six writing tasks were scored by 17 markers. The processes used to ensure reliable scoring, and the degree of accuracy obtained through those procedures are reported. An average 75% exact score marking and an average dependability of .77 were found. These levels of agreement provide sufficient basis for having confidence in the norms underlying the asTTle writing assessments for use in low-stakes classroom assessment. The processes also provide guidance to schools as to how they can conduct high quality school-based assessment of writing.
Keegan, P. J., & Pipi, A. (2002). Summary of the teacher feedback from the calibration of asTTle v2 pänui, pängarau and tuhituhi assessments (Tech. Rep. No. 27). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report details the summary of the teacher feedback from the calibration (trials) of the asTTle v2 panui, pangarau, and tuhituhi assessments. It has been modeled on earlier reports describing lessons learned from the trialling of English assessments in reading, mathematics, and writing.
Brown, G. T. L. (2002). Item signature study: Report on the characteristics of reading texts and items from calibration 3 (Tech. Rep. No. 28). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This is the report of the third item signature study conducted with Project asTTle reading assessment items. Eight practising teachers determined the cognitive and curriculum characteristics of 76 assessment items and the curriculum and structural characteristics of 10 reading texts. Ratings were conducted in pairs, providing excellent dependability (f>0.80) and efficiency.
Harland, D. (2003). Using asTTle persuasive writing: A case study of teaching argument writing (Tech. Rep. No. 29). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report outlines a case study in teaching argument writing using the asTTle persuasive writing progressive indicators. asTTle is funded by the Ministry of Education to Auckland Uniservices Ltd. at the University of Auckland to research and develop an assessment application for Reading, Writing, Mathematics, Pänui, Pängarau, and Tuhituhi for Years 5-10 (Levels 2-6) for New Zealand schools. We acknowledge this funding, and thank the Ministry of Education for their continued assistance in the development of this project.
The ability to write persuasively or to conduct an argument is an essential academic skill in a number of subject areas and which is prized in high stakes assessments. Poor performance in such writing may be attributed to student inexperience with this socio-communicative purpose and ignorance of the task demands of this type of writing. A teaching intervention for argument writing based on the asTTle ‘persuade' progress indicators for Levels 2 to 4 was implemented in four secondary school classrooms. The students were largely Maori and Pacific Nations students with scores for deep features of argument consistent with performance in the lower half of curriculum level 2. Within the course of a 25-hour instructional programme, a large average gain in deep features of argument writing of over one curriculum level (effect size = 1.6) was reliably found.
Zwiegelaar, J., & Brown, G.T.L. (2002). Teacher evaluation of the reading level 4 assessments: Summary. (Tech. Rep. No. 30). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises feedback responses of the teachers (n=20) involved in the evaluation of asTTle reading level 4 papers in November 2001. The majority of teacher and student responses to the reading assessments were positive, though more negative comment is expressed from students regarding the difficulty of items.
Schouten, J., & Brown, G.T.L. (2002). Report on teacher feedback from the first calibration of mathematics assessments. (Tech. Rep. No. 31). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises feedback responses of the teachers (n=33) involved in the second nationally representative calibration of asTTle mathematics papers in May 2002. Feedback was generally positive and from both teachers and students.
Lavery, L., & Brown, G. T. L. (2001). Report on teacher feedback from the third calibration of writing assessments (Tech. Rep. No. 32). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report summarises feedback responses of the teachers (n=78) involved in the third calibration of asTTle writing papers in December 2001. Both teachers and students were generally positive in their feedback regarding the assessments.
Lavery, L., & Brown, G. T. L. (2002). Overall summary of teacher feedback from the calibrations and trials of the asTTle reading, writing, and mathematics assessments (Tech. Rep. No. 33). Auckland, NZ: University of Auckland, Project asTTle.
Abstract: This report shows how evaluative feedback from teachers was used to improve the quality of assessment materials in mathematics, reading, and writing. Data were collected from the trial and standardisation of asTTle test papers conducted between October 2000 and June 2002. Feedback was used to improve further test forms, test items, and the asTTle software. Teachers identified as their most serious criticism of the asTTle test papers the mismatch in test paper difficulty with the ability of all students in their classes. The final asTTle software will allow teachers to customise test difficulty for the ability of their own students.
Nicholls, H. (2003). English reading curriculum framework and map: Levels 2-6 (Tech. Rep. No. 34). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report explains the principles underlying the mapping of Levels 5 and 6 of the functions in the reading strand of English in the New Zealand Curriculum and includes recommendations for the asTTle Reading Framework, Curriculum Map and Item Development Levels 5 and 6. asTTle is funded by the Ministry of Education to Auckland UniServices Ltd. at the University of Auckland to research and develop an assessment application for Reading, Writing, Mathematics, Pänui, Pängarau, and Tuhituhi for Years 5-7 (Levels 2-4) for New Zealand schools.
Ward, L., Hattie, J. A., & Brown, G. T. L. (2003). The evaluation of asTTle in schools: The power of professional development (Tech. Rep. No. 35). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report provides an evaluation of the implementation of asTTle (Reading and Writing) into 110 New Zealand schools from August 2002 – December 2002. Interviews and surveys were used to elicit teacher feedback on assessment, professional development, ICT, ability to accurately interpret asTTle reports, experience and ease at using asTTle software, and impact on teacher practice. A structural model of responses was able to clarify interpretation of feedback. The major messages were that professional development needs to be oriented most towards encouraging an attitude of the value of ICT in teaching and learning. By positively influencing the teachers' attitudes towards ICT there is a higher likelihood of correctly using asTTle as part of teaching and learning. It is noted, that many of these New Zealand teachers came to the asTTle application with already high levels of such attitudes to ICT as important in teaching and assessment. Further, the accuracy of reports is more a function of the level of impact, perceived quality of the reports, and belief that the results from asTTle will make a difference. This underlines the importance, as outlined in the Theory of Planned Behaviour, of perceived usefulness to subsequent use and correctness of use.
Thomas, G., Holton, D., Tagg, A., & Brown, G. T. L. (2003). Mathematics curriculum framework and map: Levels 2-6 (Tech. Rep. No. 36). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report reviews the Level 2-4 asTTle mathematics framework and identifies and discusses changes that need to be made in order to extend the framework to curriculum Levels 5 and 6 and provides a detailed breakdown of curriculum achievement objectives by curriculum level within the context of the revised asTTle mathematics framework. Changes are largely related to the place and nature of algebra, irrational numbers, and graphing in the curriculum.
Coogan, P., Hoben, N., & Parr, J. M. (2003). Written language curriculum framework and map: Levels 5-6 (Tech. Rep. No. 37). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report explains the principles underlying the mapping of Levels 5 and 6 of the functions in the writing strand of English in the New Zealand Curriculum and includes recommendations for the asTTle Writing Framework, Curriculum Map and scoring for Levels 5 and 6.
Christensen, I., Trinick, T., & Keegan, P. J. (2003). Pängarau curriculum framework and map: Levels 2-6 (Tech. Rep. No. 38). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report reviews the Level 2-4 asTTle Pängarau framework and identifies and discusses changes that need to be made in order to extend the framework to curriculum Levels 5 and 6 and provides a detailed breakdown of curriculum achievement objectives by curriculum level within the context of the revised asTTle Pängarau framework.
Murphy, H., & Gray, A. (2003). Review of Mäori literacy framework for koeke 2-6 panui/tuhituhi of the Mäori language curriculum statement, Te Reo Mäori i roto i ngä Marautanga o Aotearoa (Tech. Rep. No. 39). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report reviews the Level 2-4 asTTle literacy framework and identifies and discusses changes that need to be made in order to extend the framework to curriculum Levels 5 and 6 and provides a detailed breakdown of curriculum achievement objectives by curriculum level within the context of the revised asTTle mathematics framework.
Hughes, P., Hattie, J.A., Brown, G.T.L., & Leeson, H.V. (2004, September). Assessment of student number strategy development: A case study in the use of paper and pencil items. asTTle Technical Report #40, University of Auckland/Ministry of Education.
Abstract: 31 paper and pencil mental strategy assessment items were designed to elicit information about students' ability to process number problems according to six strategies of the New Zealand Number Framework. Data from end-of-year teacher assigned Numeracy Framework strategy and knowledge scores were collected from Year 5 and 6 students (N = 136) in one primary school. Consensus and consistency measures showed that the two instruments fundamentally provided different data, with consistently higher strategies assigned in the paper-and-pencil questionnaire. Suggested steps in improving paper-andpencil items are provided and a model for testing the relationships of data collection mode, strategy, knowledge, and achievement is proposed.
Brown, G. T., & Hattie, J. A. (2003). A national teacher-managed, curriculum-based assessment system: Assessment Tools for Teaching & Learning (asTTle) (Tech. Rep. No. 41). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report explains the national assessment system underlying the creation and functionality of the asTTle system. asTTle is an electronic tool designed to give teachers a curriculum-based assessment system (English, Mathematics, Pangarau, & Te Reo Maori) that they manage to obtain nationally referenced performance indicators. It argues for a curriculum-based, teacher-managed approach to national assessment as the best means of improving achievement, reporting to teachers, schools, and systems, and generating confidence in schooling.
Irving, S. E., & Higginson, R. M. (2003). Improving asTTle for secondary school use: Teacher and student feedback (Tech. Rep. No. 42). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report summarises teacher and student feedback from the calibration (trials) of the asTTle v3 mathematics items, along with focus groups conducted with teachers concerning potential enhancements to the reports contained within asTTle. Analysis of student and teacher feedback from the asTTle V3 trial papers showed that the items were perceived as difficult, that students appreciated layout and design of the test papers, and that doing tests early in the year reduced opportunity to learn. Teachers favoured a networked, digital asTTle where students sit asTTle tests on computers with computer controlled scoring, data entry, and updating of a school wide database of results. They also understood and liked the revised data display mechanisms being proposed for asTTle V3.
Hattie, J.A.C., & Brown, G.T.L. (2004, September). Cognitive processes in asTTle: The SOLO taxonomy. asTTle Technical Report #43, University of Auckland/Ministry of Education.
Abstract: The Structure of Observed Learning Outcomes (SOLO) cognitive processing taxonomy, developed in the 1 970s and 80s by two Australian academics—John Biggs and Kevin Collis, categorises mental activity by quantity and quality attributes of the activities required by students or by the observable products of student work. This taxonomy has been used in asTTle to categorise student performance on every task in Reading/Pänui and Mathematics/Pängarau. This report explains the SOLO taxonomy and its psychological basis, and provides examples of using SOLO in assessment and education in general. Other technical reports will examine the performance of students by the SOLO taxonomy categories.
Keegan, P. J., & Pipi, A. (2003). Summary of the teacher feedback from the calibration of the asTTle V3 pängarau assessments. (Tech. Rep. No. 44). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: T This report details the summary of the teacher feedback from the calibration (trials) of the asTTle V3 pangarau assessments. It has been modeled on earlier reports describing lessons learned from the trialling of English assessments in reading, mathematics, and writing. The overall teacher response to the asTTle pangarau trial was positive and many students enjoyed undertaking the trial papers. Teachers' major areas of concern included item difficulty and vocabulary within each trial paper.
Brown, G. T. L., Pipe, D., & Higginson, R. (2003, December). Writing at Level 5-6 of the New Zealand English Curriculum: Development of Progress Indicators & Tasks. (Tech. Rep. No. 45). Auckland, NZ: University of Auckland/Ministry of Education.
Abstract: This report overviews the procedures and results of a study in the development of progress indicators and writing tasks for curriculum levels 5 to 6. Progress indicators have been drafted for seven writing purposes (analyse, persuade, instruct, describe, narrate, explain, recount) and for seven scoring dimensions (audience, content, structure, language, grammar, punctuation, and spelling). Lessons learned from the process of reviewing these workshops are documented and issues for further research identified.
Keegan, P.J., & Ngaia, T. (2004, August). Summary of teacher feedback from the V4 calibration of asTTle pänui and tuhituhi assessments for Curriculum Levels 2 to 6. (asTTle Tech. Rep. 46). University of Auckland/Ministry of Education.
Abstract: This report summarises the teacher feedback from the standardisation of around 2,300 pänui (reading) and 1,800 tuhituhi (writing) scripts completed by Mäori-medium students in Years 4 to 12 in early 2004. This evaluative feedback is used to improve the quality of assessment materials in pänui and tuhituhi before their publication in the forthcoming asTTle V4 software. The asTTle software allows teachers to customise test difficulty for the ability of their own students.
Hattie, J.A.C., Brown, G.T.L., Irving, S.E., Keegan, P.J., Sussex, K., Cutforth, S., Yu, J. & MacKay, A.J. (2004, September). Use of asTTle in secondary schools: Evaluation of the pilot release of asTTle V3. asTTle Tech. Rep., 47, University of Auckland/Ministry of Education.
Abstract: A multi-method evaluation of secondary school teachers' use and understanding of asTTle V3 was conducted during the first half of 2004. Just over 50 schools provided feedback on current uses and benefits and gave feedback on future improvements. Feedback indicated that the current asTTle is well received. Requested improvements related largely to ICT, professional development, Maori-medium, new reports, and new content. Examination of the asTTle development schedule has shown that many of these requests have been anticipated, but that development of a fully digital asTTle will be necessary to meet all teacher requirements.
Brown, G.T.L., Irving, S.E. & Sussex, K. (2004, August). Scoring of a nationally representative sample of student writing at Level 5—6 of the New Zealand English curriculum: Use and refinement of the asTTle progress indicators & tasks. asTTle Technical Report 48, University of Auckland/Ministry of Education.
Abstract: This report overviews the procedures and results of a scoring panel of some 7000 writing scripts completed by students in Years 8 to 12 in response to 35 writing prompts. The panel consisted of some 50 Auckland-based secondary school teachers who were trained in the use of the asTTle progress indicators for 7 different purposes (i.e., analyse, persuade, instruct, describe, narrate, explain, recount) across the range of curriculum levels 2 to 6. Each script was scored for seven dimensions (i.e., audience, content, structure, language, grammar, punctuation, and spelling). Feedback from panellists in the process of using the rubrics led to the revisions of the indicators. Evidence for the reliability of scoring is reported using consensus, consistency, and measurement approaches. High levels of reliability (consensus adjacent agreement > 67%; consistency a > .80; measurement f > .80) were demonstrated giving users confidence in both the scoring rubrics supplied with asTTle and the norms embedded in the asTTle application.
Brown, G.T.L., Irving, S.E., Hattie, J., Sussex, K., & Cutforth, S. (2004, August). Summary of Teacher Feedback from the Secondary School Calibration of asTTle™ Reading and Writing Assessments for Curriculum Levels 4 to 6. (asTTle Tech. Rep. 49). University of Auckland/Ministry of Education.
Abstract: This report summarises the teacher feedback from the large-scale standardisation of over 10,000 reading and 7000 writing scripts completed by students in Years 8 to 12 in early 2004. This evaluative feedback is used to improve the quality of assessment materials in reading and writing before their publication in the forthcoming asTTle™ V4 software. Teachers identified as their most serious criticism of the asTTle™ test papers the mismatch in test paper difficulty with the ability of all students in their classes. The asTTle™ software allows teachers to customise test difficulty for the ability of their own students.