WO2001031610A1 - A data collection method - Google Patents

Abstract

A method of reporting school students performance upon completion of assigned tasks. The assessment is made by a teacher and permits the teacher to define a set of performance criteria for the assessment of classes that are to be completed by the school students. The teacher can assess the performance of the students in the class according to defined performance criteria either during, or upon completion of, the assigned tasks. The teacher records the results of the assessed performance of the students and reports these results in a report. Optionally, the teacher may select the format of the report from a number formats, or alternatively, the teacher may design the format of the report.

Description

A data collection method

Field of the invention

The present invention relates to a data collection method for use in a reporting system and to a method of implementing same, and to a reporting method and system Background of the invention

A number of industries require reports to evaluate systems within that industry This can be a cumbersome exercise, particularly where a large quantity of data has to be collected, evaluated and compiled into a report in a relatively short period of time

In the education service sector, for example, it is necessary for teachers who must assess the performance ot students, to record the results of each students under their charge and to make a report on his/her progress This can result in the expenditure of a significant amount of time and effort on behalf of the teacher Additionally, with a large number of students under their charge, they may be more likely to make mistakes in performing the evaluation, particularly when they are under pressure due to time constraints This may result in the accuracy and/or quality of the reports suffering because there is no easy way to generate the reports Another example of an industry which requires accurate reports is the manufacturing industry In this regard, it is desirable that the performance of unit processes to be evaluated to thereby determine if it is performing satisfactorily so that the performance of the whole process does not suffer For example, in a smelting operation there are a number of unit processes such other that the smelter which require consideration, such as fuel services, labor services, transport services and maintenance It is important to the manager of such operations that they be able to constantly monitor and review each of the services so that the overall performance of the smelter is maintained To achieve this end, the manager may rely on a number of reports to indicate to him/her, those services which require attention Again, the generation of reports in this example can be quite cumbersome

The applicant does not concede that the prior art discussed in this specification forms part of the common general knowledge in the art at the priority date of this application Summary of the invention

It is an object of the invention to provide a convenient method to collect data to allow the performance of a subject upon completion of one or more assigned tasks to be included in a report

A broad aspect of the invention provides a method for collecting data for use in reporting the performance of a subject upon completion of one or more assigned tasks, said method comprising the steps of (a) permitting an assessor to define a set of performance criteria for the assessment ot one or more assigned tasks to be completed by one or more subjects, (b) allowing said assessor to assess the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks, and

(c) recording the results of said assessed performance in step (b) The method may further includes the step of

(d) reporting the results of said assessed performance in a report. Optionally, method further includes the step of

(e) permitting said assessor to select the format ot said report from a multiplicity of formats or, to design the format ot said report. For a plurality of subjects, the method may further include the step of

(f) permitting said assessor to group each subject according to a particular category. The method may further include the step of:

(g) grouping different types of assessment criteria to thereby form a template. The method may further include the step of: (h) permitting an assessor to amend said set ot performance criteria

The method may further include the step of

(l) permitting an assessor to define one or more quality values to said results of said set of performance criteria to thereby determine the quality of each subject or the quality of a group of subjects

The method may further include the step of: (j) providing a facility to allow the assessor to make general comments in relation to the performance of said assigned task.

The method may further include the step ot:

(k) providing for a plurality of pre- recorded general comments to be available for selection by said assessor in evaluating the performance of said assigned task The Assessor may be permitted to make a selection of said general comments in relation to the performance of a subject, and thereafter rank the general comments so that they are read in said rank on said report

Optionally, a pre-defined comment in relation to the subject is, recommended to said Assessor, based upon the assessment of said subject's performance.

The method may further include the step of: (1) statistically analysing the results of the assessed performance against an assessor defined set The defined set may be a plurality of subjects. Optionally, at least part of said statistical analysis is included as statistical information in said report.

For a plurality of subjects, the report may include a comparison of the results of said assessment.

In step (c), the results can be recorded in a database. The results entered into the database may be input by using a data collection form. The data collection form can be read by a digital scanner adapted for Optical Character Recognition (OCR) and Optical Mark Recognition (OMR).

The results entered into the database may be input by using a device having a digital processor.

The subject may be a natural person and optionally, school students, while the assessor may be a teacher.

Alternatively, the subject may be production units in an process, such as an industrial process.

The report can be printed on paper or alternatively, may be available in electronic format such as a HTML file available from the Internet

External data can be included in said report from external data sources, said external data may include any one or more of the following data types. Assessee data; Assessor data; Assessment criteria data; Groupings ot assessment criteria data; Hierarchies of assessment criteria data.

The pre-defined assessment criteria may be recorded in said database for use in a future assessment session.

The results entered into the database can be input by using a device having a digital processor

The digital processor may exchange data with the database over a communications network, such as the Internet

The format of said report may be automatically generated by said digital processor upon instructions of an application program The format of the report may be automatically by the digital processor, dependent upon the assessment criteria selected by the Assessor.

In another broad aspect of the invention, there is provided a computer network memory storing thereon an application program for controlling the execution of a processor for collecting data for use in reporting the performance of a subject upon completion of one or more assigned tasks, the computer program controlling the processor to: permit an assessor to define a set of performance criteria for the assessment of one or more assigned tasks to be completed by one or more subjects; allow said assessor to assess the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks; and record the results of said assessed performance.

The computer program may further control the processor to' report the results of said assessed performance in a report. Optionally, the computer program may further control the processor to permit said assessor to select the format of said report from a multiplicity of formats or, to design the format of said report

In yet another broad aspect of the invention, there is provided a data collection system for use in reporting the performance of a subject upon completion of one or more assigned tasks, said reporting system comprising an application program having a set a set of performance criteria defined by an assessor for the assessment ot one or more assigned tasks to be completed by one or more subjects, data storage means for recording assessment data recorded by said application program, said assessment data representing the results ot said assessed performance, and assessor input means capable ot interfacing with said application program for allowing said assessor to assesses the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks

The reporting system may further include a report generating means for reporting the results ot said assessed performance in a report and to permitting said assessor to select the format of said report from a multiplicity of formats or, to design the format of said report

In another broad aspect of the invention, there is provided a method for reporting the performance of a subject upon completion of one or more assigned tasks, said method compπsing the steps of

(a) permitting an assessor to define a set of performance criteria for the assessment of one or more assigned tasks to be completed by one or more subjects, (b) allowing said assessor to assess the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks,

(c) recording the results of said assessed performance in step (b), and

(d) reporting the results of said assessed performance in a report

In a further broad aspect of the invention, there is provided a computer network memory storing thereon an application program tor controlling the execution of a processor for reporting the performance of a subject upon completion of one or more assigned tasks, the computer program controlling the processor to permit an assessor to define a set of performance criteria tor the assessment of one or more assigned tasks to be completed by one or more subjects, allow said assessor to assess the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks, record the results of said assessed performance, and report the results ot said assessed performance in a report In yet another broad aspect of the invention, there is provided a reporting system for reporting the performance of a subject upon completion of one or more assigned tasks, said reporting system comprising an application program having a set a set of performance criteria defined by an assessor for the assessment of one or more assigned tasks to be completed by one or more subjects, data storage means for recording assessment data recorded by said application program, said assessment data representing the results of said assessed performance, assessor input means capable of interfacing with said application program for allowing said assessor to assesses the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks, and a report generating means for reporting the results of said assessed performance in a report

In the description and claims of this specification the word "report" is intended to be interpreted broadly to include not just a physical paper report, but also electronic files such as HTML documents, electronic mail, audio messages such as voice mail, visual images, video images etc

In the description and claims of this specification the word "comprise" and variations of that word, such as "comprises" and "comprising" are not intended to exclude other features, additives, components, integers or steps but rather, unless otherwise stated explicitly, the scope of these words should be construed broadly such that they have an inclusive meaning rather than an exclusive one

Brief description of the drawings

Notwithstanding any other forms which may fall within the scope of the present invention, preferred forms of the invention will now be described, by way of example only, with reference to the accompanying drawings in which

Fig 1 is a block diagram of a processing system in a first preferred embodiment,

Fig 1 A is the processing system of Fig 1 in data communication with the Internet,

Fig 1 B is a block diagram of the assessment and reporting system of the first preferred embodiment, Fig 1C is a block diagram of a group of functional 'units' constituting software used in the first preferred embodiment,

Fig ID is a block diagram tor set-up of a report using an assessment template,

Fig IE shows a display of a window dialogue box for in which the Assessor may select additional academic outcomes for a student, Fig 2 shows a display of a window dialogue box as presented to an Assessor when they are using the Setup Wizard is shown,

Fig 3 is a block diagram showing the categories of KPI's, categorized according to several properties, Fig 4 is a block diagram showing an example of how the frame work for KPI's may be used to categorise a basic computer skill set,

Fig 5 is an illustration of a sample Student Report,

Fig 6 shows a display of a window dialogue box generated by the Template Wizard tor the set-up of KPI types in an assessment for a students 'personal skills' ,

Fig 7 shows a display of a window dialogue box generated by the Template Wizard tor identifying which Curriculum Outcomes,

Fig 8 shows a display of a sample scale sheet for grading KPI's on an appropriate scale defined by the Assessor, Fig 9 shows a display of a sample of a final report layout,

Fig 10 is a block diagram illustrating the functionality of a database synchonisation tool ,

Fig 11 shows a block diagram illustrating the process of printing a print job,

Fig 12 is a block diagram ot the interactions between the database and the components of the print process, Fig 13 are illustrations of a flow chart of a file assembly,

Fig 14 shows scanner interactions,

Fig 15 is a display of an example of a Mark Book,

Fig 16 is a display of a number of pre-registered comments within a database,

Fig 17 illustrates a Student Assessment form with a selection of comments, Fig 18 illustrates a display of the comment codes screen of the Mark Book,

Fig 19 is a display of the general comments panel ot the Mark Book,

Fig 20, is a display of window presented to an Assessor in order to make comments for a particular student's report,

Fig 21 is a display of a dialogue box which is presented to an Assessor in order to choose the format of the report that is to be generated,

Fig 22 is a block diagram for the creation of a new report layout, Fig 23 is a block diagram for the insertion ot a table in the report,

Fig 24 is a block diagram of saving a report layer,

Fig 25 is a window display of the Assign Layout Screen showing the Layout Design, Course Outline and Work Requirements for a report, Fig 26 is a window display of the screen for entering Course Outline and Work Requirements information,

Fig 27 displays a window of the Report Selector,

Fig 28 is a block diagram of the generation of the Report Printing process, Fig 29 is a display of the print selector where ordering is by a subject,

Fig 30 is a block diagram showing how Assessors go through a process of continually broadening their analysis of assessment results to ensure that assessments are consistent across all data,

Fig 31 is a block diagram for the steps taken by an assessor in validating the data in the database,

Fig 32 is a block diagram ot the steps taken in the rollover procedure for the database Fig 33 is a block diagram of a central server involved with an Internet client user,

Fig 34 is a block diagram of a local server involved with a LAN client user,

Fig 35 illustrates a block diagram for a second embodiment,

Fig 36 illustrates a block diagram for a third embodiment, and

Fig 37 illustrates a block diagram for a third embodiment Detailed description of the embodiments

A preferred embodiment provides a method of reporting the performance ot a school students upon completion of assigned tasks The assessment is made by a teacher and permits the teacher to define a set of performance criteria tor the assessment of classes that are to be completed the school students The teacher can assess the performance the students in the class according to the defined performance criteria, either during or upon completion of, said assigned tasks The teacher records the results of the assessed performance of the students and reports these results in a report Optionally, the teacher may select the format ot the report from a number formats, or alternatively, the teacher may design the format of the report

The present embodiment is capable of running on any general purpose computer system or computer controlled Graphical User Interface (GUI), including GUIs that have the ability to present multimedia and virtual reality information One preferred embodiment is schematically represented in a block diagram in FIG 1 A computer system 10 comprises a central processing unit (CPU) 1 1, memory storage device 12, one or more monitors or graphical interfaces 13, and entry device 14 such as a keyboard/mouse or speech recognition system

In one embodiment, a IBM RISC SYSTEM/6000 comprises a control processing unit (CPU) 10, memory storage device 12, one or more monitors 13, and a mouse 14 The mouse 140 may be used to select or icons On an IBM RISC System/6000 multiple monitors 13 can be controlled by multiple monitor adaptor cards such as the IBM

RISC System/6000 Color Graphics Display Adaptor The computer system 10 may also have audio input/output capability 17 In addition, speech synthesis or speech recognition may be provided for the entry device 14 Voice recognition may take place using a IBM VoiceType Dictation Adapter

In an alternative embodiment shown in Fig 1A, the CPU 11 can be connected via a network adaptor 15 to connect the system 10 to a distributed network Network adaptors 15 are well known Three examples of network adaptors 15 include token ring adaptors, ethernet adaptors, and modems The system 10 can be connected to other target monitors 13 through a client/server network, LAN or in this case, the Internet 16

Systems that can be used to display graphical images, like icons and windows, are well known GUIs can be used to control any apparatus having a monitor

An application program is stored in the memory 12 and will hereafter be referred to as 'ReportCard' Reportcard is primarily concerned with aiding in the process of preparing, carrying out, recording, analysing and reporting assessments Assessments may take the form of observations, measurements, tasks, activities, tests or any other action where a assessment result needs to be collected and recorded ReportCard concerns itself primarily with assessments in education which may be such things as

• Test Marks • Educational Outcomes

• General Observations

• Other educational assessment activities

Overview of Assessment System

A description of functionality of ReportCard will now be described with refrence to Fig IB which shows a block diagram of the overall assessment development and method

S] Import Data

At step SI , data related to Students, Teachers and Classes is imported from a school's administrative database to a data base stored in the memory 12 In this embodiment, an Assessor is a teacher of school pupils who are the subjects who must complete a series of assigned tasks set by the teacher S2 Set Up Curriculum Bank

At step S2, a 'Curriculum Bank' is defined by the teacher The Curriculum Bank is a relational data base stored in the memory 12 Data to the Curriculum Bank is able to be input with data relating to student outcomes and comments through any number of, or a combination of the following input sources

( 1 ) Import from data from Assessor Web Site (2) Import school designed data which has been entered into an spreadsheet template, such as a template developed on Excel™ spreadsheet provided by Microsoft Corporation Inc,

(3) Data input directly into the Curriculum Bank data base It is also possible to modify imported data in the Curriculum Bank data base from any of the above input sources

53 Select Appropriate Outcomes and Comment

At step S3, the Teacher selects the outcomes and comments which they wish to access to assess the students in their class/subject groups

54 Set up Report Templates

At step S4, the teacher sets up the Report Templates Preferably, this involves one template being set up per subject per year group

55 Print Teaching Guides, Class Lists and a sample Student Assessment Form At step S5, a checking step is performed in which Teaching Guides, Class Lists and an example of a

Student Assessment Form are provided to teachers so that they can review the assessment criteria that they have performed and check that it is correct

56 Make necessary corrections or modifications

At step S6, the teacher is provided with an opportunity to make any modifications which are required after the teacher has reviewed the above documents The changes are made in the Curriculum Bank database

57 Print Class Sets of Student Assessment Forms

At step S7, the Class sets of Student Assessment Forms are printed and distributed to teachers

58 Assess and Record Achievements

In step S8, the teachers record the achievement ot students to the selected input means such as (l) Student Assessment Forms as they are assessed (requiring a scanner),

(ll) directly into the Mark Book, or (ill) directly to the data base in memory 12 by entering data via a PC

59 Scan and Process Student Assessment Forms

Step S9 is completed if the achievement of the students are recorded either into the Student Assessment Forms or the Mark Book If this is the case, the completed forms are scanned and processed to bring the relevant information into the database located in the memory 12

S10 Design Report Layouts (including Cover Page if Required)

The layout or format of the Reports for the students are Designed in a 'Layout Designer' It should be realised that this may occur at any stage in the process before the reports are generated Sll Link Report Layouts with appropriate Report Templates - Add Course Outlines and Work

Requirements (if Required) In step SI 1, Report Layouts are linked to relevant Report Templates, it is at this stage that Course Outlines and Work Requirements are entered into the software if this feature is to be utilized This may occur as soon as Layouts have been designed and Report Templates created

512 Print and Check Report Summaries At step SI 2, summaries of the outcome achievements, grades, test marks and comments are printed for teacher's to check before issue ot the finalised reports

513 Make necessary modifications

Any final modifications to the reports are made at step S13

514 Print Reports Reports are printed at step 14 and collated ready to go home, with a cover page if required

515 Roll Over Curriculum and Template Data

At step S15, the Curriculum Bank database and Templates are rolled over into a new database so that these can be modified and used for the following Semester or year In this case, a further step of re-importing administrative data into the database in memory 12 is undertaken to populate it for the current reporting period In the following period, the steps SI to S 15 are repeated

The Set Up Wizard

The Set Up Wizard is the section of REPORT CARD which provides the overall customization framework for the client's database A description of Set-up Wizard will now follow with reference to Fig 1 D

Step S20 Select Major Outcome Descriptors At Step S20, the Assessor sets the 'Outcome Achievement Descriptors' This set of descriptors act as the default achievement descriptors in the Curriculum Bank data base

Step S21 Set Additional Outcome Achievement Descriptors

At Step S21, the Assessor may choose to set additional Outcome Descriptors such as descriptors for Class Work Skills, in contrast to previously selected academic descriptors These descriptors must be attached to the relevant outcomes in the Curriculum Bank data base

An example of a window dialogue box for Step S21 is shown in Fig IE, in which the Assessor may select additional academic outcomes in which an assessment may be made of non-gradable skills such as behavior and personal skills may be assessed

Step S22 Enable the capacity record test results At Step S22, the Assessor may choose to have the option to record Test Results If the Assessor selects

'Yes they will be required to set default test names, however, if the Assessor selects 'No' there is no option to record these results S23 Enable the capacity to record Grade Results

At S23, the Assessor chooses whether or not they wish to use Grades in the assessment of the subjects If the Assessor selects the option, they will be required to set up to five Areas which they wish to grade, otherwise the grade assessment functionality is not enabled S24 Set Grade Descriptors

At step S24, the Assessor sets descriptors for Grades (if the option to use grades has been Selected) Up to ten Grade Descriptors may be set, in addition there is the option to use the symbols + and - along with the set Grade Descriptors

525 Set T/F Student attributes At Step S25, the Assessor may set up to six attributes, requiring a true/false response relating to individual students which are to be assessed

526 Set Numerical Attributes

At step S26, the Assessor may set up to three attributes, requiring a numerical response relating to individual students S27 Set T/F Class A ttrώutes

At step S27 the Assessor may set up to four attributes, requiring a true/false response relating to particular class groups

528 Provide required configuration and option information

At step S28 the Assessor sets a series general options and settings, relating to School Organisation and the use ot REPORT CARD A window dialogue box 29 as presented to an Assessor when they are using a monitor in association with the Set-up Wizard is shown in Fig 2, for formatting the output of the school report The dialogue box 29 allows the Assessor to assign a name of the group that the student is assessed in (le 'House'), the name of the Assessor ('Tutor'), whether there will be a Cover Page on the report ('Yes'), whether personal comments will appear on the report ('Comment Blank' selected), method of entering data ('MFD' which is , or via Keyboard, OMR which is Optical Mark Recognition), the side tor the highest mark to occur in the report ('Left' in this case but 'Right' could also be chosen), and the timing for backups to be automatically created during the assessment period ('0' days)

529 Set years and faculties

At step S29 the Assessor sets the 'Years' and 'Subject Groupings' which they wish to utilise The REPORT CARD Units

There are a number of modules, known as 'units', which make up REPORT CARD These units and their relationships, where present, are shown in Fig 1C A description of the particular units will now follow

Key Point Indicator Groupings (23) Key Point Indicators (KPI) is a set of Assessor defined criteria for assessing the performance of a subject upon completing an assigned task or activity KPIs may be grouped in a variety of ways tor a range of different purposes KPI's can be categorized according to several properties as in Fig 3 below These groupings allow the KPI's to be easily manipulated In Fig 4, there is shown an example of how the frame work tor KPI's may be used to categorise a basic computer skill set KPI types are used for three main purposes

1 To allow KPI's to be grouped for the purpose of displaying different 'types' of KPIs on different parts of a report

2 To allow different 'types' of KPI's to be graded on different scales (e g 'Achieved/Not achieved' instead of 'Always, Usually, Sometimes, Rarely')

3 To allow performance of KPI' to be analysed according to particular types (subsets)

Referring to Fig 5, there is shown a sample Student Report showing two tables - one for each of the major types of KPI used - (1) Work Skills, and (2) Performance Skills

In addition the 'Parent Interview' tick box is based on a third type of outcome with a True/False grading scale

There are five major operations involving KPI Types They are

(1) Set up the KPI Types (Names/Grading Scales etc),

(2) Identify which outcomes are of each type,

(3) Grade KPI's on appropriate scale, (4) Design Report Layout to show different KPI Types, and

(5) Analyse performance for each KPI Type

Referring to Fig 6, there is shown a window dialogue box 30 generated by the Template Wizard for the set-up of KPI types in an assessment for a students 'personal skills' The dialogue 30 allows the Assessor to define a grading scale to the student's personal skills such as behavior The dialogue box 30 also allows for marks to be assigned from two input sources (1) the teacher's 'Mark Book", or (2) a 'Scan Sheet' filled in by the teacher and then passed over a scanner having OMR capabilities

Referring now to Fig 7, there is shown a window dialogue box 31 generated by the Template Wi/ard for identifying which Curriculum Outcomes 32 (defined in the Curriculum bank data base) are assigned for particular Curriculum Objectives 33 for a particular learning area 34 The KPI's can be graded on an appropriate scale defined by the teacher, as shown in the sample scale sheet in Fig 8

A sample of a final report layout is shown in Fig 9, this shows the 'Personal Outcomes' defined by the Assessor by arrow 34 Database Synchronisation Tool

The purpose of the database synchonisation tool (DST) is to provide an uplink and synchronization function to allow a primary database to link to a range of administration databases in the memory 12

Prospective clients computers of the DST have legacy administration systems used in a variety of areas of their business The DST 'feeds off the data in the existing administration systems

The amount of useful data which can be extracted varies from one client computer site to another The extraction is capable of extracting the maximum amount of useful data Data in the DST is preferably never deleted however, records which are no longer stored in the main administration system are marked as inactive

Some of the fields in DST need to be updated to reflect changes in the main administration system, others are only be updated on the first data import It is preferable that DST imposes the minimum amount ot restrictions on incoming data in order to allow it to operate with a wide range of differing database structures and formats At the import phase some data grooming needs to occur in order to ensure the successful operation of the DST Fig 10 is a block diagram illustrating the functionality of the DST

Handling Student/Class/Teacher records The student/class/teacher data is normally downloaded from the school's administration system into

REPORT CARD This data is stored in Personal Files and contains information about Staff, Students, Classes and Marks

The Student Files is broken into six sections and contains 1 General information about the student such as name, student number and pastoral group, 2 Details on classes attended,

3 Record of outcome achievements including general comments and the ability to plot a graph from the outcomes achieved,

4 Record ot absences,

5 Comments from Senior Staff and Home Room teachers, and 6 Area for additional notes

The Staff Files section is broken into three sections and contains

1 Non-confidential general information such as name, position and staff number,

2 Details of classes taught,

3 Security information, and 4 Area for additional notes

Class Records is broken into four sections 1 General information about the class including year, associated teachers, when it runs and additional information,

2 Class list contains the names of the student in the class,

3 Class outcomes achievements summary, and 4 Area for additional notes

The Assessor can add a new student record through by creating a New Record, or by editing an existing record Records can also be updated to indicate that a student has left the school The Assessor can also change the class the student is in if the same outcomes are used in both classes (if outcomes are different, it will need to be changed through Class Records) Adding or editing a teacher's record can also be made through Staff Files All changes to class details are made through Class Records REPORT CARD allows you to have more than one teacher for a particular class e g specialist teachers for Library Studies, Sport etc

The Curriculum Bank database

The Curriculum Bank database allows you to store all the General Comments and Outcomes which are needed for the teachers to draw from to create a student's report Put simply, it is a holding bay which can be added to at any time or amended

The Outcomes are grouped and separated by stages The statements in the General Comments area are linked to a general topic and can also be made subject specific

Importing Curriculum Data

Outcomes and General Comments can be imported into the Curriculum Bank database There are two

1 The Assessor can either download it from the 'Assessor's Area' on the Web or use the Excel™ spreadsheet which is provided with the REPORT CARD software and type in Outcomes/General Comments,

2 The Assessor can import the data by selecting a 'Batch Processes' option provided on the 'Main Menu' of the REPORT CARD software and select the file to import the data REPORT CARD will then go through a process of checking that the data is in the right format Once this checking is complete and no critical errors found, the data is stored in the Curriculum Bank database

Template Wizard

The Template Wizard is used to create Report Templates A Report Template is the collection of outcomes, tests and comments that a teacher will choose from when reporting on the achievements of students in a particular class Once a Template has been created, it can be used by different classes and used year after year It can also be modified each year as courses change The Template Wizard takes you step by step through the process ot creating and maintaining a Template The first step is to run the Template Wizard This is done by selecting the Reporting Template Attributes These are set up to match the needs of a particular faculty within the school and to address the areas of the curriculum taught at a particular stage in the learning continuum These attributes may include Test results, Grades, Outcomes, General Comments, and Grading Categories (such as Homework, Conduct etc) The Assessor selects which Outcomes which are to be on the Template The Outcomes and are chosen from the Curriculum Bank database The order of the outcomes on the Template can be changed easily by selecting the data field and using up/down arrow icons If the Assessor does not provide a grade for an Outcome, it will not appear on the final report For example, if there were a couple ot accelerated students in the class, it would be better to include all the outcomes for all students in the class and only grade the extra outcomes for those accelerated students

The next step is to create or edit a set of General Comments that the Assessor wishes to use with this Template The Assessor has three options

1 teachers can type the comments directly into REPORT CARD,

2 type the comments into a word processor such as Word™ word processing software provided by Microsoft Corporation and then cut and paste them into REPORT CARD, or

3 create a set of standard comments from the Curriculum Bank database

The next and final step to complete the Report Template is now to link the class/classes which will use this template A list of all classes appear on the screen and the Assessor selects the appropriate classes and clicks on

Add The Assessor can then print a Teaching Guide tor the teacher(s) to check the Outcomes, General Comments etc Any amendments are preferably made before the scan sheets are printed The Assessor can make changes at any time before the Template starts to be used Once a Report Template is in use, it cannot be altered

Scan Sheet Printing

Scan sheets are printed with the intention of

• presenting options to a Assessor about a student's performance in a particular subject area • gathering information from that Assessor about that student's performance by marking up the form

• using an automated system by which the information thus gathered is entered into the database

A scan sheet print run creates scan sheets from the database according to the database's

• current template information

• student-class-template relationships provided by the Assessor for that print run Scan sheets are printed according to advice gained from the database at the moment of printing Since template information may change (a Assessor changes the items listed on a template or the template is replaced by another etc), so the printing process must comprehensively save a snapshot of the printed template Each printed form contains enough information to find the original information Each time a print job is run, the system creates 'generation' information allowing the form to be related to its underlying data regardless of changes a Assessor might make to the original template's structure

This allows the system to impose fewer rules on the Assessor of REPORT CARD

For Example • A form can be created, printed and distributed

• The REPORT CARD Assessor may then change their mind about what should be on the form

• The Assessor changes the appropriate templates in the system

• Previously distributed forms (now out of date) may still be marked up by appropriate persons

• Out of Date forms are returned and processed and the system does not object, all data is saved in a meaningful manner in the context of the changes to the template(s)

REPORT CARD generates a print job table, containing (among other things) the following information

• Student Identification Number

• Class Identification Number

• Template Identification Number • TCS (Template Class Student) ID

The three Identification Numbers uniquely identity the form The database is invited to generate a unique number that simplifies access to the Student/Class/Template information and the associated marks and the paper form that is generated as the physical output from this process

A generation number is created to enable tracking of the template generation for later recognition Each form is tagged with the Student/Class/Template Identification Number (TCS ID) and with the generation number This is plain printed text

Error Checking and Correcting

The TCS number is important to the system's functioning and so is duplicated and a check digit is added to the number for later error detection and correction This digit is called the Longitudinal Redundancy Check (LRC)

The TCS number and generation number are concatenated and the LRC is calculated from the resulting number The resulting number is then printed in the bottom left and bottom right of the form

Alignment and Trimming

In order to facilitate later form trimming, alignment and processing of marks (all described in Scan Sheet Processing), alignment marks are added in three corners and alongside areas of interest Alignment marks are black rectangles of an agreed size Human Interface

Assessment forms are printed on a standard printer such that all relevant details are readable by a human in natural language The forms are given heading and other information in plain text so a human can identify relevant TCS details and read the information on the form The Assessor marking up the form can mark up the form and a human can interpret the marks

Machine Interface

The forms are printed such that a machine can reliably be used to read the marks

Generation Information

The printing process creates a temporary table within the database which contains all details required to print the forms for the current print job, including generation information

This table is used to create a snapshot of the template information necessary to recognise any form that was ever printed, even if the original template is subsequently altered within the database

Creating Print Job and Printing

Referring to Fig 11, there is shown a block diagram illustrating the process of printing a print job as described below, including the Calibration process Fig 12 is a block diagram of the interactions between the database and the components of the print process

Creation of the print job information is done by a program called process exe It is also responsible tor creating the ODBC DSN to enable the printing engine to print the forms

The print engine (formprt exe) is executed when process exe is about to exit Calibration

Because output from different printers could vary, the print engine should preferably be capable of printing a form which will enable the scan sheet processing step to effectively operate For this reason, the print engine can print a calibration form This form contains an example of all of the characters the scan form is expected to contain Scan Sheet Processing

Scan sheet processing is the system by which information contained on paper forms printed at the Scan Sheet Printing step and hand marked up by Assessors are entered into the database Generally, Scan Sheet Processing is a two step process as follows

1 Scan the paper and save it within the computer as a digital image 2 Process the digital image, saving marks found into the database

Both these steps are expanded further below Scanning

Scanning for our purpose is the process by which a marked form is converted to a digital image file and thereby made available to the processing step The scanning step is designed to be done using any standard scanner for which TWAIN drivers are available The process happens in the following manner • Scanner PCs are allocated an Identification Number which is unique The default is 1 , since most organisations will only have a single scanner subsystem This Identification Number is used to generate unique tile names, allowing multiple scanning subsystems to work independently and without knowledge of one another

• The scanner device is loaded with one or more scan sheets (according to its capability and the number of scan sheets to be processed) • The scan program is told to commence scanning

• Scan subsystem commands the scanner to commence scanning

• Motoπsed scanners keep scanning until they run out of paper

• Images are saved as separate files on the file system available to the PC according to information found in an initialisation file This process is repeated until no more scan sheets are available

All images are saved in monochrome (1 Bit) as TIFF files numbered sequentially in order of scanning Several scanning systems can operate at the same time saving images to the same place File names are generated in such a manner as to prevent conflicts

Resolution at which forms are saved is recommended to be 300 dots per inch, however for slower and disk capacity limited machines this can be reduced to a minimum of 200 Processing is unaffected by changes in resolution

Calibration is also performed Calibration is the process by which the scanner and printer are matched by the processing engine As each scanner and each printer has different characteristics, the system gathers information about the density of expected letters and numbers as printed and scanned This increases reliability of the entire process to such an extent, it has been made obligatory for Assessors to calibrate before any processing can be done

The processing software is launched and checks that valid and complete calibration data exists If not, it asks the print engine to print a calibration sheet Calibration sheet is printed as described in Scan Sheet Printing The calibration sheet is then put on the scanner and is scanned

The processing software opens the scanned calibration sheet and inspects the density of the printed letters and numbers on the page, storing these pixelation densities for each letter and number combination When calibration has been successful, the processing engine switches to normal operation Processing

Processing is the means by which the digital image files are examined by the PC and the information found thereon is entered into the REPORT CARD database A form is printed by the Scan Sheet Printing component of REPORT CARD or an image thereof Generally, the system takes an optimistic view on the forms it has processed Erroneous marks only appear so within its own Assessor interface From other sections of REPORT CARD, the system does not show alarming error messages to the Assessor This helps prevent Assessors without access to the original scanned image precipitating changes

The scanned image is straightened ('deskewed') Using the three alignment marks located at the top left, top right and bottom right of the form, the system determines if the image is upside down and rotates the image through 180 degrees it necessary The same alignment marks are then used to trim the margins from the images

Having trimmed the form to approximately known dimensions, the system finds and reads (using Textbπdge OCR) the form identification numbers at the bottom left and right of the page The page number is also identified

Error Checking and Correcting using LRC Once the numbers have been recognised, the accuracy of the OCR is verified using the LRC as explained above The left hand number is verified

• If an error is detected, the number on the right is verified

• If an error is detected with this number, the form is saved for Assessor intervention later

• If all went well, processing continues • The generation number is stored, and the database is interrogated for the Student, Class and Template

Identification Numbers

• Appropriate form generation information (stored during printing) is retrieved and cached

Form Pages - Completeness check

If another page has earlier been found with the same TCS number, and that page did not contain the complete set of responses, the current page is joined to that page in page number order The two joined forms become one and that form is processed as follows

• The alignment marks showing mark locations on the page are counted and the page is assessed tor completeness

• If more alignment marks are expected and , the page is stored and will await other pages with the same Identification Number

The flow chart on Fig 13 and Fig 14 an illustration of page assembly Read Markable Sections

In order to find the sections, during the completeness check described above, the number and location of marked sections for this particular form are stored Because no mark location is absolutely guaranteed (forms are not expected to be exactly straight or to have marks only in regulation areas) each form is individually examined to find the marked sections

Detecting Marks

Marks are detected by examining areas next to the alignment mark and comparing the density of black to the calibrated value for each particular letter Each letter is examined separately, however if a letter is over a threshold value greater than the calibrated value it is considered to be a mark If a letter is near this value, and no better marks are found, it is considered to be a doubtful mark and the form and mark are flagged for later Assessor intervention Possibly the person marking the form makes light marks or has poorly erased a mark The person later correcting errors is therefore advised to check the rest of the form too

It two marks are found greater than the threshold, and one is significantly darker than the other, only the dark mark is stored, otherwise an error is flagged if multiple marks contravene the rules Various rules apply to different types of grading that allow or disallow blanks, multiple marks and so on

More about Pixelation Thresholds

Pixelation threshold vary between characters according to the amount of ink on the printed page For example, a '-' contains very little ink whereas the symbol 'N/A' (considered by the system to be a single character) contains a much greater amount of ink Using similar threshold for both these 'characters' is not appropriate since it would take a great deal less of a mark to push the '-' over the threshold and so have a mark recorded The result is an unmarked '-' (among other characters) will have a great number of 'doubtful' marks recorded whereas 'N/A' will likely have none

The calibration subsystem makes up for this kind of threshold anomaly in various ways Examples of the simplest of these follow • The threshold is not considered crossed until a certain overall percent pixelation is reached

• Characters with less 'ink' have a higher threshold

• Characters are grouped in various ways

• Defaults are set for unknown characters (e g double byte or Unicode characters)

The anomaly differs between different scanner-printer combinations and so although a safe value is set for simplicity, advanced Assessors may make threshold alterations at a very deep level Rules about Overwriting Existing Marks

Each form is broken into sections, each having a particular style of options If any mark is found in a section, all existing marks in the database (however entered), are removed and replaced by found marks If no marks are found, any marks in the database relating to that section ot the form are left untouched Image Archiving

As images are processed (either successfully or unsuccessfully) they are moved to another place in the file system for later examination or archive purposes

Error Correcting - Human Interface

During Scan Sheet Processing, a number of errors can arise The processing engine is designed to catch these errors, deal with them whenever possible or ask a Assessor when it cannot of itself resolve the problem

That having been said, all forms are processed without human intervention That is the engine may be unattended during processing and will only prompt for Assessor intervention on any errors when asked Errors are dealt with in batch form wherever possible

During error correction, the Assessor is shown both the original scanned image (in a floating window) and a special data entry interface (in the background) Segments of the image that are or interest can be zoomed into and out from using the Assessor's pointer device (usually mouse)

These errors and the process by which they are dealt with are enumerated here

When a Assessor terminates correction ot a form, they have the option of marking the form as error tree If the form is marked as error tree, it is removed from the error checking cycle altogether Error in OCR shown by LRC mismatch or other Identification Number Checks

This can be caused by various reasons, e g Obliteration of both form identification marks, something that looked like an alignment mark was found and mistakenly used by the engine as an alignment mark, poor scanning procedure etc

The Assessor is shown the area on the form where the processing engine expects to find the form Identification Number and is asked to type the number The typed number is examined (including LRC) and the form is processed as above It the LRC is incorrect or the form or generation doesn't exist, or the Student, Class or Template doesn't exist, the Assessor is asked to enter the form by hand

Pixelation Thresholds

Forms that contained marks that approached pixelation thresholds but did not exceed them by a significant margin are shown to the Assessor with highlights on the offending marks and comments from the engine to provide guidance in correcting potential errors

It is important to note that this level of errors is a relatively benign one which is mainly designed to flag forms and marks which are 'doubtful' Mostly these forms will be found to be error free Number of Marks found is Outside the Rules

Where the scan engine has determined that any marks on a form contravene the business rules for the engine, these marks are flagged, together with advice, for a Assessor to decide Examples of these marks include

• Percentage mark >100 • Section has too many marks (more marks detected than allowed by the rules)

• Section has insufficient marks (a mandatory section was not filled in)

The Mark Book

The Mark Book allows Assessors to view, edit and add assessment information stored in the REPORT CARD database This enables REPORT CARD Assessors to correct assessment data that has been 'scanned' in, or entered into the database by other means, at their PC The assessment information, in the Mark Book, is usefully grouped by both

(1) Class, and

(2) Report Template

An example of a Mark Book is shown in Fig 15, which is a display 35 of a virtual mark book presented to a teacher on their PC The display 35 details the marks assigned by the teacher for a class of students

Assessment information for a particular assessment subject in the class can be viewed, changed, added or deleted as required To assist in finding the desired class of assessment subjects, the list of classes and Report Templates to choose from can be reduced to a sub set or all available by using the "NARROW" button 36 This allows the lists to be restricted by (a) 'Year' - arrow 37 - selection of field only displays classes for the year chosen

(b) 'Faculty' - arrow 38 - selection of field only show classes in the faculty chosen

(c) 'Reporting Period' -only shows templates associated with that reporting period

Information in the Mark Book is principally shown on 6 'tabs' of the Mark Book window 'Outcome Achievements', 'Outcome Key' , 'Test Results' , 'Grades', 'General Comments' , and 'Analysis' Assessment information in REPORT CARD is divided into 4 types

(1) Outcomes

(2) Test Results

(3) Grades

(4) General Comments There are two more 'tabs' showing (1) Outcomes Key

(2) Analysis

On the outcomes tab the outcomes associated with the current Report Template are numbered from 1 to the total number of outcomes used A tab is given as a legend/key to match the outcome numbers to their descriptions

The assessment information can also be interpreted graphically by using the 6^lh tab labeled "Analysis" The graphical tool allows configuration of the display in normal ways present in typical charting/graphical tools commonly available The information that will be displayed in the graph can be chosen by selecting a number of different options on the tab The data is displayed in a grid which can be viewed before the graph window is created

The graphical element of the Mark Book is also incorporated into the Test Results tab

All data displayed in the Mark Book using the grids can be copied to the Win clipboard by clicking the clipboard button at the top of the window

There are two ways to change 'General Comments' (1) General Comments may be added/edited in free form text, or

(2) choosing from the list of general comments associated with that report template

Web Data Entry

Web Data Entry is the ability to enter marks into the Markbook via the web Markbook screens have been duplicated as closely as possible using web pages - permitting approved Assessors to enter marks remotely Web Data Entry provides capabilities similar to the Desktop version of the Markbook All Web Data Entry function can be accessed by (and only by) Teachers

Login and Authentication using standard security techniques (username and password)

- Selection of 'Class' and 'Report Template' from web form popup to select the particular class the teacher wants to update - Selection of Markbook function from 'tabs', to give one of the following data entry pages

Outcome achievements

Displays student names down side of page, and outcomes across page

Allows Assessor to select an outcome for a particular student and enter a new value for that outcome

Assessor repeats entry of changes as required - Assessor clicks on the single 'Update' button to commit changes to database System verifies that all modified outcomes are valid, reporting any errors (valid outcomes are shown by an onscreen legend)

Outcome keys

Displays the keys and descriptions for the outcomes achievements page. (Read only; no update possible)

Test Results

Displays student names down side of page, and test results across page

Allows Assessor to select a test result for a particular student and enter a new value for that test

Assessor repeats entry of changes as required - Assessor clicks on the single 'Update' button to commit changes to database

System verifies that all modified test are valid, reporting any errors (in range 0 to 100)

Optionally allows Assessor to see graph of student test results by clicking on 'Graph' button. Display changes to show horizontal bars (per student per test) across display representing marks scored in each test. (Student results for a particular test are indicated in the same colour; colour used for display changes between tests)

Grades

Displays student names down side of page, and grades {achievement, effort etc} across page

Allows Assessor to select a grade for a particular student and enter a new value for that grade

Assessor repeats entry of changes as required - Assessor clicks on the single 'Update' button to commit changes to database

System verifies that all modified grades are valid, reporting any errors (valid grades are shown by an onscreen legend; + and - also accepted)

General Comments

Displays student names down side of page, and comments across page - Allows Assessor to select a comment for a particular student and enter a new text

Assessor clicks on the 'Update' button to commit changes to database; one button per comment field (i.e. per student)

System verifies spelling of comment, reporting any errors in spell checking dialog (Change, Change All, Add, Ignore, Ignore All, Stop and Cancel - meaning of these buttons is described in legend on spell checking form) The Comment Bank

All comments generated by codes are stored in the Curriculum Bank database When comments are entered into the Curriculum Bank database they are given a value ranking from 1 to 1 1 with 1 being the most positive comment, 10 being the most critical comment and 1 1 being a value for suggestions for improvement As a selection ot comment statements is compiled REPORT CARD will automatically place the most positive comments first, it will then place more critical comments in the body of the comment, and place suggestions tor improvement last Fig 16 is a display 16 of a number of pre-registered comments within the Curriculum Bank database which can be selected by a teacher in making General Comments tor a student's performance The display 16 allows teacher's to have complete flexibility over the available comment set While a standard set of comment is provided with the software, Assessors can modify, add or delete the comments stored in the Comment Bank database Teachers wishing to access the Comment Bank have the option to select which sets of comments they wish to select from, by narrowing their available selection through the Template Wizard

REPORT CARD has three interfaces tor the creation of subject specific comments tor students ( 1 ) Select codes related to a Comment Bank on Student Assessment Forms

(2) Select codes from the Mark Book

(3) Type a comment or edit a comment directly in the Mark Book

REPORT CARD also enables the Assessor to have specific Pastoral style comments entered into the database (e g Head of Year) tor use on the Cover Page of a students report, this is done by typing the comment directly into the Student's Record

Select codes related to a Comment Bank on Student Assessment Forms

After Report Templates have been created and Student Assessment Forms generated, teachers can create their comment for a student, by selecting the relevant boxes against the appropriate comment category Teachers use a Teaching Guide as a reference to the comment statements in full, to assist them to make their choices Fig 17 illustrates a Student Assessment form with a selection of comments

Select codes from the Mark Book

Comment Bank database comments can be compiled or modified in the Codes section This can be simply achieved by selecting the appropriate Comment Number and Letter and clicking add REPORT CARD automatically orders the comment statements according to the values attributed to them in the Curriculum Bank Fig 18 illustrates a display of the comment codes screen of the mark book Type a comment directly into the Mark Book

Comments can be typed directly into the Comment panel of the mark book They can also be edited in this section of the program, where changes are necessary Fig 19 is a display of the general comments panel of the mark book Entering Pastoral Comments

Principals, Tutors or Year Heads are able to enter a pastoral comment - which can then be produced on the cover page of reports - through the Student Files Referring to Fig 20, this is done by simply accessing the appropriate student's file, selecting the comments tab, and selecting the appropriate Reporting Period and Position, then typing The Process Reports

Presentation - in the case of a paper embodiment, is known as a 'Report' This allows the Assessor to

Choose a defined information area to be presented

Choose to restrict the information returned for that area

Choose to apply formatting options on the presentation of the information - Choose a source tor the information if more than one is available

Save their choices with a name to be used again at a later date

Make their choices the default for you when using the information presentation again

View the presentation on the screen

Print the presentation to paper - Save the raw information used for the presentation

The term 'Process Reports' covers the presentation ot information held in REPORT CARD for printing, viewing or sending to others in various formats but not the core assessment report tor the individual assessment subject (for the schools educational assessment environment this would be the students 'school report')

Process reports allow the REPORT CARD assessor to consider assessment information by a variety of different groupings, with a variety ot different calculations applied such as averages, totals, etc They also allow the Assessor to select only information they desire by giving choices for set ot information that will be presented in the report

Process reports are specific and unique presentations of assessment information from the REPORT CARD database For ad hoc information retrieval and presentation any of the widely available query and reporting tools would be suitable For example Microsoft Access has such tools built in

Process reports are delivered in discrete units allowing the Assessor to add new reports as they become available without having to install a new version ot REPORT CARD The main components involved with running a process report are

Common Components - REPORT CARD Database

Process Reports System components Fig 21 is a display 40 of a dialogue box which is presented to an Assessor in order to choose the format of the report that is to be generated tor a particular student The display 40 also has a 'Preview' button that allows an assessor to preview the format of the report prior to printing

Report Designer

The Report Designer allows the Assessor to create and place objects on a virtual page of a report, known as a Layout The objects may contain literal data or data that is populated from the database in memory 12 The layout of these objects and their constituent properties and actual contents can then be saved to a file on either diskette or hard disk as a template, in a unique, custom-designed file type for later use with data from the database

Report Templates consist of a combination of the Group + Subject + Key Reporting Areas A printed report is created by combining a Report Template with a Report Layout When reports are printed, the appropriate layout can be populated with data from the REPORT CARD Database about each "person" being reported As

"each person" may belong to many groups, it is possible tor one "person" to have several report templates associated with them

There are two distinct layout types that can be created on the designer (1) Cover Page, and (2) Report

The Cover Page contains more merge fields than the Report definition These merge fields refer to information that is not available in each ARC Report Template During a print run, a Cover Page is used once tor each "person" being reported, while a "person" may have multiple Report definitions used with different report templates This allows the supervising body to build a comprehensive reporting structure, based on different facets ot the individual subjects, for each "person"

The objects that can be placed on the virtual page are

Text boxes containing literal text and marker fields that can be populated with data merged from the ARC database

Non-embedded images - which are referenced from local storage on the computer - Graphs - descriptive markers which are populated with merged data

Tables - descriptive markers which are populated with merged data These objects can be formatted in different ways to produce distinctive and different report layouts The formatting includes the expected text formatting of font style, size, bold, italic, underlining, left and right alignment, text justification and tab settings Each object may have borders applied and the images, graphs and tables may have titles applied, separately to the properties of the object Images ot varying types (bitmaps, jpeg, portable network graphics, etc ) may be displayed on the pages

There are several graphing styles from which the Assessor may choose For example, Bar Chart, Line Chart, Rocket Graph and others Each of these graphs may be varied in the details being graphed by selecting database fields that contain numerical data and are marked as being available to be included on a design layout Tables are built from the database store of various Outcome Types for subjects The formatting ot the tables is done by choosing various properties on a dialog box and applying these properties to a particular outcome type After placement the table's text contents can be formatted using the normal text formatting procedures

The layout and placement of these objects is performed by "drag & drop" operations of the mouse on the computer screen The objects may be sized by selecting them with the mouse and then dragging on the markers to fit Objects may be locked in place The saved file is a unique file type to allow quick loading of the files and information for rapid display of the designs The layouts can be printed directly from the designer to allow the Assessor to see how the layout design will look

The nature of the activity of the designer is unstructured, allowing the Assessor to nominate which activity they wish to perform at any stage There are five (5) areas that follow a structured approach These are / Opening the Designer/ Creating a New Layout

The Assessor is presented with the option of creating either a Cover Sheet or a Report Layout The choice determines the number ot merge fields available to be inserted in text boxes The list of merge fields is drawn from the database All the default values are loaded, l e Page size, margins, etc The first page of the new document is then drawn on the screen, along with the page ruler The default insert type of Text is chosen and marked on the menu bar Refer to Fig 22 tor an image of this operation

2 Inserting an Image

The Assessor is presented with a dialog box to allow the selection of an image file A list of suitable files will be created from the available picture files on the local disk The Assessor can select these through a standard dialog box Upon selection of the file the image is inserted to the position described by the drag & drop operation of the Assessor, on the layout

A reference to the location of the file is stored in direct memory, prior to saving the file, along with the rest of the layout descriptions 3 Inserting a Graph

Refer to Fig 23 for the block diagram of this operation The Assessor is presented with a dialog box containing a variety of properties from which to choose The choice of properties determines the format and contents of the graph to be displayed The Assessor chooses the properties ot Graph Type, Title, Border, Horizontal and Vertical axes scale and description, the colours tor each type ot merge data being graphed and the merge fields indicating the names to be displayed in the legend for the graph When all the selections have been made, the Assessor closes the dialog box and the graph is inserted to the screen

4 Inserting a Table Refer to Fig 24 tor the block diagram of this operation The Assessor is presented with a dialog box containing a variety of properties from which to choose The choice of properties determines the format and contents of the table to be displayed The Assessor chooses the properties of the Outcome Type options provided from the database, width of the description column, the type of table to be used, the number ot columns (where appropriate), the marking type for the results ot each outcome, any shading and colours to be used on the table The Assessor may choose to include a legend when the table is written to the screen

When the dialog box is closed, the table is drawn on the screen If the legend is to be displayed, it will be drawn on the screen as well

5 Saving the Layout

The Assessor is presented with a dialog box to determine it this layout will be available tor use with the Report Templates in REPORT CARD The Assessor is then asked to provide a unique name for the layout, so that it may be referenced in the database The Assessor is then asked to provide a name for the disk file These two name may not necessarily be the same

Assigning Layouts

The Assign Layout feature enables the Assessor to attach particular Report Layouts to the Report Templates which were created in the Template Wizard This functionality enables the Assessor to have any number of Layouts and to quickly and simply attach them to the relevant Report Template Course Outlines and

Work Requirements can also be entered into the Assign Layout feature and then attached to the relevant Report

Templates Selecting and Attaching the Relevant Report Layouts to Report Templates The Assessor is able to select from a drop down list any of the available Report Layouts The Assessor may then Attach the chosen Report Layout to any number of given Report Templates

2 Attaching Course Outlines and Work Requirements to Report Templates

The Assessor is able to create a series Course Outlines and/or Work Requirements, which can then be attached to the relevant Report Templates This function enables the same Report Layouts to be used for a series of subjects, even though they have vastly different Course Outlines and Work Requirements Fig 25 is a display of the Assign Layout Screen showing the Layout Design, Course Outline and Work Requirements which have been attached to the relevant classes Fig 26 is a display of the screen for entering Course Outline and Work Requirements information Report Printing

The REPORT CARD Report Printing Engine combines the details captured in the REPORT CARD database, with the layouts created using the Report Designer, to produce a final report consisting ot an optional cover sheet and a series ot printed sheets for each person/class combination in a Assessor-defined selection list

There are preparatory steps to be done within the REPORT CARD application, prior to the printing of reports These are

Define a Class/Subject Template in which to capture information about each person

Capture all the information

Design a Layout to be used with the Class/Subject Template

Assign the Layout to the Class/Subject Template When the Print Selector section of the application is selected, a list of available people in classes, using a

Class/Subject template (descπbed as a Report Name) is displayed in the Print Selector The list is filtered according to the selection of a combination of 'Year' and 'Faculty and Period' categories The list ot potential candidates may be further filtered by selecting all students or only those students that are current at the location

The Report Selector allows the Assessor to select items from the left side of the screen, adding them to a list on the right (refer Fig 27), building this list as a batch of reports to be printed A cover page Layout Design may be selected This design will apply to each student in the list The order of items to be printed may be adjusted through preset options or manually The Assessor can choose to preview a report prior to printing, or they may print a sample ot the reports prior to a complete run

When the selection is completed to the Assessor's satisfaction, the print engine will calculate the average scores and ranking of the scores and students, prior to assigning the values to each student in turn Attendance figure7s are examined for each student This data is stored in a temporary database ready for use during the print run and ready tor re-use should the Assessor require to reprint the reports

The computer iteratively examines each line in the selection list, evaluating if a "new" student is being reported If a Cover sheet is required, then it will be printed prior to any other report for the student, retrieving merge information about the student from the database

Each report template for the student is then evaluated to determine the layout design to be used The layout is found and loaded to the computer The various merge fields on the layout are populated The Graphs are populated according the values of the fields being graphed The tables are populated according to the values of the outcomes for the subject being reported, along with any comments from the bank of comments stored about the student Upon completion of the compilation of all reports for the student, a batch is sent to the printer for physical printing and the next student's reports are processed, and so on to the end of the list As each students reports are printed, a dialog box indicating the progress is displayed on the screen A block diagram of generation of the Report Printing process is shown in Fig 28 Fig 29 is a display of the print selector (unfiltered selection) where ordering is by student Note the first student has been selected for printing Each row may use the same or a different layout

Reports on the Web (Internet)

Reports published by REPORT CARD can be output physically (l e printed) or electronically It the later option is chosen, a PDF and associated XML tile is produced tor each report The PDF file is an electronic version of the printed page, the XML file contains information required for REPORT CARD to index the PDF file These files must be uploaded to REPORT CARD REPORT CARD then indexes the reports so that they can then be viewed in a Browser by Students, Parents, and Teachers once they have successfully logged into REPORT CARD

Reports on the World Wide Web (Web) are accessed by a number ot different Assessor types Assessors types will be indicated by "[ ]" in this section This functionality is divided into two parts, namely Report Upload and Report Viewing

Report Upload [Secretarial Administrator]

- Reports are produced as PDF files and XML by the Desktop version of REPORT CARD

Files are zipped together into one file 'reports zip' - Secretarial Admin executes 'Report Upload bat' which cause zip file to be uploaded to REPORT

CARD During this process they will be asked for Assessor name and password to verity that the upload is from a legitimate source

REPORT CARD extracts the all files from the zip files, using the XML files to index the PDF files, then moves them into the appropriate directory - Reports are now available for viewing

Report Viewing As [Student]

Select 'View Reports'

Select Reporting Period from popup, then click on 'Get Subject' - Select Subject from popup, then click 'show'

View report (PDF file) in browser

Use 'Previous' and 'Next' buttons to go between subject in the current reporting period As [Teacher]

Select 'View Reports'

Select Student from popup (shows all students), then click on 'Get Student'

Select Reporting Period from popup, then click on 'Get Subject' - Select Subject from popup, then click 'show'

View report (PDF file) in browser

Use 'Previous' and 'Next' buttons to go between subject in the current reporting period

As [Parent/Guardian]

Same as [Teacher] except Student popup only shows relevant students (sons/daughters) Student Portfolios

Through the student portfolio feature, Students are able to customise and upload material and maintain their own work portfolios Portfolios require "moderation" or approval by Teachers prior to being made accessible to Public Viewer or Parents

Student Portfolio functionality is accessed by a number ot different Assessor types Assessors types will be indicated by "[ ]" in this section This functionality is divided into a number of parts

Portfolio Customisation [Student]

Student selects 'Edit Portfolio' from main menu, then 'Customise' from submenu

Student select template page to edit, then proceeds to customise title and content areas Can select from range ot font styles (size, colour, text face, text style), background colours etc - Student clicks on preview, followed by save

Student must request moderation following this using 'Moderation Request'

File Upload [Student]

Student selects 'Edit Portfolio' from main menu, then 'Files' from submenu

Student Clicks on 'browse' button, this produces a file selection box from which the student select a file on their local computer to upload

Once file is selected, Student names the "link" which will represent this new file in their portfolio

Student clicks on 'Upload' to upload file

Student must request moderation following this using 'Moderation Request'

Report Publishing [Student] - Student selects 'Edit Portfolio' from main menu, then 'Files' from submenu Student selects 'Year' and 'Semester' from popup menus on web form

Panels show 'Reports Available' and 'Report Published', and can be moved from one panel to the other using buttons

Student can request moderation following this using 'Moderation Request' Moderation Request [Student]

Student selects 'Edit Portfolio' from main menu, then 'Submit' from submenu

Student fills out message form (reason tor moderation), then clicks 'Send' button

Message is sent using email to the school's portfolio moderation email account, and a moderation request is saved in the database - Moderation Approval/Rejection [Teacher]

Teacher selects 'Moderation' from main menu, then 'Show Requests' from submenu

System presents a list of moderation requests (specifies who, time and date of request) extracted from its database

Teacher clicks on students name to go to that students unmoderated portfolio - Teacher views entire portfolio, then goes back to first screen

Teacher selects students they wish to approve, then clicks 'approve' ('Reject', and 'Approve All' are other buttons available here)

If approved, REPORT CARD copies portfolio contents to public viewing area, updating database entries as required Portfolio Viewing

As [Student]

Student selects 'View Portfolio' from main menu, then selects either 'Portfolio Home' or 'Portfolio Files' from submenu

Portfolio Home is the home page for the portfolio - this normally where the student places some introductory material about themselves

Portfolio Files is the file list page for the portfolio Portfolio content is represented as links on this page Clicking a link makes that page appear in the browser Once a page is selected, 'Previous' and 'Next' links move between pages in the portfolio

As [Parent] - Similar to process followed by Student, except that the first step is to select which student (from lists of sons/daughters) that they wish to view As [Public Viewer]

Same as for Students

Analysis Functions

The 'Analysis Functions' provide direct, easy-to-use analysis tools for Assessors whilst providing them with the ability to conduct sophisticated data analysis functions

Analysis is required in the present embodiment to meet two conflicting objectives, on the one hand it should be easy to use by people with minimum training and low level IT skills, and on the other it should be capable of sophisticated Assessor defined analysis functions In order to meet these differing objectives two levels on Analysis have been developed Level 1 Predefined Analysis Functions - these cover the standard range of analysis functions required by a Assessor and are available at the touch of a button

Level 2 Assessor defined Analysis Functions - these allow the Assessor to manipulate data and queries to construct a wide variety of innovative analysis tools

Level 1 Analysis Tools include 1 Assessment Consistency Functions

Compared assessments of different assessors

Compared proportions of assessments awarded at each level of performance

Compared current assessments to previous performance

Compare assessments to 'standard' assessments 2 Individual Performance Analysis

Compared to the performance of others

Compared to own performance in other areas

Compared to 'standard' performance

Compared to own previous performance 3 Group Performance Analysis

Target Groups performance compared to other Groups

Target Groups performance compared to 'standard' performance

Target Group performance compared to previous group performance 4 Whole Sample Performance - Performance of the entire sample by Major Grouping Performance ot the entire sample compared to previous performance

Performance of the entire sample compared to 'standard' performance

Analysis of performance trends

Analysis of effects of special initiatives 5 Statistical Functions

Mark aggregation across topics

Mark ranking and ordering

Generation of a Means b Standard Deviation c Distribution d Modal e Normal Distribution t Weighted totals and mark aggregation - Mark Analysis by topic

All Level 1 analysis tools feature step analysis functions (select target grouping and then select analysis function)

Results displayed on a table

At the push of a button results can be displayed as a graph Level 2 Analysis Functions use the OLAP Data Cube Model This allows predefined queries to be manipulated to provide 'drill down' data analysis functions For example in a school context the Assessor may use the data cube to examine 'how many boys in year 7 have achieved Spelling outcomes

Use of the Analysis tool

Analysis occurs at the end of the assessment process and usually occurs before and after the results of the assessment are reported It usually occurs before reports are generated to allow results to be standardised and to ensure consistency of assessment Analysis after reports are generated are used to assess performance ot the individual, groups or the entire sample

As can be seen in Fig 30, Assessors go through a process of continually broadening their analysis of assessment results to ensure that assessments are consistent across all data Repairing (Maintenance of Database)

As the Database in memory 12 is used by many separate application and throughout the course ot adding or deleting personnel, class, subject, template and many other records, these records are linked in a variety of ways During these linkages, it may be possible tor items to become crosslinked or unlinked, either situation could provide possibilities for loss or corruption of data To prevent this occurring, there are many checks in the activities ot storing and retrieving data, however this may not always prevent problems occurring in the database Such problems can occur as a result of loss of power, sudden fluctuations in power supply, the interaction of other applications on the computer or hard disk anomalies

There are two ways that REPORT CARD can deal with repairing the database Both of these sections can be initiated by the Assessor of the application The first applet called "Check Data and Report" checks the database tor invalid use of the single and double quotation marks within certain fields in the Statements, Outcome, ReportNames, Classes, Students and Staff sections of the relational database

The quotation marks could cause problems when performing queries on the database and are therefore removed and replaced with an innocuous character that has no impact on the querying of the database The applet then checks for classes with null teachers, faculties and ensures that they are for the correct years and Stages The database is then checked for various database tables without required data This applet provides a report, both on the screen and a printer, to the Assessor concerning the various elements of the database on which the Assessor must perform some maintenance, such as entering required data in certain tables

The second applet concerning the repair of the database may be initiated by selecting a 'Check and Repair DB' in the menu of REPORT CARD The second applet performs the following functions

• Evaluates specific database tables where the connections to other database tables don't follow rules created by the software administrator of REPORT CARD Obsolete entries are removed, while current entries are repaired by rebuilding the connections

• Evaluates specific database tables for illegal multiple entries, retaining the entry that conforms to the most rules and deleting all others

• Evaluates specific database tables for total unconnected entries, deleting all entries that do not contain valid data

Fig 31 is a block diagram ot the steps taken by an assessor in repairing the database using either ot the two options described above Rollover

The Database allows the data to be "rolled over" at the end of the time period, providing a consistency of data from one time period to the next, without the administrative effort of re-keying existing data Rollover is achieved by creating a virtual copy of the existing database, in situ, and deleting those data elements that will not be used in the next time period Where data items are required to be "promoted" or changed in some way automatically, the activities are done without any Assessor intervention Some data elements are not carried forward from one time period to the next and are therefore automatically deleted

The rollover of some data elements is ambiguous and can therefore be left to the discretion of the Assessor Therefore prior to the execution of the procedure, a screen display is presented to the Assessor to indicate which elements will be retained from one time period to the next Where the Assessor is required to make changes to the new data, a form specific to the task is displayed to allow the Assessor to complete the operation

Fig 32 is a block diagram of the steps taken in the rollover procedure for the database

System Architecture

The REPORT CARD embodiment utilises two types of web servers referred to as the "Central" and "Local" Servers These web servers are accessed by standard web browsers to use REPORT CARD functions

1 Central Server

An illustration of a central server is shown in Fig 33 The Central Server provides all ot REPORT CARD functionality except for 'Web Data Entry' All information about schools, Student Portfolios, Student Reports are accessible on a per school basis from this server Central Server Operation — File upload to Student Portfolio

For the purposes of illustration, the following example scenario will be used - "A Student updating their online Portfolio" Numbers enclosed in parenthesis indicate interaction between components of the central server - refer to Fig 33 To perform this task the Student would perform the following steps

1) Student Logs into the Central Server The Student uses their Client browser (41) to access the Central Server 42 The server engine (43) will respond by sending them a Web page (44) containing the list of all possible schools that they could log into This list is obtained from the Schools DB (45) Once the Student selects the school they wish to log into, they are presented with a login screen The Student enters their user name and password These are validated against values stored in this particular schools user DB (46) The server engine (43) responds to a successful login by sending the client browser a cookie, which is stored on the client machine (47) The server also updates the users statistics updating the last login time in the Stats DB (48) Each time the user attempts some further action, the client browser 41 sends the cookie 47 to the server 43, this is used by the server 43 to validate the user and ensure they can only access functions available to them Other system functions remain inaccessible 2) Upload new content, Request Moderation

The Student (client machine) selects an "Edit Portfolio" (not shown) form the main menu bar (not shown) The Student selects "Files" (not shown) from the submenu The Student clicks the browse button (not shown), then selects a file (49) (from their local machine) to upload to the server from the open dialog which appears The Student names the 'link' which represents this file in their portfolio, then clicks the 'Upload' button. A successful upload places the new file in this particular School's file system (50) where it can be approved by a Teacher - a process know as moderation. It is not currently part of any published (i.e. publicly accessible) portfolio. The student submits this portfolio update for moderation by a Teacher by clicking on the "Submit" submenu (not shown), then filling out the message form, before finally clicking on the 'Send' (not shown) button.

3) Teacher perform Moderation function and Approves update

The Teacher logs into the Central Server and approves this Student's update. Once the update is approved, the new portfolio content is made accessible to anyone with the correct portfolio password. Local Server: An illustration of a local server is shown in Fig. 34. A local server may be situated at each school. Teachers use the local server associated with their school to access the 'Web Data Entry' feature to update Student marks in the Markbook. A school does not require a local server if it does not use the 'Web Data Entry' feature.

For the purposes of illustration, the following example scenario will be used with reference to Fig. 34, in which a teacher is able to update test results in the Markbook. Numbers enclosed in parenthesis indicate interaction between components of the local server.

To perform this task the Teacher would perform the following:

The teacher Logs into the Local Server. The Teacher uses their Client browser (52) to access the Local

Server. The server engine (51) responds by sending them a Web page (53) inviting them to log in. The Teacher enters their username and password. These are validated against values stored in the Assessor DB (54). The server engine (51) responds to a successful login by sending the client browser a cookie 55, which is stored on the client machine.

Each time the teacher attempts some further action, the client browser 52 sends the cookie to the server engine 51 ; this is used by the server engine 51 to validate the teacher and ensure they can only access functions available to them.

Selection of 'Class' and 'Report Template'

The teacher can select the appropriate class and report template from popup forms on the web page 53. The currently selected Mark book function is Outcome Achievements' (not shown). Once the teacher selects a class and report template, the current list of students and their marks are fetched from the Markbook DB (54). Select appropriate Mark book Function

Since the Teacher actually wants to enter in test results, the Teacher clicks on the 'Test Results' Tab (now shown). The appropriate web page (53) is displayed, and current values of any test scores are fetched from the Markbook DB (54) Entering of test scores

The teacher enters scores in the appropriate editable cells (Results for more than one student can be entered without requiring each to be saved individually)

Saving Test Results The teacher clicks on the 'Update' button

The server engine (51) first validates the entries (entered marks lie between 0 to 100) before updating the required Mark book database entries (54)

Advantages

It will be appreciated that the present embodiment provides a number ot the unique advantages For example, the ability to use pre-defined or user-defined groupings and hierarchies with the assessment information allows groups of assessment criteria to be recorded in the outcome bank and for groups of different types of assessment criteria to be bought together to form a template provides flexibility in the collection and reporting system Additionally, the ability to add new pre-defined or user-defined assessment criteria into a bank ot criteria and the ability to add pre-defined or user-defined groupings and hierarchies of criteria as attributes of the criteria bank further enhances the flexibility of the system in generating a number of unique reports for a number ot students

The collection and reporting system saves time for the Assessor, because the Assessors may use one report layout tor many sets of assessment results or to use one report template (collection of assessment criteria) with many classes (assessee groups) By using a computer to store assessment criteria, assessors and assesses in a structured manner with defined hierarchies, allows the user to follow a simple assessment and reporting process Furthermore, as the system allows the Assessor to view pre-defined presentations of assessment information and assessment results information in a multiple of ways, the assessor is assisted in the preparation, organization, and checking ot the collection of assessment results The system also saves time for the Assessor by allowing the templates to be saved, copied and altered for re-use Furthermore, the scan sheets and mark book are constructed to make the collection and recording of assessment results more efficient Assessment results can be presented in a variety of pre-defined or user-defined formats and Assessors can create their own presentation formats and store them for reuse

The use of the 'Wizard' aids in the setting up of the system and assessment information The Assessor may assign one layout to multiple class/template combinations, thereby allowing users to use report layouts for one or more groups of assessment results

The production of reports is simplified as REPORT CARD allows you to group and sort report layouts of assessment results by a number of different attributes to aid in production and distribution REPORT CARD'S rollover feature which allows you to carry forward assessment information and configurations to use in a new assessment period

REPORT CARD also allows you to assign different symbols to grades depending where that grade will be shown in the system REPORT CARD'S maintenance feature checks for potential problems in data imports, thereby providing database integrity

REPORT CARD also allows you to move a student, with their already recorded assessment results, from one class to another

REPORT CARD allows Assessors to create forms for the collection of assessment results in the first instance ( the first generation of the form ), then decide they would like to change the assessment information that can be collected and create a new version of the form ( the second generation of the form ) where both versions (generations ) of the form can still be used and the assessment results will still be recorded in a meaningful way

REPORT CARD permits the Assessor to view the scanned image of the form and also a table ot the interpreted results from that scanned image and then make corrections to errors where required Additionally, REPORT CARD allows both selecting general comments from a 'bank' ot pre-defined general comments or entering free form general comments or using a combination of both methods

Other Applications

It will be appreciated that although a assessment system in the above embodiment relates to a an assessment of school pupils performing the tasks of school work, it will be appreciated that other assessment systems may be covered by the present invention An overview of a number of alternative assessment applications will now be described with reference to Fig 35 to Fig 37

Turning first to Fig 35 there is illustrated a block diagram the general assessment of assessment criteria Assessment criteria 1', 2' and 3 includes key performance indicators and supporting materials In this embodiment, a comment generator 2' is included Assessment data also includes assessment subject data 4' and assessment subject groupings 5' Assessor data 6' is provided to assist assessors in collecting assessment data

The data 1 ', 2', 3', 4', 5' and 6' is input into a software interface - the "template wizard" 7', which takes the user through a step by step process of selecting assessment criteria, etc and producing an assessment data collection form tor use by an assessor Thus template wizard 7' interfaces with form designer 8' which then generates the assessment data collection form, in this embodiment taking either the form ot an optical mark reader data collection form 9' or a digital scanner data collection form 10'

At this stage, an assessor provides human intervention, which involves assessing activities and applying professional judgment as shown at 1 1 ' Form 9' and 10' are scanned and interpreted as shown by 12' and 13' Entry may also be via network or Internet, as shown at 14' and 15'

The assessment data in relation to the target is analysed, on the basis of the key performance indicators, etc , by report manager 16' Report manager 16' may also take into account additional data 17' Report manager 16' can generate electronic reports 18' or paper reports 19' Report manager 16' can also generate action documents such as follow up letters, report and work orders 20' If desired, report manager 16' can provide performance analysis tools 21' and can attend to data archiving as shown at 22'

Turning now to Fig 36, it will be readily seen that this is an adaptation of the general system of Fig 35 to the specific assessment by a council of the sate of its roads Thus assessment criteria 31 ', 32', 33', 34', 35' and 36' are input into template wizard 37' which interfaces with form designer 38' As in the previous example, form designer 38' generates forms 39' or 40' and an assessor enters assessment activities using professional judgment as shown at 41' The assessment data are scanned and interpreted at 42' and 43' and analysed by report manager 44' Report manager 44' may incorporate data from previous road repairs data base 45' Report manager 44' generates electronic reports 46' or paper reports 47' and may also generate work orders 48' Performance analysis tools 39' and data archiving 50' are also an option

Turning now to Fig 37, this is a version of the system in Fig 35 which has been customised for personnel use The assessment criteria includes desired attributes 51 ', a comment bank 52' and interview questions 53' as well as applicant data 54', job categories 55' and personnel officer's data (assessor's data) 56' This data is input into template wizard 57' which, as before, interfaces with form designer 58' to generate form 59' or 60' On form 59' or 60' a personnel officer enters the assessment activities using professional judgment as shown at 61 ' and the resultant data is scanned and interpreted as shown at 62' and 63' and input into report manager 64' Report manager 64' also analyses the data captured from the target's resume 65'

Report manager 64' generates electronic reports 66' or paper reports 67' and can also generate performance analysis tools 68' and attend to data archiving 69' Importantly, report manager 64' can automatically generate a rejection letter or an invitation for a second interview letter 70'

Hence, in light of the above examples, it will be appreciated that the reporting method can be applied to assessment applications other than students, such as a unit process in an industrial process The reporting method thus has may applications production processes such as in mining processes, beverage processes, manufacturing processes Additionally, the reporting method also has applications in the services industry, such as legal and accounting services, cleaning services and maintenance services etc

The above described embodiments provide a convenient method of reporting upon the performance of a subject upon completion of assigned tasks Generation of the reports is made convenient to the Assessor because it is the Assessor who permitted to define a set of performance criteria tor the assessment of the assigned tasks

It would be appreciated by a person skilled in the art that numerous variations and/or modifications may be made to the present invention as shown in the specific embodiments without departing from the spirit or scope ot the invention as broadly described. The present embodiments are therefore, to be considered in all respects to be illustrative and not restrictive.

Claims

Claims

1 A method for collecting data tor use in reporting the performance of a subject upon completion of one or more assigned tasks, said method comprising the steps of

(a) permitting an assessor to define a set of performance criteria for the assessment of one or more assigned tasks to be completed by one or more subjects,

(b) allowing said assessor to assess the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks, and

(c) recording the results of said assessed performance in step (b) 2 A method as claimed in claim 1 , wherein said method further includes the step of

(d) reporting the results of said assessed performance in a report

3 A method as claimed in claim 2, wherein said method further includes the step of

(e) permitting said assessor to select the format of said report from a multiplicity of formats or, to design the format of said report 4 A method as claimed in claim 2 or claim 3, wherein for a plurality of subjects, said method further includes the step of

(f) permitting said assessor to group each subject according to a particular category

5 A method as claimed in any one of claims 2 to 4, wherein said method further includes the step of

(g) grouping different types of assessment criteria to thereby form a template 6 A method as claimed in any one of the preceding claims, wherein said method further includes the step of

(h) permitting an assessor to amend said set of performance criteria

7 A method as claimed in any one of the preceding claims, wherein said method further includes the step ot (ι) permitting an assessor to define one or more quality values to said to the results of said set of performance criteria to thereby determine the quality of each subject or the quality of a group of subjects

8 A method as claimed in any one of claims 2 to 7, wherein said method further includes the step of

(j) providing a facility to allow the assessor to make general comments in relation to the performance ot said assigned task 9 A method as claimed in any one of claims 2 to 7, wherein said method further includes the step ot (k) wherein a plurality of pre- recorded general comments are available tor selection by said assessor in evaluating the performance of said assigned task

10 A method as claimed in claim 8 or claim 9, wherein said assessor is permitted to make a selection ot said general comments in relation to the performance of a subject, and thereafter rank the general comments so that they are read in said rank on said report

11 A method as claimed in any one of claims 2 to 10, wherein said method further includes the step ot (1) statistically analysing the results of the assessed performance against an assessor defined set

12 A method as claimed in claim 11 , wherein said defined set is a plurality of subjects

13 A method as claimed in claim 11, wherein at least part of said statistical analysis is included as statistical information in said report

14 A method as claimed in any one of claims 2 to 13, wherein for a plurality of subjects, said report includes a comparison of the results of said assessment

15 A method as claimed in any one of claims 2 to 14, wherein in step (c), the results are recorded in a database 16 A method as claimed in claim 15, wherein the results entered into the database are input by using a data collection form

17 A method as claimed in claim 16, wherein the data collection form is read by a digital scanner adapted for Optical Character Recognition (OCR) and Optical Mark Recognition (OMR)

19 A method as claimed in claim 15, claim 16 or claim 17, wherein the results entered into the database are input by using a device having a digital processor

20 A method as claimed in any one ot the preceding claims, wherein said subject are people

21 A method as claimed in claim 20, wherein said people are school students and said assessor is a teacher

22 A method as claimed in any one of claims 1 to 19, wherein said subject are production units in an process

23 A method as claimed in any one of claims 2 to 22, wherein said report is printed on paper

24 A method as claimed in any one of claims 2 to 22, wherein said report is available as an electronic file, audio message file or video file

25 A method as claimed in any one of claims 2 to 24, wherein external data is included in said report from external data sources, said external data may include any one or more ot the following data types Assessee data,

Assessor data, Assessment criteria data, Groupings ot assessment criteria data, Hierarchies of assessment criteria data 26 A method as claimed in any one ot the preceding claims, wherein said pre-defined assessment criteria are recorded in said database for use in a future assessment session

27 A method as claimed in claim 15, wherein the results entered into the database are input by using a device having a digital processor 28 A method as claimed in claim 27, wherein the digital processor is able to exchange data with the database over a communications network

29 A method as claimed in claim 28, wherein the communications network is the Internet

30 A method as claimed in claim 27, wherein the format of said report is automatically generated by said digital processor upon instructions of an application program, wherein the application program causes the digital processor to select particular formats for said report, dependent upon the assessment criteria selected by the Assessor

31 A computer network memory storing thereon an application program for controlling the execution of a processor for collecting data for use in reporting the performance of a subject upon completion of one or more assigned tasks, the computer program controlling the processor to permit an assessor to define a set of performance criteria for the assessment of one or more assigned tasks to be completed by one or more subjects, allow said assessor to assess the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks, and record the results of said assessed performance 32 A computer network memory as claimed in claim 31, wherein said computer program further controls the processor to report the results of said assessed performance in a report

33 A computer network memory as claimed in claim 32, wherein said computer program further controls the processor to permit said assessor to select the format of satd report from a multiplicity of formats or, to design the format of said report

34 A data collection system tor use in reporting the performance of a subject upon completion of one or more assigned tasks, said reporting system comprising an application program having a set a set of performance criteria defined by an assessor for the assessment of one or more assigned tasks to be completed by one or more subjects, data storage means for recording assessment data recorded by said application program, said assessment data representing the results of said assessed performance, and assessor input means capable of interfacing with said application program for allowing said assessor to assesses the performance ot at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks

35 A reporting system as claimed in claim 34, wherein said system further includes a report generating means for reporting the results of said assessed performance in a report and to permitting said assessor to select the format ot said report from a multiplicity of formats or, to design the format of said report

36 A method for reporting the performance of a subject upon completion of one or more assigned tasks, said method comprising the steps of

(a) permitting an assessor to define a set of performance criteria tor the assessment of one or more assigned tasks to be completed by one or more subjects,

(b) allowing said assessor to assess the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks,

(c) recording the results of said assessed performance in step (b), and

(d) reporting the results of said assessed performance in a report 37 A computer network memory storing thereon an application program for controlling the execution ot a processor for reporting the performance of a subject upon completion ot one or more assigned tasks, the computer program controlling the processor to permit an assessor to define a set of performance criteria for the assessment of one or more assigned tasks to be completed by one or more subjects, allow said assessor to assess the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks, record the results of said assessed performance, and report the results of said assessed performance in a report

38 A reporting system for reporting the performance of a subject upon completion of one or more assigned tasks, said reporting system comprising an application program having a set a set of performance criteria defined by an assessor for the assessment of one or more assigned tasks to be completed by one or more subjects, data storage means for recording assessment data recorded by said application program, said assessment data representing the results of said assessed performance, assessor input means capable ot interfacing with said application program for allowing said assessor to assesses the performance of at least one task completed by said one or more subjects according to said defined performance criteria, during or upon completion of, said assigned tasks, and a report generating means for reporting the results of said assessed performance in a report 39 A method as claimed in claim 2, wherein a pre-defined comment in relation to the subject is recommended to said Assessor, based upon the assessment of said subject's performance

40 A method of reporting the performance of a subject upon completion of one or more assigned tasks, substantially according to any one of the examples described herein with reference to the accompanying drawings

41 A computer network memory storing thereon an application program for controlling the execution ot a processor for reporting the performance of a subject upon completion of one or more assigned tasks, substantially as herein described with reference to the accompanying drawings

42 A reporting system for reporting the performance ot a subject upon completion of one or more assigned tasks, substantially as herein described with reference to the accompanying drawings