WO2011122930A1

WO2011122930A1 - Assessment system and method for innovative learning

Info

Publication number: WO2011122930A1
Application number: PCT/MY2010/000185
Authority: WO
Inventors: Yon Haniza; Pek Kuan Ng; Lukose Dickson
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2010-03-29
Filing date: 2010-09-30
Publication date: 2011-10-06
Anticipated expiration: 2012-09-29
Also published as: MY150352A

Abstract

The present invention relates to an assessment system (102) and method for innovative learning in an e-learning system which (102) comprises a profiling module (104), a planning module (106), a display module (108) and an assessment module (110). The profiling module (104) comprises means for profiling the user based on a psychometric profile. The planning module (106) comprises means for selecting learning material based on the psychometric profile. The display module (108) comprises means for displaying the learning material to the user. The assessment module (110) comprises means for assessing the user by randomly selecting a plurality of questions for display to the user, scoring a plurality of answers from the user using weighted scoring, determining a proficiency level of the user and suggesting a learning pathway to the user based on the proficiency level.

Description

ASSESSMENT SYSTEM AND METHOD FOR INNOVATIVE LEARNING

FIELD OF INVENTION The present invention relates to an assessment system and method for innovative learning using an e-Iearning system.

BACKGROUND ART Current assessment systems, particularly e-assessment systems, do not fully utilize users' or students' results to improve their learning effectively. At present, the results are merely used to provide feedback on students' understanding of the lesson and not used to automatically provide remedial lessons in areas in which students still appear to be less proficient. Therefore, there is a lack of personalization of learning paths.

Moreover, current e-assessment systems present the same set of questions to students who learn the same topic or chapter during every stage of assessment; regardless of how many times the students have seen or learnt the topic or chapter. Therefore, it is very likely that students who learn the same topic or chapter will be able to cheat during assessment or share answers with each other. This is known as the memory effect, and it can have a significant impact on assessment. The memory effect is especially important when students are being re-assessed, and it can compromise accurate estimation of each student's proficiency level. In addition, for ease of calculation, current assessment systems typically assign equal weights to multiple-choice questions in the scoring formula, regardless of the difficulty level of the questions. Moreover, the distribution of questions in terms of difficulty level is loosely determined, while the score range for the different proficiency levels are set to be the same across all assessments. This practice results in an unfair comparison of proficiency levels between students, especially when the students take different sets of assessments.

SUMMARY OF INVENTION

In one embodiment of the present invention is an assessment system for innovative learning using an e-learning system. The system comprises a profiling module, a planning module, a display module and an assessment module. The profiling module comprises means for profiling the user based on a psychometric profile. The planning module comprises means for selecting learning material based on the psychometric profile. The display module comprises means for displaying the learning material to the user. The assessment module comprises means for assessing the user by randomly selecting a plurality of questions for display to the user, scoring a plurality of answers from the user using weighted scoring, determining a proficiency level of the user and suggesting a learning pathway to the user based on the proficiency level.

In another embodiment of the present invention is an assessment method for innovative learning using an e-learning system. The method comprises the steps of profiling the user based on a psychometric profile, selecting learning material based on the psychometric profile, displaying the learning material to the user and assessing the user by randomly selecting a plurality of questions for display to the user, scoring a plurality of answers from the user using weighted scoring, determining a proficiency level of the user and suggesting a learning pathway to the user based on the proficiency level.

The present invention consists of features and a combination of parts hereinafter fully described and illustrated in the accompanying drawings, it being understood that various changes in the details may be made without departing from the scope of the invention or sacrificing any of the advantages of the present invention. BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS

To further clarify various aspects of some embodiments of the present invention, a more particular description of the invention will be rendered by references to specific embodiments thereof, which are illustrated, in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail through the accompanying drawings in which: FIG. 1 is an illustration of the overview of an e-learning system with an assessment system and method for innovative learning.

FIG. 2 is a flowchart illustrating the development of an assessment engine. FIG. 3 is an illustration of the architecture of an assessment engine.

FIG. 4 is a flowchart illustrating the assessment flow in an assessment engine.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention relates to an assessment system and method for innovative learning. Hereinafter, this specification will describe the present invention according to the preferred embodiments of the present invention. However, it is to be understood that limiting the description to the preferred embodiments of the invention is merely to facilitate discussion of the present invention and it is envisioned that those skilled in the art may devise various modifications and equivalents without departing from the scope of the appended claims.

The assessment system and method for innovative learning of the present invention relates to an e-assessment system and method that utilizes assessment results effectively to improve quality of learning. The assessment system and method provides multiple learning pathways based on the assessment results to cater for individual user's or student's learning needs. For instance, a student will be provided with different remedial lessons depending on their proficiency level gauged based on the assessment results.

The assessment system and method of the present invention also provides different learning pace by allowing students to either proceed to learn a subsequent lesson i.e. the next topic or chapter or to repeat learning areas in the same topic or chapter which students are still weak in.

The assessment system and method for innovative learning of the present invention applies randomized questions and weighted scoring method to more accurately estimate the proficiency level of the students and to promote fair assessment. Randomized questions reduce memory effects or cheating which will result in an unfair comparison of proficiency level between students across all assessments. Randomization comprises setting each assessment from randomly selected questions. However, the distribution of questions with different difficulty levels is pre-determined. Weighted scoring ensures different weights are assigned to questions with different difficulty levels in order to promote fair comparison of proficiency level between students.

In another embodiment of the present invention, the assessment system and method for innovative learning provide personalized learning material and assessment questions are randomly assigned to each individual student based on the student's proficiency level. Personalization is according to students' weak areas and level of mastery. Based on the assessment results, assessment system and method identify the areas that the student is weak in and selects appropriate learning materials for remedial lessons based on the level of mastery in the weak areas. Thereafter, the students will be assessed using questions which are related to their weak areas. Reference is first made to FIG. 1. FIG. 1 is an illustration of the overview of an e-learning system with an assessment system and method for innovative learning. The system (102) comprises a profiling module (104), a planning module (106), a display module (108) and an assessment module (110). The profiling module (104) comprises means for profiling the user based on a psychometric profile (112). The planning module (106) comprises means for selecting learning material based on the psychometric profile (114). The display module (108) comprises means for displaying the learning material to the user (116). The assessment module (110) comprises means for assessing the user by randomly selecting a plurality of questions for display to the user (118), scoring a plurality of answers from the user using weighted scoring (120), determining a proficiency level of the user (120) and suggesting a learning pathway to the user based on the proficiency level. In one embodiment of the present invention, the proficiency level of the user may be determined as either advanced.proficient or basic. There are 2 predefined thresholds i.e. one is between advanced level and proficient level, and the other one is between proficient level and basic level. If the assessment score of the user meets a predefined threshold for the advanced level, then the user may progress as having an advanced level of proficiency. If the assessment score of the user does not meet the predefined threshold for advanced level, then the user progresses as having a basic or proficient level of proficiency. If the proficiency level of the user meets the predefined threshold for the advanced level, the planning module (106) selects a subsequent learning material based on the psychometric profile. The display module (108) then displays the subsequent learning material to the user. The assessment module (110) further assesses the user by randomly selecting a subsequent plurality of questions for display to the user, scoring a subsequent plurality of answers from the user using weighted scoring, determining a subsequent proficiency level of the user and suggesting a subsequent learning pathway to the user based on the proficiency level.

If the proficiency level of the user does not meet the predefined thresholds for the advanced level, the assessment system further identifies a plurality of weak areas of the user based on the plurality of answers from the user (124), assesses a level of mastery for*each of the plurality of weak areas (126) and maps the level of mastery to a remedial lesson selection scheme (128). The planning module (106) then selects a remedial lesson material based on the psychometric profile, the plurality of weak areas and the remedial lesson selection scheme (130) for display to the user (132) by the display module (108). The assessment module (110) selects a plurality of questions based on the plurality of weak areas and a question selection scheme for display to the user (134) the display module (108). The assessment module (110) further scores the plurality of answers from the user using weighted scoring (136), determines the proficiency level of the user and suggests the learning pathway to the user based on the proficiency level.

Reference is now being made to FIG. 2. FIG. 2 is a flowchart illustrating the development of an assessment engine. The development or preparation of the e-assessment engine comprises suggesting annotation properties for the question (202), suggesting schema of the difficulty level (204), suggesting score for each difficulty level (206), annotating questions with difficulty level and other annotation properties (208), suggesting scheme for level of mastery (210), suggesting remedial lesson selection scheme (212), suggesting question selection scheme (214), suggesting proficiency level scheme (216), laying-out weighted scoring algorithm and formulas (218) and finally developing the e- assessment engine (220).

The annotation properties for the plurality of questions are suggested (202) by a domain expert, psychometrician and a researcher based on the curriculum specified by a statutory body. It not only includes academic annotations regarding the content and learning objectives of the syllabus, but also the skills and values which need to be cultivated through learning. The schema of difficulty level is prepared and defined (204) by the domain expert to enable a consistent annotation of the questions. The suggested schema of difficulty level is based on Bloom's taxonomy. The difficulty level is mapped to the cognitive process dimensions in the taxonomy as shown in Table 1. Cognitive Process Dimensions Difficulty Level

Knowledge - Comprehension Basic

Comprehension - Application - Analysis Intermediate

Analysis - Synthesis - Evaluation High / Advanced

Table 1

After the schema for difficulty level is defined (206), a score is assigned to each difficulty level as suggested by measurement experts. An example of score based on difficulty level is as shown in Table 2.

Table 2

All the questions are then annotated with difficulty level and other annotation properties (208) by the domain experts according to the predefined schema.

The scheme for identifying the level of mastery (210) of the identified weak areas is suggested by the domain expert and measurement experts based on the complexity level of the area. Score range that depicts different level of mastery is suggested for each complexity level. Remedial lesson selection scheme (212) is suggested by the domain expert. A mapping between the level of mastery of the identified weak areas to the difficulty level and fulfilled learning outcome of the learning materials is provided for the remedial lesson selection. Question selection scheme (214) is suggested by the domain expert and measurement experts. A control on the distribution of the difficulty level of the question in the test set is made configurable. The questions are selected based on the learning outcomes, difficulty level and whether the questions are answered previously by the students or not. The second round assessment questions are personalized to cater for areas that students are weak in, which have been identified in the previous test.

The proficiency level scheme (216) is suggested by the domain expert and measurement experts by taking into consideration the fact that the score range for each proficiency level might be different depending on the subject-matter and the grade level of study.

The weighted scoring algorithm and formula is laid out (218) clearly to ease the development process. The e-assessment engine is developed (220) to comprise of the test set fetcher module and the test set scorer module. Reference is collectively being made to FIGs. 3 and 4. FIG. 3 is an illustration of the architecture of the assessment engine. FIG. 4 is a flowchart illustrating the assessment flow in the assessment engine.

The test set fetcher module (302) will select and fetch the questions from an item bank (206) to be displayed to the student while the test set scorer module (304) will score the answers submitted by the student based on the predefined scoring model and identify the areas that students are weak in. The test set fetcher module (302) comprises an ontology loader module (308), a scheme loader module (310), a question selection module (312), a randomization module (314) and a question fetcher module (3 6). The ontology loader module (308) loads the item bank into the assessment system (402). The scheme loader module (310) loads all the schemes used in the assessment system (404). The question selection module (312) selects the questions for test set composition (406). The randomization module (314) randomizes the question selection and order of display (408). The question fetcher module (316) fetches the questions and its annotation from the item bank (410).

The test set scorer module (304) comprises an ontology loader module (318), a response collector module (320), an answer retrieval module (322), a scoring module (324), a proficiency level determination module (326), a weak area identification module (328), a level of mastery identification module (330), a mapping of the level of mastery to selection scheme module (332) and an answer key fetcher module (324).

The ontology loader module (318) loads the item bank into the assessment system. The response collector module (320) collects the students' responses or answers (412). The answer retrieval module (322) retrieves answers and scores assigned to the questions from the item bank (414). The scoring module (324) scores the students' responses using weighted scoring (416). The proficiency level determination module {326) determines the proficiency level of the student on a certain topic or chapter based on the score (418). The weak area identification module (328) identifies the weak area of the students (420). The level of mastery identification module (330) identifies the level of mastery of each weak area based on the scheme (422). The mapping of the level of mastery to selection scheme module (332) maps the level of mastery to the selection scheme of the remedial lesson and the subsequent plurality of questions (424). The answer key fetcher module (334) retrieves the answer key from the item bank which consists of the correct answer and the explanation for the answer key (426).

Claims

1. An assessment system (102) for innovative learning, which resides in an e- learning system (102) that comprises

a profiling module (104);

a planning module (106);

a display module (108); and

an assessment module (110);

characterized in that

the profiling module (104) comprises means for profiling the user based on a psychometric profile;

the planning module (106) comprises means for selecting learning material based on the psychometric profile;

the display module (108) comprises means for displaying the learning material to the user; and

the assessment module (110) comprises means for assessing the user by: randomly selecting a plurality of questions for display to the user; scoring a plurality of answers from the user using weighted scoring; determining a proficiency level of the user; and suggesting a learning pathway to the user based on the proficiency level.

2. The assessment system (102) according to claim 1 , wherein the assessment module (110) comprises means for determining the proficiency level of the user to be advanced, proficient or basic .

3. The assessment system (102) according to claim 1, wherein if the assessment score of the user meets a predefined threshold for the advanced proficiency level,

the planning module (106) comprises means for selecting a subsequent learning material based on the psychometric profile;

the display module (108) comprises means for displaying the subsequent learning material to the user; and

the assessment module (110) further comprises means for assessing the user by:

randomly selecting a subsequent plurality of questions for display to the user;

scoring a subsequent plurality of answers from the user using weighted scoring;

determining a subsequent proficiency level of the user, and suggesting a subsequent learning pathway to the user based on the proficiency level.

4. The assessment system (102) according to claim 1, wherein if the assessment score of the user does not meet the predefined thresholds for the advanced level, the assessment module (110) further comprises means for

identifying a plurality of weak areas of the user based on the plurality of answers from the user;

assessing a level of mastery for each of the plurality of weak areas;

mapping the level of mastery to a remedial lesson selection scheme;

selecting a remedial lesson material based on the psychometric profile, the weak areas and the remedial lesson selection scheme for display to the user;

selecting a plurality of questions based on the plurality of weak areas and a question selection scheme for display to the user; scoring the plurality of answers from the user using weighted scoring;

determining the proficiency level of the user; and

suggesting the learning pathway to the user based on the proficiency level.

5. An assessment method for innovative learning, the method comprises the steps of:

profiling the user based on a psychometric profile (112);

selecting learning material based on the psychometric profile (114);

displaying the learning material to the user (116); and

assessing the user by:

randomly selecting a plurality of questions for display to the user (118);

scoring a plurality of answers from the user using weighted scoring (120);

determining a proficiency level of the user; and suggesting a learning pathway to the user based on the proficiency level.

6. The assessment method for innovative learning according to claim 5, the method further comprises determining the proficiency level of the user to be advanced, proficient, or basic.

7. The assessment method according to claim 5, wherein if the assessment score of the user meets a predefined threshold for the advanced proficiency level, the method further comprises the steps of:

selecting a subsequent learning material based on the psychometric profile;

displaying the subsequent learning material to the user; and assessing the user by:

randomly selecting a subsequent plurality of questions for display to the user;

scoring a subsequent plurality of answers from the user using weighted scoring;

determining a subsequent proficiency level of the user; and suggesting a subsequent learning pathway to the user based on the proficiency level.

8. The assessment method according to claim 5, wherein if the assessment score of the user does not meet the predefined threshold for the advanced level, the method further comprises the steps of:

identifying a plurality of weak areas of the user based on the plurality of answers from the user (124);

assessing a level of mastery of the plurality of weak areas (126);

mapping the level of mastery to a remedial lesson selection scheme (128); selecting a remedial lesson material based on the psychometric profile, the plurality of weak areas and the remedial lesson selection scheme (130) for display to the user (132);

selecting a plurality of questions based on the plurality of weak areas and a question selection scheme for display to the user (134);

scoring the plurality of answers from the user using weighted scoring

(136);

determining the proficiency level of the user; and

suggesting the learning pathway to the user based on the proficiency level.