WO2014092535A1

WO2014092535A1 - System and method for the evaluation of effectiveness of learning content

Info

Publication number: WO2014092535A1
Application number: PCT/MY2013/000231
Authority: WO
Inventors: Sean LIM YING; Chyi LEW CHIA; Sefat MANI HAGH; Selan NOR EZAM; Lukose Dickson
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2012-12-11
Filing date: 2013-12-05
Publication date: 2014-06-19
Anticipated expiration: 2015-06-11

Abstract

A system (100) for the evaluation of effectiveness of learning content comprising: a learning content repository (101) containing learning content; an assessment repository (102) that contains assessment questions; a learning activity storage module (106) that stores learning activity of a learner (104); a feedback storage module (107) for storing feedback from the learner; a sentiment analysis module (109) adapted to analyse the feedback; an assessment results module (108) adapted to receive assessment results based on the assessment questions and analyse the assessment results; and an evaluation engine (110) adapted to generate an evaluation report on the basis of (i) the learning activity of the learner (104), (ii) sentiment analysis information generated from the feedback of said learner (104), and (iii) assessment result analysis information generated from the assessment results.

Description

SYSTEM AND METHOD FOR THE EVALUATION OF EFFECTIVENESS OF LEARNING CONTENT

FIELD OF INVENTION

The present invention relates to a system and method for the evaluation of the effectiveness of learning content. In particular, the invention relates to a system and method that conducts automatic evaluation based on feedback from a learner and assessment results of the learner.

BACKGROUND ART

When students fail an assessment, there are currently few methods available that enable the identification of the root cause of the failure. As an unfortunate result, a student may be categorised as a bad student. However, potential root causes for failure may extend beyond the particular capabilities of the student. For example, failure may be due in part or in whole to poor design of learning content for the course being studied, or poor preparation of the learning content. Current methods for evaluation of learning content focus primarily on content quality and content aesthetics. However, most of these methods do not consider the context of the learning content and whether this provides the leaning content with superior effectiveness, or whether this is lacking and the content therefore does not impart quality knowledge to the students.

United States Patent Publication No. US 2004/0024776 A1 describes a system and method for managing content that utilizes a relational database management system. The relational database management system organizes materials such as curriculum materials into, for example, an organizational hierarchy. The system and method allows a user to align materials to a variety of standards including, for example, local, school district, state, and national standards. The system and method also enables the assessment of student performance to determine which areas of a curriculum are meeting standards and which require improvement and modification. The system and method further enables the user to assess the effectiveness of a curriculum and curriculum materials by integrating student performance data with such factors as curriculum materials used, assignment histories, and student profile data. Assessment of student performance may also be achieved by integrating data from a first plurality of different user-sites, for example, a first plurality of schools within a school district or a first plurality of school districts within a state. The method and system described, however, does not facilitate feedback relating to learning content and does not provide an indication of the level of a learner's understanding of the learning content. Also, learning content selection is based on learner search input, learning content interaction is not stored and the method and system does not take account of the learner's particular learning style.

United States Patent No. 7,860,736 describes learning solutions that utilize a content development module that is nested inside a total learning solution and which identifies, develops, and manages required course content. The content development module is particularly adapted to assemble and compile custom learning content materials for use in the learning solution by appropriately taking business strategy alignment and other information into consideration. The learning content may be developed as a third party service in accordance with a service level agreement. A service level agreement is a contract between the learning solution services provider and the learner (i.e., outsourcing) organization that establishes the requisite quality and effectiveness levels of the learning solution services as measured by one or more performance metrics. In such cases where a service level agreement is present, internal performance metrics are constantly evaluated to ensure that the content development and management resources necessary for a given learning effort are properly reserved, allocated and/or utilized. However, the learning solutions described involve feedback from the instructor, rather than from a learner. Assessment results, generally a fail or pass, are based on a predefined level of acceptable performance. All learning content is provided to the learner and learning content interaction is not stored. Again, the learner's particular learning style is not taking into account. It would be advantageous if a system and method could be devised that analyses the quality and aesthetics of the learning content with sentiment analysis and which identifies contextual knowledge that is missing in the learning content.

The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practice.

SUMMARY OF INVENTION

One aspect of the present invention provides a system for the evaluation of effectiveness of learning content comprising:

a learning content repository containing learning content;

an assessment repository that contains assessment questions;

a learning activity storage module that stores learning activity of a learner;

a feedback storage module for storing feedback from the learner;

a sentiment analysis module adapted to analyse the feedback;

an assessment results module adapted to receive assessment results based on the assessment questions and analyse the assessment results; and

an evaluation engine adapted to generate an evaluation report on the basis of (i) the learning activity of the learner, (ii) sentiment analysis information generated from the feedback of the learner, and (iii) assessment result analysis information generated from the assessment results.

Another aspect of the invention provides a system wherein the assessment questions are annotated with one or more of metadata, learning objectives, difficulty level and student level of understanding hierarchy (SLOUH).

A further aspect of the invention provides a system wherein the assessment results module is adapted to identify weaknesses of a learner by (i) selecting assessment questions the learner failed to answer correctly, (ii) identifying the selected assessment questions using a student level of understanding hierarchy (SLOUH) model, and (iii) identifying learning objectives fulfilled by the selected assessment questions.

Yet another aspect of the invention provides a system wherein the evaluation engine is adapted to calculate the percentage of learners that pass the assessment, determine whether the percentage of learners is below a predetermined threshold value, and, if the percentage is below the threshold value, identifying the highest SLOUH model category which cannot be answered. Another aspect of the invention provides a method for the evaluation of effectiveness of learning content comprising:

providing a learner with learning content;

monitoring learning activity of the learner;

receiving feedback from the learner on the learning content and performing sentiment analysis on the feedback;

assessing the learner by providing the learner with assessment questions and analysing assessment results provided by the learner to the assessment questions to identify weaknesses of the learner;

evaluating the learning activity, sentiment analysis and assessment results to determine the effectiveness of the learning content.

Yet another aspect of the invention provides a method wherein the assessment questions are annotated with one or more of metadata, learning objectives, difficulty level and student level of understanding hierarchy (SLOUH).

A further aspect of the invention provides a method wherein the monitoring of the learner activity comprises monitoring session time and/or repetition of review of the learning content. Another aspect of the invention provides a method wherein the analysing the assessment results comprises (i) selecting assessment questions the learner failed to answer correctly, (ii) identifying the selected assessment questions using a student level of understanding hierarchy (SLOUH) model, and (iii) identifying learning objectives fulfilled by the selected assessment questions, thereby identifying weaknesses of the learner.

A further aspect of the invention provides a method wherein the evaluating comprises calculating the percentage of learners that pass the assessment, determining whether the percentage of learners is below a predetermined threshold value, and, if the percentage is below the threshold value, identifying the highest student level of understanding hierarchy (SLOUH) model category which cannot be answered.

The present invention consists of features and a combination of parts hereinafter fully described and illustrated in the accompanying drawings, it being understood that various changes in the details may be made without departing from the scope of the invention or sacrificing any of the advantages of the present invention.

BRIEF DESCRIPTION OF ACCOMPANYING DRAWINGS

To further clarify various aspects of some embodiments of the present invention, a more particular description of the invention will be rendered by references to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail through the accompanying drawings in which: FIG. 1 illustrates the system architecture of an embodiment of the system of the invention.

FIG. 2 illustrates a flow diagram of the method of an embodiment of the invention. FIG. 3 illustrates a flow diagram of the process flow for an evaluation engine of an embodiment of the system of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, this specification will describe the present invention according to the preferred embodiments. It is to be understood that limiting the description to the preferred embodiments of the invention is merely to facilitate discussion of the present invention and it is envisioned without departing from the scope of the appended claims.

Referring to Figure 1 , the system architecture of a system (100) according to an embodiment of the invention is illustrated. The system (100) includes a learning content repository (101) that contains relevant learning content and an assessment repository (102) that contains questions that are annotated with metadata, learning objectives, difficulty levels and student level of understanding hierarchy (SLOUH) (103). A learner (104) is provided with the learning content and the learning process (105) is initiated. During the learning process (105), the learning activity of the learner (1040 is stored in learning activity storage (106). The learner (104) provides feedback on the learning content that has been provided to them and that feedback is stored in feedback storage (107). The feedback is analysed through a sentiment analysis module (109). The learning activity and the sentiment analysis value are then evaluated through an eval uation engi ne (110).

When the learner (104) completes his or her course, that is has completed the learning content, the learner (104) will be prompted to sit an assessment based on the assessment questions stored in the assessment repository (102) resulting in assessment results (108). The assessment results are also evaluated using the evaluation engine (110). The learning activity, feedback and assessment results are used to compute the effectiveness of the learning content and an evaluation report (111) is generated on this basis. Turning to Figure 2, the methodology of an embodiment of the invention is illustrated. In this embodiment, the method (200) involves selection of learning content (201) generally based on the learning style of the learner. Once the learning process is established, the method includes storing feedback (202) received from the learner and storing learning activities (203), of the learner, such as session times, repetitive nature of learning and so on.

If it is determined (204) that studies are incomplete, the process continues with learning content being selected (201) based on the learning style of the learner until the studies are complete. Once complete, the learner is assessed (205) and the assessment results processed (206) to identify any perceived weakness of the learner. All assessment scores for learners that have been provided with the same learning content are obtained and the percentage of learners who have failed the assessment is calculated. Based on SLOUH, the weaknesses of the learning content are identified. Content quality and content aesthetics are identified on the basis of the learner feedback on the learning content. Feedback received from the learner is analysed (207) using sentiment analysis to arrive at a sentiment analysis value.

As discussed above, on the basis of the sentiment analysis (207) results, learning activities of the learner and assessment results, the method (200) involves evaluation

(208) of the effectiveness of the learning content and generation of an evaluation report

(209) .

With reference to Figure 3, the evaluation engine (110) of the system (100) and the methodology of evaluation of learning activities (301), assessment results (302) and sentiment analysis values (303) are illustrated in more detail. The learning activities, such as session times and repetition, are processed (304). The assessment scores for learners that have been provided with the same learning content are obtained (305) and the percentage of learners that passed the assessment is calculated (306). If the percentage of learners is below a predetermined threshold value (307), the learning content is considered ineffective and the highest SLOUH model category which cannot be answered is identified (308). A report is then complied (309) on the basis of the inputted information and the SLOUH model category identified (308). If the percentage of learners is not below the predetermined threshold value (307), the effectiveness of the learning content is considered satisfactory and the report is compiled (309). An evaluation report is then generated (310). Unless the context requires otherwise or specifically stated to the contrary, integers, steps or elements of the invention recited herein as singular integers, steps or elements clearly encompass both singular and plural forms of the recited integers, steps or elements. Throughout this specification, unless the context requires otherwise, the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated step or element or integer or group of steps or elements or integers, but not the exclusion of any other step or element or integer or group of steps, elements or integers. Thus, in the context of this specification, the term "comprising" is used in an inclusive sense and thus should be understood as meaning "including principally, but not necessarily solely".

It will be appreciated that the foregoing description has been given by way of illustrative example of the invention and that all such modifications and variations thereto as would be apparent to persons of skill in the art are deemed to fall within the broad scope and ambit of the invention as herein set forth.

Claims

1. A system (100) for the evaluation of effectiveness of learning content comprising a learning content repository (101) containing learning content;

an assessment repository ( 02) that contains assessment questions; a learning activity storage module (106) that stores learning activity of learner;

a feedback storage module (107) for storing feedback from said learner; a sentiment analysis module (109) adapted to analyse said feedback; an assessment results module ( 08) adapted to receive assessment results based on said assessment questions and analyse said assessment results; and

an evaluation engine (110) adapted to generate an evaluation report on the basis of (i) said learning activity of said learner, (ii) sentiment analysis information generated from said feedback of said learner, and (iii) assessment result analysis information generated from said assessment results.

2. A system according to claim 1 , wherein said assessment results module (108) is adapted to identify weaknesses of a learner by (i) selecting assessment questions said learner failed to answer correctly, (ii) identifying said selected assessment questions using a student level of understanding hierarchy (SLOUH) model, and (iii) identifying learning objectives fulfilled by said selected assessment questions.

3. A system according to claim 1 , wherein said evaluation engine (110) is adapted to calculate the percentage of learners that pass the assessment, determine whether said percentage of learners is below a predetermined threshold value, and, if said percentage is below said threshold value, identifying the highest SLOUH model category which cannot be answered. A method (200) for the evaluation of effectiveness of learning content comprising providing a learner with learning content (201);

monitoring learning activity of said learner (203);

receiving feedback from said learner on said learning content (202) and performing sentiment analysis on said feedback (207);

assessing said learner by providing said learner with assessment questions and analysing assessment results provided by said learner to said assessment questions to identify weaknesses of said learner (205); and

evaluating said learning activity, sentiment analysis and assessment results to determine the effectiveness of said learning content (208).

A method (200) according to claim 4, wherein said assessment questions are annotated with one or more of metadata, learning objectives, difficulty level and student level of understanding hierarchy (SLOUH).

A method (200) according to claim 4, wherein said monitoring of said learner activity comprises monitoring session time and/or repetition of review of said learning content.

A method (200) according to claim 4, wherein said analysing said assessment results comprises (i) selecting assessment questions said learner failed to answer correctly, (ii) identifying said selected assessment questions using a student level of understanding hierarchy (SLOUH) model, and (iii) identifying learning objectives fulfilled by said selected assessment questions, thereby identifying weaknesses of said learner.

A method (200) according to any of claim 4, wherein said evaluating comprises calculating the percentage of learners that pass the assessment, determining whether said percentage of learners is below a predetermined threshold value, and, if said percentage is below said threshold value, identifying the highest student level of understanding hierarchy (SLOUH) model category which cannot be answered.