WO2016105182A1

WO2016105182A1 - Method for automatic diagrammatic question generation and answer assessment

Info

Publication number: WO2016105182A1
Application number: PCT/MY2015/050058
Authority: WO
Inventors: Qiang Liu; Min Xian Benjamin CHU; Dickson Lukose; Khalil BOUZEKRI
Original assignee: Mimos Bhd
Current assignee: Mimos Bhd
Priority date: 2014-12-22
Filing date: 2015-06-16
Publication date: 2016-06-30
Anticipated expiration: 2017-06-22
Also published as: MY190013A

Abstract

Automated method for diagrammatic question generation and answer assessment. The method includes providing at least one topic and at least one question based on the topic (100), providing at least one range of a question difficulty level (QDL) (100) and a question mode, proportioning question difficulty type (QDT) based on the range of question difficulty level (101), selecting at least one question template based on the proportioned question difficulty type (QDT) and question difficulty level (QDL) (105), and generating a question based on the selected template (106). In one embodiment, the question mode includes questions in the form of diagrammatic multiple-choice type (DMCQ) and/or hand drawing diagrammatic type (HDDQ).

Description

METHOD FOR AUTOMATIC DIAGRAMMATIC QUESTION GENERATION

AND ANSWER ASSESSMENT

FIELD OF INVENTION

[0001] The present invention generally relates to generating question types, and more particularly to a method for automatically generating diagrammatical questions and answer assessment for various purposes.

BACKGROUND OF INVENTION

[0002] One of the countless ways for effective learning of the knowledge paradigms and schemes for various applications or purposes is to enquire and thus understand the relationship between the answers provided based on the enquiries made on a certain topic or subject matter. In this technique, at least a set of questions is provided to a participating individual to assess the individual's understanding of a certain topic based on the answers provided. Such method can be generally referred to as diagrammatic reasoning or diagrammatic learning.

[0003] At present, apart from the arduous manually generated questions, the existing methods and techniques have partially evolved and are primarily focused on semiautomatic and automatic question assessments. Nevertheless, a great majority of the existing automatic methods suffer from several shortcomings including in many cases the diagrammatic questions and solutions still have to be created manually, thus can be ludicrously time consuming. In addition to this, the automatic answer assessment is performed based on one or more manually created solution diagrams. Other existing methods only offer limited question modalities thus may result to restricting the depth of assessment outcome and accuracy.

[0004] US Patent Publication no. US20090280456 Al discloses a system and method for generating family of question for a programming language. Specifically, the system is adapted for generating conventional multiple choice question (MCQ). The system offers template questions having one or more variables, where the system generates a question by assigning fix values to the template questions, and automatically generates a correct answer corresponds the generated question and one or more incorrect answer. The system is limited to conventional MCQ only, it is not adapted to handle diagrammatical questions and answer rendering.

[0005] US Patent Publication no. US20130196305 Al discloses a method and apparatus for generating question based on a generated graph. Based on the generated graph, the system and method synthesize a textual representation of the generated graph to render a question. The graph is used to render the textual question, the questions rendered by the system is not in diagrammatical form.

[0006] Hence it would be highly desirable to have a method and system that can automatically generate diagrammatic questions and with multiple modalities as well as difficulty levels of questions thereby providing a thorough assessment in learning or understanding a topic or subject matter.

SUMMARY

[0007] In one aspect, there is disclosed an automated method for diagrammatic question and answer assessment. The method comprising of providing at least one topic and at least one question based on the topic, providing at least one range of a question difficulty level (QDL), providing at least one question mode, proportioning question difficulty type (QDT) based on the range of question difficulty level, selecting at least one question template based on the proportioned question difficulty type (QDT) and question difficulty level (QDL), generating a question based on the selected template, wherein the question mode includes questions in the form of diagrammatic multiple choice type (DMCQ) or hand drawing diagrammatic type (HDDQ). [0008] In one embodiment, proportioning question difficulty type (QDT) includes providing a predetermined percentage range of the QDT based on the question difficulty level (QDL). [0009] In another embodiment, selecting at least one question template based on the proportioned question difficulty type (QDT) and question difficulty level (QDL) includes selecting the question template based on the question mode of DMCQ or HDDQ and selecting the question template based on mix mode being both DMCQ and HDDQ. [0010] In a further embodiment, generating a question based on the selected template includes providing at least one topic, at least one question difficulty type (QDT) and at least one question template; retrieving at least one topic concept; retrieving a question generation strategy (QGS) based on at least one question mode; generating a question body; generating a question answer; generating a question distractor; and aggregating the question body, answer and distractor.

[0011] In yet a further embodiment, generating a question further includes: generating drawing nodes and arcs; and aggregating the question body and drawing nodes & arcs.

[0012] In a further embodiment, generating a question body further comprises providing at least one topic concept, a question template, and a question generation strategy (QGS); determining whether an instance is required to which if YES, randomly retrieving an instance based on the topic concept and replacing the topic concept with the retrieved instance; if no instance is required; applying at least one type of question generation strategy (QGS) to generate the question body; wherein the question generation strategy (QGS) includes using label or definition or case strategy or a combination thereof; using a correct diagram strategy or filled-in-the-blank strategy or a combination thereof.

[0013] In yet another embodiment, using label or definition or case strategy includes: retrieving at least one label or definition or a case strategy or a combination thereof and aggregating the label or definition or use case with the question template. [0014] In a further embodiment, using a correct diagram strategy includes: retrieving the conceptual structure (CS) based on the topic concept; and aggregating reference of the CS with the question template.

[0015] In yet another embodiment, using filled-in-the-blank strategy includes: retrieving the CS based on the topic concept; randomly selecting at least one node and one arc based on the CS; storing the selected node and arc in a vector list; and aggregating the reference of the CS and vector list with the question template.

[0016] In a further embodiment, generating a question answer includes: providing at least one topic concept, a vector list and at least one generation strategy (QGS); determining whether an instance is required; if YES to requiring an instance, randomly retrieving an instance of the topic concept and replacing the topic concept with the instance ; if NO to requiring an instance; determine whether to use definition or label strategy or correct diagram strategy or nodes or arc strategy in generating the question answer.

[0017] In another embodiment, using definition or label strategy in generating the question answer includes: retrieving at least one definition or a label based on the topic concept.

[0018] In yet a further embodiment, using correct diagram strategy in generating the question answer includes: retrieving the CS for the topic concept. [0019] In another embodiment, using nodes or arcs strategy in generating the question answer includes: retrieving the CS for the topic concept and extracting the nodes or arcs from the CS.

[0020] In yet another embodiment, generating a question distractor includes: providing at least one topic concept, at least one QDT, at least one question generation strategy; determining whether an instance is required; if YES in requiring an instance, randomly retrieving an instance of the topic concept and replacing the topic concept with the instance; if NO in requiring an instance, selecting a QGS; wherein selecting the QGS includes selecting anyone from the group comprising: a definition or label strategy; a nodes or arcs strategy; a diagram of topic concept distractor strategy; diagram with node and arc in wrong position strategy; and diagram with distractor node and arc strategy; or a combination thereof.

[0021] In yet a further embodiment, using definition or label strategy includes: generating concept distractor based on topic concept or node or arc; retrieving the definition or label of the topic concept or node or arc; and saving the retrieved definition or label and distractor into a distractor list.

[0022] In a further embodiment, using nodes or arc strategy includes: retrieving the

CS of the topic concept; extracting at least one node and one arc from the retrieved CS; selecting at least one node and arc to generate a distractor; replacing the selected node and arc with generated distractor; and grouping and saving the selected node and arc and distractor in the distractor list.

[0023] In yet a further embodiment, generating distractor with diagram of topic concept includes: generating a distractor concept; retrieving a CS based on the distractor concept; and saving the reference of the CS into the distractor list.

[0024] In another embodiment, generating distractor with diagram including node and arc in wrong position strategy includes: retrieving the CS for the topic concept; randomly switching position of the node and arc in the CS; and storing the distractor.

[0025] In yet a further embodiment, generating distractor with node and arc strategy includes: retrieving the CS for the topic concept; selecting nodes and arcs form the CS; generating distractor concept of the selected node and arc and replacing the selected node and arc with the respective distractor; and storing the distractor CS.

[0026] In another embodiment, generating the drawing nodes and arcs includes: providing at least one topic concept, and a question generation strategy (QGS); retrieving the node and arc for the topic concept; randomly selecting the node and arc from the retrieved node and arc; generating the distractor for the selected node and arc; and aggregating the node and arc of the topic concept and their distractors. [0027] In a further embodiment, generating a question distractor further includes generating concept distractor which comprises: providing at least one concept, a question difficulty type (QDT) and a question generation strategy; retrieving at least one type hierarchy distance range (THDR); retrieving at least one result item within the THDR based on the concept; filtering the result item which are instances or sub-classes of the concept in the event that the concept type is a class; determining the type hierarchy distance between the result items and the topic concept; ranking the result items; selecting a result item with highest rank; and using the result item to generate a distractor concept.

[0028] In yet another embodiment, assessing of answers which comprises: providing at least one question list; selecting at least one question; obtaining a correct answer and user answer based on the question; and determining and evaluating the correct answer and user answer.

[0029] In yet a further embodiment, assessing of answers further includes: selecting user answer conceptual structure (CS) reference and retrieving the answer CS; selecting topic concept and retrieving the CS for the topic concept; determining the similarity score between the conceptual structures (CS); and accruing all scores and obtaining a total score.

BRIEF DESCRIPTION OF DRAWINGS

[0030] The invention will be more understood by reference to the description below taken in conjunction with the accompanying drawings herein: [0031] FIG. 1 shows the overall process flow of the method for automated diagrammatic question generation and answer assessment in accordance with an embodiment of the present invention; [0032] FIG. 2 shows the process flow for the question template selection in accordance with an embodiment of the present invention;

[0033] FIG. 3A shows the process flow for question generation for a type of question mode in accordance with an embodiment of the present invention;

[0034] FIG. 3B shows the detailed process flow for question generation for another type of question mode in accordance with an embodiment of the present invention;

[0035] FIG. 4 shows the process flow for question body generation in accordance with an embodiment of the present invention;

[0036] FIG. 5 shows a process flow for answer generation with respect to a type of question mode in accordance with an embodiment of the present invention;

[0037] FIG. 6 shows a process flow for distractor generation in accordance with an embodiment of the present invention;

[0038] FIG. 7 shows a process flow for another distractor generation in accordance with an embodiment of the present invention;

[0039] FIG. 8 shows an example of a question mode in accordance with an embodiment of the present invention;

[0040] FIG. 9 shows the drawing nodes and arcs generation process flow in accordance with an embodiment of the present invention;

[0041] FIG. 10 is an example of another question mode in accordance with an embodiment of the present invention; [0042] FIG. 11 shows the process flow for generating a distractor concept in accordance with an embodiment of the present invention;

[0043] FIG. 12 shows a type hierarchy tree as part of generating a distractor concept in accordance with an embodiment of the present invention; and

[0044] FIG. 13 shows the process flow for assessing answers in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

[0045] in line with the above summary, the following description of a number of specific and alternative embodiments is provided to understand the inventive features of the present invention. It shall be apparent to one skilled in the art, however that this invention may be practiced without such specific details. Some of the details may not be described at length so as not to obscure the invention. For ease of reference, common reference numerals will be used throughout the figures when referring to the same or similar features common to the figures.

[0046] The present invention provides a method for automated diagrammatic question and answer assessment based on a knowledge base. Generally, the method in accordance with an embodiment of the present invention is adapted to provide effective proportionating question difficulty type distribution, the ability to select question template based on the selected question mode, various question difficulty levels and types and then the ability to generate at least one question based on the selected template.

[0047] In accordance with an embodiment of the present invention, there are generally at least two basic components in generating the questions as part of the assessment which are Question Difficulty Level (referred herein as QDL) and Question Difficulty Type (referred herein as QDT). Each of which will be described for the purpose of elucidating the objectives of the present invention. Question Difficulty Level ( QDL)

[0048] The commonly known Revised Bloom's Taxonomy is used for determining QDL. The taxonomy is restructured into four (4) difficulty levels: Remembering - Level 1; Understanding - Level 2; Applying - Level 3; and Analysing, Evaluating and Creating - Level 4.

[0049] For Level 1, being the Remembering level, the Question template is generally focused in testing the individual's memory capacity in a topic, whereby the questions can be in the form of Fill-in-blank, or the question can ask about all the involved parties in a process or the likes. The answers, distractors and drawing nodes and arcs in this level are restricted to concepts associated to a specific class type. [0050] For Level 2, being the Understanding level, the Question template is generally focused in testing the individual's understanding capacity in a topic, whereby for example, the questions can provide a definition of an application process and asks the individual to select or draw the correct diagram. Similar to that of Level 1, the answers, distractors and drawing nodes and arcs in this level are restricted to concepts associated to a specific class type.

[0051] For Level 3, being the Applying level, the Question template is generally focused for testing the individual's application of knowledge, whereby in one example the questions given may provide a definition of an application process and requires the individual to select or draw the correct diagram. The answers, distractors and drawing nodes and arcs in this level can be instance type or class type.

[0052] For Level 4, being the Analysing, Evaluating and Creating level, the

Question template is generally focused on testing the individual's capabilities in analysing, evaluating and creating. The set of questions may give a use case of an application process and the individual may need to select or draw the correct diagram. Answers, Distractors, Drawing Nodes and Arcs may include concepts with either instance or class type. Question Difficulty Type ( QDT)

[0053] In accordance with an embodiment of the present invention, Question difficulty type

(QDT) is used to control the difficulty of distractor in questions. A distractor as generally known in the ait and as used in tire context of describing the embodiments of the present invention defines the incoirect alternatives for that of an accurate answer. For the method of the present invention, the distractor generation is based on the type hierarchy distance in knowledge base. In one embodiment, a distractor concept can be more difficult when it is semantically nearer or closer to the answer concept. Nevertheless, in another embodiment, a distractor concept can be easier when it is semantically further to the answer concept. One option, but not limiting to, which may be used to determined the distance between the distractor and answer is the type hierarchy similarity measure. Based on the distractor difficulty control methodology, generating questions with various difficulties within a specified question difficulty level can be performed. Question difficulty type (QDT) is used to classify question mode when question mode is "Mix" mode. The "Mix" mode will be further elaborated herein.

[0054] FIG. 1 provides an overall architecture on the operational process flow of the method in accordance with an embodiment of the present invention. The method starts at 100 with providing at least one topic, at least one question and at least one question difficulty type (QDT) configuration file 10 and question difficulty level (QDL) file 15. In the next step 101, the QDT configuration file 10 and question difficulty level (QDL) file 15 are used to proportionate the difficulty of at least one question distribution. Then, a percentage of QDT within the QDL is provided at 102. In one embodiment, during this step in determining the question distribution, it can be within the range of easy, intermediate and difficult. For instance, a distribution of 30% can be "easy" types; 40% can be "intermediate" types and 50% can be "difficult" question types.

[0055] Further, for the Question Difficulty Level (QDL) in accordance with an embodiment of the present invention, the QDL will be initially specified where the value maybe of QDL can be of Level 1, 2, 3 and 4 respectively where these values may be referred to the Revised Bloom's Taxonomy. At 103, the method detennines whether sufficient amount of questions has been generated for the QDT. In the event that a YES is obtained for step 103, the overall process thereby ends at 103A. If NO is obtained for step 103, the process continues to select a question template at 104 based on the Question Mode, QDL, and QDT 15 A. The selected question template 20 is then used for generating questions at 105. In one embodiment, the generated question 25 can be either in the form of a Diagrammatic Multiple Choice Question (DMCQ) or Hand Drawing Diagrammatic Question (HDDQ) being the final the output. Topic and QDT containing input 30 can be further used to generate question.

[0056] The Question Template Selection of step 104 from FIG. 1 process flow will now be described with reference to FIG. 2. Upon initiated at 200, the question mode which can be of DMCQ or HDDQ, can be selected from a repository based on the Question Mode and QDL 24. In the event that the Question Mode is a "mixed" mode, thus YES at 201 the process proceeds to classify the Question Mode at 202. In addition, at 203 the Question Mode configuration file 35 will be used for classifying at 202. Accordingly, the configuration file 35 will utilize a list of classification to specify a Question Mode based on the QDT 10, whereby the output includes the value of DMCQ or HDDQ to be used in selecting the question template. In an example, the configuration file may have a classification provided below:

[0057] If QDT = "easy", then Question Mode = "DMCQ"

[0058] Else QDL = "difficult", then Question Mode = HDDQ"

[0059] In the event that at 201 a NO is obtained, the process proceeds to 203, for selecting question template. During the step of 203 in selecting the question template, input to aid the selecting process can be retrieved from the Question Template Repository 250, whereby further input may include question mode and QDL 24. At 204, the question template is obtained and the process ends at 205.

[0060] The Question Generation of Diagrammatic Multiple Choice Question (DMCQ) being step 105 from FIG. 1. The process flow for question generation will now be described with reference to FIG. 3A, particularly when the Question Mode is Diagrammatic Multiple Choice Question (DMCQ). The process starts at 300, whereby input, which includes Topic 40, Question Difficulty Type (QDT) and Question Template are subjected to a retrieving of Topic Concept step 301. During this retrieving of Topic Concept step, source of input may include from Curriculum Knowledge Base 44. As mentioned, in the event that the Question Mode chosen is Diagrammatic Multiple Choice Question (DMCQ) at 302, the process proceeds to retrieve Question Generation Strategy (QGS) with input retrieved from Question Template Repository 250 and question template 20 to determine the Question Body, Answers and Distractors at 303. Once when the question body and vector list 47, question answer 48 and question distractors 49 are generated at 304, 305 and 306 respectively, all of these are aggregated at 307 and the final output will be an aggregated question in the form of DMCQ at 308. In one embodiment, the process utilizes topic concept, vector list, QDT and selected strategy input 43 to generate the question body, answer and distractors. The process ends at 309. If the Question Mode is not DMCQ thus a NO at 302, the process proceeds to the generation of HDDQ 302A, which will be described herein. [0061] Another process of the Question Generation of Hand Drawing Diagrammatic

Question (HDDQ) being another question mode as part of step 105 from FIG. 1 process flow will now be described with reference to FIG. 3B. The process starts at 310 (from 302A in previous FIG. 3A), and proceeds to retrieving Question Generation Strategy (QGS) for Question Body and Drawing Nodes& Arcs at 311. Upon selected the QGS, it is used for generating the Question Body 51 at 312, and generating drawing nodes and Arcs 52 at 313. During these steps 312 and 313, input which includes Question Template 20, Topic Concept, and QGS 25A, 25B may be utilized. At 314, the Question Body and Drawing Nodes & Arcs 51, 52 are aggregated prior to generating the HDDQ generation strategy at 315. The process ends at 316.

[0062] The Question Body generation being step 312 of FIG. 3B process flow in accordance with an embodiment of the present invention will now be described with reference to FIG. 4. The process is initiated at 401, whereby the initial input parameter includes, but not limiting to, Topic Concept, question template and QGS for the question body. At 402, a decision is made to determine whether an instance is required, whereby a YES response is received, the process proceeds to retrieving an Instance of the Topic Concept and replace Topic Concept with the Instance at 402A. The Instance may be retrieved from a Domain Knowledge base 60. Examples of controlling and providing the question difficulties in accordance with an embodiment of the present invention are as shown in EXAMPLE 1 and EXAMPLE 2.

EXAMPLE 1

[0063] Question Difficulty Level - 2 [0064] The selected topic concept is Salam;

The selected HDDQ template is "Draw a correct diagram that represents <Topic Concept>;

The selected QGS generates question body with the label of topic concept; The final output of the question body generation process will be "Draw a correct diagram that represents Salam. "

EXAMPLE 2

[0065] Question Difficulty Level - 3

[0066] The selected topic concept is Salam;

The selected HDDQ template is "Draw a correct diagram that represents

<Topic Concept>;

The Selected QGS generates question body with the use case of topic concept instance;

[0067] From the examples, the Question body process retrieves an instance of Salam that is salam_l and subsequently retrieves the use case of the instance salam_l and aggregate with the question template. An example of the final output of the question body generation process may be: "Draw correct diagram that represents Framer Muhammad wants to purchase fertilizer from Hap Seng Consolidate Bernard for this farm with Maybank Islamic." [0068] Based on the examples above, in one embodiment of the present invention, the same question template attached with various QGS can be used to generate question body for various difficulty levels.

[0069] It should be noted that in accordance with an embodiment of the present invention, there are three primary ways or types of the QGS being elucidated in FIG. 4. These are generating questions based on label, definition or use case; secondly generating question body with the correct diagram; and thirdly generating question body with a fill-in-the-blank diagram. Each of these types will be described herein below. [0070] Still referring to FIG. 4, in the event that a NO in requiring the instance is received as a response at step 402, the process proceeds to the next step of deteixnining whether to retrieve and thus use any one or combination of properties such as, but not limiting to; label, definition or use case strategy based on the selected Topic Concept at 403.1f any one of the properties is used and thus a YES response is obtained at 403, the selected property is retrieved at 403A from the knowledge base. Upon retrieved the selected property which can be of a label, definition or use case is aggregated with the Question Template at 403B, to finally form a question body at 406. The process ends at 407. [0071] In one embodiment, in the event that a NO is obtained at 403, the process proceeds to determine whether to use the corcect diagram strategy at 404. If a YES is obtained at 404, the process proceeds to retrieve the conceptual structures (CS) for Topic Concept at 404A. And then the CS are aggregated based on reference of the CS with Question Template at 404B. From here at least one question body is generated at 406 and the process ends at 407.

[0072] If a NO is obtained as a response at 404, the process proceeds to deterrnine whether a fill-in-the-blank diagram strategy can be used at 405. If the fill-in-the-blank strategy is selected thus a YES is obtained at 405, the CS for the Topic Concept are therefore accordingly retrieved at 405A. And then at 405B, the system randomly selects at least one node; or arc from the CS. The selected node(s) or arc(s) are stored in a vector list at 405C. From here at 405D, the reference of the CS and vector list is accordingly aggregated with Question Template, and thus generating question body and vector list at 405E. The process then ends at 407. It should be 5 mentioned that in this fill-in-the-blank type the node(s) and arc(s) will be used to generate the answer and distractors.

[0073] FIG.5 shows the process flow for Diagrammatic Multiple Choice Questions

(DMCQ) answer generation being step 305 of FIG. 3A in accordance with an embodiment of the

10 present invention. In this process, the initial input parameter of the process includes, but not limiting to Topic Concept, Vector List and QGS. The process flow starts at 500, whereby based on input, the process decides whether an instance is required at 501. If a YES is obtained as a response at 501, the process randomly retrieves at 502 at least one instance of the Topic Concept and replace the Topic Concept with the selected instance. The process may retrieve the instance from a Domain

15 Knowledge Base 60.

[0074] In the event that a NO is obtained as a response at 501, the process flow provides three types QGSs for DMCQ answer generation; firstly; based on definition/label strategy 503; secondly based on correct diagram strategy 504 and finally the third being the nodes/arcs strategy 20 505. Each of these types will be described within the process flow herein below.

[0075] Still referring to FIG.5, in the event that the process decides with using definition or label for generating question answer at 503, the process retrieves at 503A label or definition of the Topic Concept from Domain Knowledge base 60 as the coirect answer. In another embodiment, 25 label or definition of the node and arc may be retrieved as the coirect answer as well. Upon completion of retrieving the said properties, the process proceeds to generate the question answer at 506 and the process ends at 507.

[0076] In another embodiment, in the event that the process decides with using a correct

30 diagram strategy in generating question answer at 504, the process therefore proceeds with retrieving

CS for Topic Concept from the Domain Knowledge Base 60 at 504A and the reference of the CS will be used as part of generating the question answer at 506 and the process ends at 507. [0077] Further with reference to the process flow of FIG.5, in the event that the process decides with using node(s) or arc(s) stetegy at 505, the process proceeds to retiieve the CS for the Topic Concept at 505A and then extracts at least one node and at least one arc from the CS at 505B. Based on the retrieved properties, the question answer is generated at 506 and the process ends at 5 507.

[0078] In accordance with an embodiment of the present invention, there are two approaches in generating DMCQ distractors being step 306 of FIG. 3A. These are shown in the process flows depicted in FIG.6 and FIG.7, respectively. With reference to FIG.6, the process flow0 starts at 600, whereby the input collected to be used for this process flow includes Topic Concept, QDT, Question Generation Strategies (QGS) in generating distractors. Next in 601, the process deteiTnines whether an instance is required, to which if a YES is obtained as a response, the process proceeds to 601A, for retrieval of at least one instance of the Topic Concept and then replacing the Topic Concept with the instance. In the event that there are not enough distractors at 602, the process proceeds with 602A, whereby a question distractor list is generated and the process ends at 602B.

[0079] There are five (5) types or approaches of QGS for the DMCQ distractors generation in accordance with an embodiment of the present invention. Each of them will be described herein with reference to FIG. 6 and FIG.7 as part of the process flows therein. Proceeding0 from the above, at 602, if there are enough distractors, the process proceeds with selecting a QGS at

603. And then based on the selected QGS, the distractor is generated using several approaches. The process then determines at 604 whether a definition or label stetegy may be initiated. If YES, the process proceeds to generate concept distractor at 604A and then retrieving the definition and label of the Topic Concept at 604B. All retrieved properties are saved into a list at 604C. The concept distractor may be retrieved or extracted from a Curriculum Knowledge Base 44.

[0080] In the event that the process decides to proceed with node/arcs stetegy at 605, the retrieval of CS of the Topic Concept is therefore initiated at 605A. And then the nodes/arcs are extiacted from the retrieved CS at 605B. At 605C, at least one node and one arc are selected for generating the distractors, to which at 605D, the selected nodes and arcs are replaced with generated distractors. The distractors are then grouped and saved in a distractor list at 605E.

[0081] The approach in generating distractors then continues to process flow A at 606, which will be described with reference to FIG.7. The process continues with determining whether a distractor concept diagram strategy can be initiated at 701, whereby if a YES is obtained as a response, the process proceeds to generate distractor concept at 701A and then retrieving the CS based on the distractor concept at 701B. All reference to CS then saved to distractor list at 701C.

5 10082] If a diagram with node and arc in wrong position strategy is initiated at 702, the process proceeds with retrieving the CS for the Topic Concept at 702A and then randomly switching positions of the nodes and arcs in the CS at 702B. The distractor CS is stored at 702C. In another embodiment, if a diagram with distractor approach is initiated at 703, the process proceeds with retrieving the CS for the Topic Concept at 703A. Next, at least one node and one arc are selected 0 from the CS at 703B. From the selected node and arc, the distractor concepts are generated at 703C and then replace the selected nodes and arcs with the respective distractor. The distractor(s) CS are stored at 703D, into a Question Generation Knowledge Base 80 and the reference of the CS stored in the distractor list. An example of a DMCQ is shown as FIG. 8. 5 [0083] FIG. 9 depicts the process flow for generating HDDQ Drawing Nodes and Arcs being step 313 of FIG. 3B process flow in accordance with an embodiment of the present invention. The process starts at 900, whereby the input parameter used includes Topic Concept and QGS. In the event that an instance is required at 901, the process retrieves at least one instance of the Topic Concept and replaces the Topic Concept with the instance at 901 A. The instance may be retrieved0 from the Domain Knowledge Base 60.At 902, the process proceeds to retrieve at least one node and arc of the Topic Concept. If a distractor is not needed at 903, the process proceeds to aggregate the node(s) and arc(s) of the Topic Concept at 903A. On the contrary, if a distractor is needed, the process then randomly selects at least one node and one arc from the retrieved node(s) and arc(s) at 904 and then generating at least one distractor for the selected node and arc at 905. The node(s) and5 arc(s) are aggregated at 906 and thus the drawing nodes and arcs are generated at 907. The process ends at 908. FIG. 10 shows an example of the HDDQ with distractor Node and Arcs.

[0084] FIG. 11 shows the process flow for generating a distractor concept in accordance with an embodiment of the present invention. For this process, the initial input0 parameters includes, Concept, QDT and QGS. It should be noted that the concept in this context can either be referred to Topic Concept or concept of a node or arc in the knowledge base 60. The process starts at 1100, with the mentioned input parameters, the Type Hierarchy Distance Range (THDR) configuration file 65 will be utilized to retrieve THDR based on the QDT at 1101. For example, the THDR can be 1 <rangc<3, 2<range<5, or 4<range, whereby it shows the distractor needs to be 1 to 2 nodes away from the concept. Suitable, the question difficulty will be more difficult when distractor is nearer to the concept, and will be easier when the distractors are further from the concept. Once the THDR is defined is retrieved, the process the result items 66 are retrieved at 1102, which are within the THDR from the domain knowledge base 60. The concept type is firstly checked at 1103, before detennining whether the concept is of class type at 1104. And then if a YES is obtained at 1104, the process then proceeds with filtering the instance of the concept, the subclasses and instance of the subclasses if the concept is class type at 1104A. The process then proceeds with determining whether an instance is required at 1105. If an instance is not required, the result items that are of instance type are filtered at 1105A. On the other hand, in the event that the instance is required, the result items that are of class type are filtered at 1106. It should fuither be noted that filtering plays a pivotal role in this process because the distractors and answers in the lower question difficulty levels are only Umited to concepts that are of class type, while distractors and answer are limited to instance type. In this way sample question template can be used to create questions in various question difficulty level. Subsequently at 1107 the process proceeds with determining the semantic similarity between result items and the Topic Concept. Upon completion of determining the similarity, the result items are accordingly ranked based on hierarchy similarity at 1108. And then from the ranking result, a result item is selected based on the highest rank at 1109. A distractor concept is therefore generated based on the result item selected at 1110. The process ends at 1111.

[0085] FIG. 12 shows an example of the tree structure or hierarchy involved in distractor generation simulation in accordance with an embodiment of the present invention. With reference to FIG. 12, the aim of this tree structure is to generate the distractor concept of "Bank". Accordingly, from the knowledge base, the concepts that are not instances/subclasses of the concept "Bank" will therefore be distractors. The concepts that are outside of the red box will be the distractors of the concept "Bank". In this example, based on the type hierarchy distance, those concepts that are 4 nodes and not 7 nodes away will be selected. Example of selected concepts will be Cash, Capital, Cash_BBA, Cashjstisna, Capitaljnan and Capital_kafalah. A type hierarchy similarity measure approach will be used to detennine the type hierarchy similarity between "Bank" and selected concepts and then filter out the selected concept, which are instance type based on question difficulty level 1. [0086] Still referring to FIG. 12, there are two concepts remaining, which are "Cash" and "Capital". Accordingly a similar hierarchy measuring approach is used to determine the similarities between the 2 concepts and concept "Bank". For example, in order to determine^' type hierarchy similarity between Cash and Bank, the following steps can be performed: Identifying a common ancestor of Cash and Bank, and then determine the distance between the common ancestors and the root of the hierarchy tree. In this example with reference to FIG. 12, the common ancestor is the root of the hierarchy tee therefore the distance should be 0. Upon identified the distance, the distance value is used to increase 1 in order to avoid 0 value that will result in calculation error later and then use the final value multiply with 2 (the common ancestor of two concepts); and finally the value A from above process is determined. Fuither in accordance with an embodiment of the present invention, the distance of the Cash to the common ancestor (CA) and Bank to the common ancestor (BA) is determined. (In this example they are 2 and 3). Once CA and BA value is determined, then determine the total value of the CA, BA and A,; and finally using of value A to normalize with the total value of the CA, BA and A to get the type hierarchy similaiities. From here, the distractor with the highest type hierarchy similarity measure is selected as distractors. For this simulation, "Cash" is selected as the distractor of "Bank".

[0087] FIG. 13 shows the process flow for answer assessment in accordance with an embodiment of the present invention, to be described herein below. The process starts at 1301, whereby the input parameter for this process includes a question list containing a list of question and user's answer. At 1302, the process determines whether the question list is empty or otherwise. If YES, the process proceeds to calculate the total score with reference to the question score 91 at 1301A. Next at 1301B the total score is obtained and the process ends at 1301C. If NO, the process proceeds to 1302 whereby at least one question is selected. If the question is DMCQ at 1303, the process proceeds to retrieve the correct answer and the user's answer from the question at 1303A, and subsequently evaluate the correct answer and user's answer to produce the answer score at 1303B. The resultant selection and retrieved conceptual structure of the answer are used as input for converting a graph-based similarity result to question score - being the final step of the process at 1310. [0088] On the other hand, if the question is HDDQ thus not DMCQ at 1303, the reference of the user answer's conceptual structure is selected and retrieves the conceptual structure of the answer at 1304. Next the Topic Concept and CS are retrieved at 1305, 1306, 1307 and 1308; these properties are then used in determining the similarity between the conceptual structures at 1309, whereby a user semantic graph similarity matching method may be applied during this step for determining the similarity. It should be noted that in accordance with an embodiment of the present invention, in deteimining the similarity of the conceptual structures; nodes and arcs of two CS are matched and paired; whereby, the type hierarchy similarity measure will be determined for the paired nodes and arcs. The total value of the type hierarchy similarity measure of nodes and arcs are then nomialized by the total number of the nodes and arcs in the CS to determine the similarity value between the two CSs. During the graph similarity matching process, a threshold value can be involved to determine the similarity results. For instance, the value 0.6 will be assigned as the threshold value; the similarity result will be 0 if it is lower than threshold.

[0089] Accordingly, a graph-based similarity result may be generated as an output.

Finally at 1310, the resulting graph is then used and thus converted into a question score. Suitably, both of the result of the semantic based graph similarity matching to the answer score and each of the answer scores are accrued by iterating each of the questions from the question list to produce the total score.

[0090] As would be apparent to a person having ordinary skilled in the art, the afore- described methods and components may be provided in many variations, modifications or alternatives to existing methods and systems. The principles and concepts disclosed herein may also be implemented in various manners which may not have been specifically described herein but which are to be understood as encompassed within the scope of the following claims.

Claims

1. An automated method for diagrammatic question and answer assessment; the method comprising: providing at least one topic and at least one question based on the topic (100); providing at least one range of a question difficulty level (QDL) (100);

providing at least one question mode (100);

proportioning question difficulty type (QDT) based on the range of question difficulty level (101);

selecting at least one question template based on the proportioned question difficulty type (QDT) and question difficulty level (QDL) (104); and

generating a question based on the selected template (105);

wherein the question mode includes questions in the form of diagrammatic multiple choice type (DMCQ) or hand drawing diagrammatic type (HDDQ).

2. The method as claimed in Claim 1 wherein proportioning question difficulty type (QDT) includes providing a predetermined percentage range of the QDT based on the question difficulty level (QDL) (102).

3. The method as claimed in Claim 1 wherein selecting at least one question template based on the proportioned question difficulty type (QDT) and question difficulty level (QDL) further includes:

selecting the question template based on the question mode of DMCQ or HDDQ

(203) and;

selecting the question template based on mix mode being both DMCQ and HDDQ (201, 202,203).

4. The method as claimed in Claim 1 wherein generating a question based on the selected template (105) includes:

providing at least one topic, at least one question difficulty type (QDT) and at least one question template (300);

retrieving at least one topic concept (301); retrieving a question generation strategy (QGS) (302,303) based on at least one question mode;

generating a question body (304);

generating a question answer (305);

generating a question distractor (306);

aggregating the question body, answer and distractor (307) generating drawing nodes and arcs (313); and

aggregating the question body and drawing nodes & arcs (314).

5. The method as claimed in Claim 4 wherein generating a question body (304) further comprises:

providing at least one topic concept, a question template, and a question generation strategy (QGS) (401);

determining whether an instance is required to which if YES, randomly retrieving an instance based on the topic concept and replacing the topic concept with the retrieved instance (402 A);

if no instance is required, applying at least one type of question generation strategy (QGS) to generate the question body (403, 403A);

wherein the question generation strategy (QGS) includes using label or definition or case strategy or a combination thereof; using a correct diagram strategy or filled-in-the-blank strategy or a combination thereof.

6. The method as claimed in Claim 5, wherein in using label or definition or case strategy or a combination thereof; using a correct diagram strategy or filled-in-the-blank strategy or a combination thereof; at least one CS is retrieved based on the topic concept and the CS is aggregated with the question template.

7. The method as claimed in Claim 4, wherein generating a question answer (305) includes:

providing at least one topic concept, a vector list and at least one generation strategy (QGS) (500);

determining whether an instance is required (501);

if YES to requiring an instance, randomly retrieving an instance of the topic concept and replacing the topic concept with the instance (502); if NO to requiring an instance, determine whether to use definition or label strategy or correct diagram strategy or nodes or arc strategy in generating the question answer (503,504,505).

8. The method as claimed in Claim 7, wherein using definition or label strategy in generating the question answer (503) include retrieving at least one definition or a label based on the topic concept (503A).

9. The method as claimed in Claim 7, wherein using correct diagram strategy in generating the question answer (504) includes retrieving the CS for the topic concept

(504A).

10. The method as claimed in Claim 7, wherein using nodes or arcs strategy in generating the question answer (505) includes retrieving the CS for the topic concept (505A) and extracting the nodes or arcs from the CS (505B).

11. The method as claimed in Claim 4 wherein generating a question distractor (306) includes: providing at least one topic concept, at least one QDT, at least one question generation strategy (600);

determining whether an instance is required (601 );

if YES in requiring an instance, randomly retrieving an instance of the topic concept and replacing the topic concept with the instance (601A);

if NO in requiring an instance (602), selecting a QGS (603);

wherein selecting the QGS includes selecting anyone from the group comprising a definition or label strategy, a nodes or arcs strategy, a diagram of topic concept distractor strategy, diagram with node and arc in wrong position strategy, and diagram with distractor node and arc strategy, or a combination thereof.

12. The method as claimed in Claim 4, wherein generating the drawing nodes and arcs (313) includes: providing at least one topic concept, and a question generation strategy (QGS) (900); determining whether instance is required (901);

if YES in requiring an instance, randomly retrieving an instance of the topic concept and replacing the topic concept with the instance (901 A);

if NO in requiring an instance, retrieving the node and arc for the topic concept (902);

randomly selecting the node and arc from the retrieved node and arc (904);

generating the distractor for the selected node and arc (905); and

aggregating the node and arc of the topic concept and their distractors (906).

13. The method as claimed in Claim 4 wherein generating a question distractor (306) further includes generating concept distractor which comprises: providing at least one concept, a question difficulty type (QDT) and a question generation strategy (1 100);

retrieving at least one type hierarchy distance range (THDR) (1 101);

retrieving at least one result item within the THDR based on the concept (1102); filtering the result item which are instances or sub-classes of the concept in the event that the concept type is a class (1104A);

determining the type hierarchy distance between the result items and the topic concept (1107);

ranking the result items (1108);

selecting a result item with highest rank (1109); and

using the result item to generate a distractor concept (1 1 10).

14. The method as claimed in Claim 1 wherein the method further includes assessing of answers which comprises: providing at least one question list (1300);

selecting at least one question (1302, 1303),

wherein,

when the question is a diagrammatical question,

obtaining a correct answer and user answer based on the question (1303A); and determining and evaluating the correct answer and user answer (1303B); or

when the question is not a diagrammatical question,

selecting user answer conceptual structure (CS) reference and retrieving the answer CS (1304);

selecting topic concept and retrieving the CS for the topic concept (1305, 1306,1307,1308);

determining the similarity score between the conceptual structures (CS) (1309); and;

accruing all scores and obtaining a total score (1310).