CN117670269A - Method and device for realizing questions for multidimensional forced selection test and electronic equipment - Google Patents
Method and device for realizing questions for multidimensional forced selection test and electronic equipment Download PDFInfo
- Publication number
- CN117670269A CN117670269A CN202311655474.5A CN202311655474A CN117670269A CN 117670269 A CN117670269 A CN 117670269A CN 202311655474 A CN202311655474 A CN 202311655474A CN 117670269 A CN117670269 A CN 117670269A
- Authority
- CN
- China
- Prior art keywords
- question
- questions
- answer
- selecting
- topic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 81
- 238000012360 testing method Methods 0.000 title claims abstract description 70
- 230000009191 jumping Effects 0.000 claims abstract description 7
- 230000008569 process Effects 0.000 claims description 34
- 239000011159 matrix material Substances 0.000 claims description 24
- 238000010276 construction Methods 0.000 claims description 22
- 238000005259 measurement Methods 0.000 claims description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000014509 gene expression Effects 0.000 claims description 5
- 238000004458 analytical method Methods 0.000 claims description 3
- 230000002441 reversible effect Effects 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 5
- 238000011156 evaluation Methods 0.000 description 9
- 230000008901 benefit Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000003044 adaptive effect Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000013100 final test Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000010187 selection method Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000009469 supplementation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/105—Human resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Databases & Information Systems (AREA)
- Computational Mathematics (AREA)
- Entrepreneurship & Innovation (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Operations Research (AREA)
- General Engineering & Computer Science (AREA)
- Pure & Applied Mathematics (AREA)
- Strategic Management (AREA)
- Mathematical Physics (AREA)
- Evolutionary Biology (AREA)
- Software Systems (AREA)
- Algebra (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Economics (AREA)
- Marketing (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The method comprises a first selection step, based on random selection, of not repeatedly selecting a plurality of questions from a question library as first answer questions, and obtaining answer data obtained by answering corresponding questions by a target subject, wherein the measured dimensions of all the first answer questions cover all the dimensions to be measured; a second selecting step of selecting a question from the remaining questions in the question bank as a second answer question and obtaining answer data of the corresponding question answered by the target subject, and evaluating whether dynamic selecting can be performed based on Fisher information quantity, if yes, jumping to execute a third selecting step, otherwise continuing to execute the second selecting step; and thirdly, performing dynamic question answering based on the Fisher information quantity method until the question answering condition of the target subject meets the preset requirement. The method and the device are beneficial to more efficiently completing the related test.
Description
Technical Field
The application belongs to the technical field of evaluation, and particularly relates to a method and a device for realizing questions of multidimensional forced selection tests and electronic equipment.
Background
The forced selection test is a test form for measuring the non-ability characteristics (such as personality, attitude and thinking) of a tested person in personnel selection, the division of personality traits is more and more refined at present, so that the requirement for high-dimensional tests (referring to more feature dimensions related to the tests) is increased, and the number of feature dimensions measured by the forced selection test of the current commercial application is more than 20.
In the technical field of talent evaluation, in order to realize evaluation paths which are different from person to person, the questions answered by each testee are different, meanwhile, only the questions matched with the level of the testee can be answered, so that the number of the answered questions is reduced, the test efficiency is improved, a computer self-adaptive test CAT (Computerized Adaptive Test) appears, and CAT is a measuring means which appears along with the project reaction theory IRT (Item Response Theory) and the development of computer technology. In the related art, CAT realizes topic selection based on Fisher information in application implementation, and a topic selection method based on Fisher information is relatively mature in a single-dimensional topic scenario.
However, the existing CAT implementation method does not consider the influence on Fisher information matrix calculation when the dimension is higher; if in the multi-dimensional forced selection test scene, the number of dimensions to be measured is large, and the following problems can be caused by migrating the traditional Fisher information matrix-based adaptive question selection method: 1) The number of the prepositioned questions is unstable, and a plurality of questions can be tested to start computing the Fisher information matrix, so that the overall test efficiency is affected; 2) The same title is always selected in the pre-stage, so that part of the title is overexposed to influence the final test effect.
Therefore, how to realize more effective CAT choice questions under the multi-dimensional forced choice test scenario becomes a technical problem to be solved urgently.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
In order to overcome the problems existing in the related art at least to a certain extent, the application provides a method, a device and electronic equipment for realizing the questions for the multidimensional forced selection test, so as to solve the technical problem of how to realize more effective CAT questions under the scenario of the multidimensional forced selection test.
In order to achieve the above purpose, the present application adopts the following technical scheme:
in a first aspect of the present invention,
the application provides a method for realizing the questions of a multidimensional forced selection test, which comprises the following steps:
a first selecting step, based on random selection, of not repeatedly selecting a plurality of questions from a pre-constructed question library as first answer questions, and obtaining answer data obtained by answering each first answer question by a target subject, wherein the measured dimensions of all the first answer questions cover all the dimensions to be measured;
a second selecting step, namely selecting the questions from the rest questions which are not selected in the question library in a preset mode as second answer questions, acquiring answer data corresponding to the second answer questions by the target subjects, evaluating whether dynamic questions can be selected based on Fisher information amount or not based on all acquired answer data and item parameter information of the corresponding questions, if yes, jumping to execute a third selecting step, otherwise, continuing to execute the second selecting step;
and a third selecting step, namely selecting a third answer question from the rest questions which are not selected in the question library based on the Fisher information amount, and acquiring answer data of the corresponding third answer questions of the target subject until the question answer condition of the target subject meets the preset requirement.
Optionally, in the first selecting step, a process of selecting the first answer sheet specifically includes:
step one, establishing a dimension vector comprising all dimensions to be measured, and randomly selecting a question from the question library as a current selected question;
step two, performing first marking on the title corresponding to the currently selected title in the title library, and removing the dimension measured by the title from the dimension vector;
judging whether the vector number in the dimension vector is zero, if so, taking all the questions marked by the first marks as the first answer questions and ending the selection process, otherwise, continuing to execute the fourth step;
and step four, respectively carrying out matching analysis on other topics which are not marked by the first mark in the topic library according to the dimension vector, correspondingly obtaining the dimension coincidence number of the dimension measured by the corresponding topic as the dimension in the dimension vector, comparing the dimension coincidence number corresponding to the other topics, taking the topic with the largest dimension coincidence number as the new currently selected topic, and jumping to execute the step two.
Optionally, in the second selecting step, the evaluating whether the dynamic question can be selected based on the Fisher information amount based on all the obtained answer data and the item parameter information of the corresponding question includes:
calculating a Fisher information matrix according to all acquired answer data and item parameter information of the corresponding subject;
when the Fisher information matrix obtained through calculation is reversible, determining that dynamic questions can be selected based on the Fisher information quantity, or determining that dynamic questions cannot be selected based on the Fisher information quantity.
Optionally, in the third selecting step, a third answer question is selected from the remaining questions not selected in the question bank based on the Fisher information amount method, including:
combining each of the residual questions with the answered questions respectively aiming at the unselected residual questions in the question library to obtain a question combination comprising the corresponding residual questions;
calculating the Fisher information matrix corresponding to each question combination, comparing the inverted trace of the obtained Fisher information matrix, and taking the remaining questions in the question combination corresponding to the Fisher information matrix with the smallest trace as the third answer questions selected at the time.
Optionally, each topic in the topic library is a fixed n-element forced choice topic group topic, where n is greater than or equal to 2.
Optionally, in the pre-construction process of the topic library, selecting axiom based on Luce to perform model construction so as to obtain a topic block measurement model required by project parameter estimation;
the mathematical constraint of the block measurement model comprises that the sum of difficulty parameters of all items in each block is zero.
Optionally, each question in the question library is a fixed quaternary forced choice item group question, and the answer form is ordering;
in the pre-construction process of the topic library, the expression description of the constructed topic block measurement model comprises the following steps:
P j (a＞b＞c＞d|θ a ,θ b ,θ c ,θ d )＝P j (a＞b,c,d|θ a ,θ b ,θ c ,θ d )P j (b＞c,d|θ b ,θ c ,θ d )P j (c＞d|θ c ,θ d )
wherein,
a, b, c, d represent four items in the question block j,
θ a ,θ b ,θ c ,θ d corresponding to the trait parameters representing the dimensions tested for by items a, b, c, d,
P j (a＞b＞c＞d|θ a ,θ b ,θ c ,θ d ) Representing the probability that the choice of the test person is a > b > c > d when the answer block j is madeThe rate of the product is determined by the ratio,
P j probability of selection, Q j (*)＝1-P j The probability of not choosing is indicated,
α j* the distinguishing parameter of the item representing the question block j,
β j* the difficulty parameter representing an item in the question block j.
Optionally, in the pre-construction process of the question bank, according to the obtained test response data aiming at the question bank, performing model parameter estimation by adopting an iStEM algorithm based on the question measurement model so as to obtain item parameter information of each question.
In a second aspect of the present invention,
the application provides a choice question realization device for multidimensional forced selection test, which comprises:
the first selecting and processing module is used for not repeatedly selecting a plurality of questions from a pre-constructed question library to serve as first answer questions based on random selection in a first selecting process, and acquiring answer data obtained by answering each first answer question by a target subject, wherein the measured dimensions of all the first answer questions cover all the dimensions to be measured;
the second selecting and processing module is used for selecting the questions from the rest questions which are not selected in the question library in a preset mode as second answer questions and obtaining answer data of corresponding second answer questions of a target subject, evaluating whether dynamic selection questions can be carried out based on Fisher information quantity or not based on all the obtained answer data and item parameter information of the corresponding questions, if yes, skipping to execute a third selecting step, otherwise, continuing to execute the second selecting step;
and the third selecting and processing module is used for selecting a third answer question from the rest questions which are not selected in the question library based on the Fisher information amount method in the third selecting process, and acquiring answer data of the corresponding third answer questions of the target subject until the question answer condition of the target subject meets the preset requirement.
In a third aspect of the present invention,
the application provides an electronic device, comprising:
a memory having an executable program stored thereon;
and a processor for executing the executable program in the memory to implement the steps of the method described above.
The application adopts the technical scheme, possesses following beneficial effect at least:
in the technical scheme of the application, the method for realizing the topic selection comprises the following steps: a first selecting step, based on random selection, of not repeatedly selecting a plurality of questions from a pre-constructed question library as first answer questions, and obtaining answer data obtained by answering each first answer question by a target subject, wherein the measured dimensions of all the first answer questions cover all the dimensions to be measured; a second selecting step of selecting the questions from the rest questions not selected in the question library in a preset mode as second answer questions and obtaining answer data corresponding to the second answer questions by the target subjects, evaluating whether dynamic questions can be selected based on Fisher information amount or not based on all the obtained answer data and item parameter information of the corresponding questions, if yes, skipping to execute the third selecting step, otherwise continuing to execute the second selecting step; and a third selecting step, namely selecting a third answer question from the rest questions which are not selected in the question library based on the Fisher information amount, and acquiring answer data of the corresponding third answer questions of the target subject, until the question answer condition of the target subject meets the preset requirement. Compared with the existing CAT topic selection implementation, the technical scheme of the application has the advantages that the prepositive control topic selection link is added before dynamic topic selection based on Fisher information quantity, the defect that topic selection is carried out by directly applying Fisher information quantity is avoided, and the method is beneficial to more efficiently completing related tests in a high-dimensional test scenario.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention.
Drawings
The accompanying drawings are included to provide a further understanding of the technical aspects or prior art of the present application and constitute a part of this specification. The drawings, which are used to illustrate the technical solution of the present application, together with the embodiments of the present application, but do not limit the technical solution of the present application.
FIG. 1 is a schematic illustration of each flow link of talent assessment implementation in the context of the present application;
FIG. 2 is a schematic flow diagram illustrating a method for implementing questions for multidimensional forced selection testing according to one embodiment of the present application;
FIG. 3 is a schematic illustration of the implementation of the first selection step of the method for implementing the questions for the multidimensional forced selection test in one embodiment of the present application;
FIG. 4 is a schematic illustration of the application of the iStEM algorithm in the process of question bank construction in one embodiment of the present application;
FIG. 5 is a schematic diagram of a device for implementing questions for multidimensional forced selection test according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail below. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, based on the examples herein, which are within the scope of the protection sought by those of ordinary skill in the art without undue effort, are intended to be encompassed by the present application.
In order to facilitate understanding of the technical scene related to the application, each flow link for realizing talent evaluation in the scene of the application is briefly introduced below.
As shown in FIG. 1, the implementation of talent evaluation mainly comprises several links of question bank construction, question bank test, question bank project parameter calibration, test operation, question replenishment and iteration, project parameter calibration and new question calibration.
In the application, the question bank construction, the question bank test and the question bank item parameter calibration are collectively called as a pre-construction stage of the question bank, the test operation link refers to an actual test stage, the question selecting implementation method in the application is applied to the stage, then the question supplement and iteration, the item parameter calibration and the new question calibration are substantially the same as the question bank construction and the question bank item parameter calibration from the technical point of view, the method belongs to further improvement of the question bank, and the links are conceptually introduced.
Question bank construction
In the question bank construction link (or initial question bank construction link), a test designer needs to determine the number and content of the characteristics to be measured, namely one dimension, write item items according to operation definition of each dimension, only measure one dimension for each item, and finally combine the items into question-forcing blocks (one question-forcing block) according to a certain design, so that each question-forcing block contains a plurality of items with different dimensions, and the question-forcing test measures the characteristics of the tested person in different dimensions by requiring the tested person to select the item meeting the requirement of the tester in each question block. The question bank is now the first version of the test, in which there should be enough questions included to ensure measurement accuracy and accuracy of parameter estimation.
The design of the forced choice question block comprises two elements, wherein one element is the number of items contained in each question block, and the common design is a binary group, a ternary group and a quaternary group, namely, each question block comprises two, three and four items; the second type of answer format mainly comprises choosing (Pick, selecting the most suitable item from a plurality of items of a question block), the best and worst (mobile, selecting the most suitable item and the non-suitable item), and sorting (Rank, sorting the items in the question block according to the degree of being suitable for itself). In practice, the three answer formats of two to four elements, which cover most practical application requirements, are too complex for the testee to use and are not suitable for commercial evaluation and evaluation of most academic classes.
In this application, specific methods and processes for making the forced choice questions in the question bank in the related embodiments are explicitly described in the related literature, and are not described here in detail.
Question bank test
The test link of the question bank is mainly used for collecting a certain amount of test data so as to realize the preliminary estimation of the difficulty and distinguishing degree parameters of the questions in the question bank in the subsequent parameter calibration of the question bank project. During testing, each tested person participating in test answers to all the questions in the question bank, and the test can be implemented through an online test platform or through a mode of paper pen answer re-input in practical implementation. For example, if the question block is a triplet ordering question, there may be 6 answer results for each question, and the final test answer matrix (test data) is a matrix of the number of tested persons and the number of questions, and each element in the matrix is a certain value (used for representing a certain answer result) in 1-6.
Question bank project parameter calibration
In the item parameter calibration of the question bank, the difficulty and the distinguishing degree parameter of each item in the question bank are estimated by using the test data obtained in the last link. As known to those skilled in the art, the related basic principles of the project reaction theory IRT (Item Response Theory) are referred to herein, mainly in terms of the construction of a measurement model and the estimation of model parameters. Improvements relating to this aspect of the technical solution of the present application will be described in detail later, and will not be described in detail here.
Test operation
The link is the development of the actual test, and is realized through a relevant answer system, and CAT mentioned in the background technology can be realized for a certain individual subject based on the answer system, so that the test efficiency is improved. The final result of the link is to evaluate the subject according to the response data, and the evaluation is based on the statistical analysis of crowd data samples, and the description of the relative situation of individuals in the whole crowd is not artificial subjective evaluation.
Question replenishment and iteration, project parameter calibration and new question calibration
In practice, the initial question library often has many limitations, such as the measurement accuracy and efficiency cannot fully exert the advantages of adaptive questions, and meanwhile, the test questions may need to be replaced in the process of test operation. In some special forced choice tests, such as conditional inference tests, the questions lose measurement effectiveness due to increased exposure and therefore need to be replaced from time to time, i.e., question replenishment and iteration.
The project parameter calibration and the new question calibration are carried out by combining the new and old question parameter calibration based on the new and old question parameter calibration, which is characterized by expanding a statistical analysis sample in essence, so that the related estimated parameters are closer to ideal practice.
In a word, the two links of question supplementation and iteration, item parameter calibration and new question calibration are performed in the process of perfecting a question bank, and the two processes can be circularly performed based on the duration of test operation in practice.
Based on the above description of the technical scenario, the description of the method for implementing the questions for multidimensional forced selection test proposed in the present application is provided below.
As described in the background art, the feature dimensions measured simultaneously by the forced selection test applied commercially today are all above 20, but the influence on Fisher information matrix calculation when the dimensions are higher is not considered in the existing CAT implementation method; in a multidimensional forced selection test scene, more dimensions need to be measured, and the following problems can be caused by migrating the traditional Fisher information matrix-based self-adaptive question selection method: 1) The number of the prepositioned questions is unstable, and a plurality of questions can be tested to start computing the Fisher information matrix, so that the overall test efficiency is affected; 2) The same title is always selected in the pre-stage, so that part of the title is overexposed to influence the final test effect.
In view of this, the present application proposes a method for implementing a question for multidimensional forced selection test, where the method is applied to a test operation link in fig. 1, and an execution subject is a relevant answer system background, and in an embodiment, as shown in fig. 2, the method for implementing a question of the present application includes:
a first selecting step, based on random selection, of not repeatedly selecting a plurality of questions from a pre-constructed question library as first answer questions, and obtaining answer data obtained by answering each first answer question by a target subject, wherein the measured dimensions of all the first answer questions cover all the dimensions to be measured;
specifically, as a specific embodiment, in order to complete the first selection step as soon as possible, the selection stage is completed with the least amount of questions as possible, as shown in fig. 3, in the first selection step, the process of selecting the first answer sheet specifically includes:
step one, establishing a dimension vector comprising all dimensions to be measured, and randomly selecting a question from a question library as a current selected question, wherein the description is that in a question selecting stage, characteristic dimension information and related item parameters required to be measured of items of each question in the question library are known (in practice, the information is obtained and confirmed in a question library pre-construction stage and is built in a system);
step two, performing first marking on the title corresponding to the currently selected title in the title library, and removing the dimension measured by the title from the dimension vector;
step three, judging whether the vector number in the dimension vector is zero, if so, taking all the questions marked by the first marks as first answer questions and ending the selection process, otherwise, continuing to execute the step four;
and step four, respectively carrying out matching analysis on other topics which are not marked by the first mark in the topic library according to the dimension vector, correspondingly obtaining the dimension corresponding to the dimension measured by the corresponding topic as the dimension corresponding number of the dimension in the dimension vector, comparing the dimension corresponding to the other topics, taking the topic with the largest dimension corresponding number as the new currently selected topic, in other words, selecting the topic with the largest measured (or not related) dimension as the new currently selected topic, and jumping to execute the step two.
In the implementation of the actual test, in the implementation of answering with the target subject, based on specific requirements, the method can push a first marked question to the subject for answering and obtain corresponding answering data each time, or can obtain all the first answering questions at one time and push the first answering questions to the subject for answering and obtain answering data in any mode.
Continuing to return to fig. 2, after the first selecting step, performing a second selecting step, selecting a question from the remaining questions not selected in the question bank in a preset manner as a second answer question, acquiring answer data corresponding to the second answer question by the target subject, evaluating whether dynamic selecting based on Fisher information amount is possible or not based on all the acquired answer data (including the answer data obtained in the first selecting step) and item parameter information of the corresponding question, if yes, jumping to perform a third selecting step, otherwise continuing to perform the second selecting step;
it should be noted that the preset manner mentioned in the second selecting step may be any manner; for example, in an implementation scenario, for convenience of implementation, the preset manner is a random manner;
specifically, in the second selecting step, based on all the obtained answer data and the item parameter information of the corresponding questions, whether dynamic questions can be selected based on Fisher information amount is evaluated, including:
calculating a Fisher information matrix according to all acquired answer data and item parameter information of corresponding topics, wherein the related calculation mode is the same as that of the prior art, and the application is not repeated here;
when the Fisher information matrix obtained through calculation is reversible, determining that dynamic questions can be selected based on the Fisher information quantity, or determining that dynamic questions cannot be selected based on the Fisher information quantity.
As shown in fig. 2, in the technical solution of the present application, a third selecting step, based on a Fisher information amount method, selects a third answer question from the remaining questions not selected in the question library, and obtains answer data corresponding to the third answer question by the target subject until the answer condition of the target subject meets a preset requirement, if the estimation accuracy of the characteristic parameter of the target subject reaches a set standard or the test length reaches a set maximum length, the answer process of the selected question is ended, for example, the test length set based on experience is 80 questions, and if the answer condition of a subject has reached 80 questions, the answer process of the selected question is ended;
it should be noted that, the related principles of the Fisher information amount-based method or Fisher A/D method choice are already described in the related publications, and this is not repeated here; in the third selecting step, regarding the implementation process, the selecting the implementation process includes:
aiming at the residual questions which are not selected in the question library, respectively combining each residual question with the answered questions to obtain a question combination comprising the corresponding residual questions;
calculating the Fisher information matrix corresponding to each question combination, comparing the inverted trace of the obtained Fisher information matrix, and taking the remaining questions in the question combination corresponding to the Fisher information matrix with the smallest trace as the third selected answer questions.
Compared with the existing CAT topic selection implementation, the technical scheme of the application has the advantages that the prepositive control topic selection link is added before dynamic topic selection based on Fisher information quantity, the defect that topic selection is carried out by directly applying Fisher information quantity is avoided, and the method is beneficial to more efficiently completing related tests in a high-dimensional test scenario.
The following describes the relevant improvements related to the present application in the pre-construction stage of the topic library.
In the prior art, the most widely used is the TIRT model and the R packet of parameter estimation thereof, but the optimization space exists in both the model and the parameter estimation method, and in particular,
1) In the model, the TIRT model adopts the rule of the Compositon, simulates the psychological decision process of selecting a testee in a question block, and considers that all items in one question block are compared one by one, for example, in one question block containing three items ABC, the testee performs three times of AB/AC/BC comparison. This way of decision may make the information redundant, for example, when we confirm a > B, B > C, we can confirm the ordering of the three items, and the comparison of AC is the redundant information, and a > C can be obtained without the comparison. There is also a simpler decision process, the Luce selection axiom, which regards the ordering as a series of mutually independent processes for best selection, e.g. in the same triplet, the subject only needs to make two comparisons, in the first time, the choice most conforming to himself is selected from ABC, e.g. a; in the second time, the options more conforming to the user's own option, such as B, are selected from the remaining two BC items, so far the ordering of the three items can be confirmed. The application of the rule of comparison of the sephson in the quadruple requires 6 comparisons, whereas the advantage of the Luce selection axiom is greater, requiring only 3 comparisons.
Therefore, in the technical scheme of the application, the model is constructed based on the Luce selection axiom to obtain the problem block measurement model required by project parameter estimation, the model is favorable for saving the cognitive load, and the mathematical limitation specific to the multi-element group is increased through strict mathematical deduction, so that the model is more suitable for the sorting process of the tri-element group and the larger problem type and is named as a 2PL-RANK model. In other words, in the pre-construction process of the question bank, each question in the question bank is a fixed n-element forced choice item group question, where n is greater than or equal to 2, and it is easy to understand that "fixed" herein means that question blocks of different sizes cannot occur simultaneously in one test.
For example, in one embodiment, each topic in the topic library is a fixed quaternary forced choice item group topic, and the responses are ordered;
in the pre-construction process of the topic library, the expression description of the constructed topic block measurement model (2 PL-RANK model) comprises the following steps:
P j (a＞b＞c＞d|θ a ,θ b ,θ c ,θ d )＝P j (a＞b,c,d|θ a ,θ b ,θ c ,θ d )P j (b＞c,d|θ b ,θ c ,θ d )P j (c＞d|θ c ,θ d ) (3)
in the expressions (3) to (7),
a, b, c, d represent four items in the question block j,
θ a ,θ b ,θ c ,θ d corresponding to the characteristic parameters representing the dimensions tested by items a, b, c, d respectively,
P j (a＞b＞c＞d|θ a ,θ b ,θ c ,θ d ) The probability that the subject selects a > b > c > d when answering block j is indicated,
P j probability of selection, Q j (*)＝1-P j The probability of not choosing is indicated,
α j* the distinguishing parameter of the item representing the question block j,
β j* the difficulty parameter representing an item in the question block j.
It will be readily appreciated that a > b > c > d in the above description of the model expression is merely an example of one possible outcome, and that the probability of each possible outcome is calculated in practice.
In the technical scheme of the application, the mathematical constraint of the block measurement model comprises that the sum of difficulty parameters of all items in each block is zero, so that because the difficulty of a plurality of items in one block is relative rather than absolute, for example, in a triplet consisting of abc three items, the difficulty of a is relatively high, which indicates that the probability of selecting a is relatively small under the same capability, but the small is relatively small, only smaller than b and c, and the sum of the difficulty of abc is unchanged, and the accuracy of model parameter estimation can be increased by adding the constraint in an expanded model.
2) In the prior art, in parameter estimation, a TIRT model is generally estimated by using a Mplus software by using a least square method, or related R packets are called to be estimated by using an MCMC algorithm, but the Mplus software cannot generate Standard Errors (SE), can not converge in a high-dimensional environment, and occupies a higher memory; the MCMC algorithm has long estimation time, and each parameter estimation needs to be self-coded, so that the method is not convenient.
In the technical scheme of the application, in the pre-construction process of a question bank, model parameter estimation is carried out by adopting an iStEM algorithm based on a question measurement model according to obtained test response data aiming at the question bank, so as to obtain item parameter information of each question.
As shown in fig. 4, in practical implementation, the algorithm is implemented as follows:
(1) E, step E: extracting capability parameters from the gibbs sampler;
(2) M steps: the BFGS algorithm is utilized to maximize likelihood functions, and difficulty and distinguishing degree parameters are estimated;
and (5) continuously iterating the step E and the step M, and storing parameter estimation results of each group of iterations. When the Geweke statistic reaches the standard value, only the last 10 groups of iteration results are reserved, iteration is continued, and the parameter estimation result of each group of iteration is reserved; and stopping iteration when the group variance reaches a standard value, and calculating the average value of all the group iteration results as a final estimated value.
The algorithm belongs to an EM algorithm, and because the estimation result of the M steps forms a time-aligned Markov chain, the difficulty and the distinguishing degree parameter obtained by each iteration can be stably converged to a true value finally. The convergence characteristic of the Markov chain ensures the accuracy of sampling, and compared with the MCMC algorithm which completely relies on sampling, the iStEM algorithm introduces a Newton-like algorithm in the M step, so that the efficiency is improved, and therefore, the algorithm has enough accuracy and higher time efficiency than the MCMC algorithm.
In the technical scheme, the time efficiency and the estimation precision of the project parameter estimation implementation mode are higher, so that the calculation and time cost are saved, and the test development and iteration are facilitated; the measurement model has strong flexibility and wider question type adaptation, and is beneficial to meeting different test requirements.
Fig. 5 is a schematic structural diagram of a topic implementation device for multidimensional forced selection testing according to an embodiment of the present application, as shown in fig. 5, the topic implementation device 500 includes:
a first selection processing module 501, configured to, in a first selection process, based on random selection, not repeatedly select a plurality of questions from a pre-constructed question library as first answer questions, and obtain answer data obtained by answering each first answer question by a target subject, where all dimensions measured by the first answer questions cover all dimensions to be measured;
the second selection processing module 502 is configured to select, in a second selection process, a question from the remaining questions that are not selected in the question bank in a preset manner, as a second answer question, obtain answer data corresponding to the second answer question by the target subject, evaluate whether dynamic selection of the question can be performed based on the Fisher information amount based on all the obtained answer data and item parameter information of the corresponding question, and skip the third selection step if yes, otherwise continue the second selection step;
and a third selection processing module 503, configured to select a third answer question from the remaining questions not selected in the question bank based on the Fisher information method in the third selection process, and obtain answer data of the target subject for answering the corresponding third answer question until the question answer condition of the target subject meets the preset requirement.
With regard to the topic implementation apparatus 500 in the related embodiment described above, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application, as shown in fig. 6, the electronic device 600 includes:
a memory 601 on which an executable program is stored;
a processor 602 for executing the executable program in the memory 601 to implement the steps of the above method.
The specific manner in which the processor 602 executes the program in the memory 601 of the electronic device 600 in the above embodiment has been described in detail in the embodiment related to the method, and will not be described in detail here.
The present invention is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the scope of the present invention are intended to be included in the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims (10)
1. The method for realizing the questions for the multidimensional forced selection test is characterized by comprising the following steps:
a first selecting step, based on random selection, of not repeatedly selecting a plurality of questions from a pre-constructed question library as first answer questions, and obtaining answer data obtained by answering each first answer question by a target subject, wherein the measured dimensions of all the first answer questions cover all the dimensions to be measured;
a second selecting step, namely selecting the questions from the rest questions which are not selected in the question library in a preset mode as second answer questions, acquiring answer data corresponding to the second answer questions by the target subjects, evaluating whether dynamic questions can be selected based on Fisher information amount or not based on all acquired answer data and item parameter information of the corresponding questions, if yes, jumping to execute a third selecting step, otherwise, continuing to execute the second selecting step;
and a third selecting step, namely selecting a third answer question from the rest questions which are not selected in the question library based on the Fisher information amount, and acquiring answer data of the corresponding third answer questions of the target subject until the question answer condition of the target subject meets the preset requirement.
2. The method for realizing question selection according to claim 1, wherein in the first selecting step, the process of selecting the first answer item specifically includes:
step one, establishing a dimension vector comprising all dimensions to be measured, and randomly selecting a question from the question library as a current selected question;
step two, performing first marking on the title corresponding to the currently selected title in the title library, and removing the dimension measured by the title from the dimension vector;
judging whether the vector number in the dimension vector is zero, if so, taking all the questions marked by the first marks as the first answer questions and ending the selection process, otherwise, continuing to execute the fourth step;
and step four, respectively carrying out matching analysis on other topics which are not marked by the first mark in the topic library according to the dimension vector, correspondingly obtaining the dimension coincidence number of the dimension measured by the corresponding topic as the dimension in the dimension vector, comparing the dimension coincidence number corresponding to the other topics, taking the topic with the largest dimension coincidence number as the new currently selected topic, and jumping to execute the step two.
3. The method according to claim 1, wherein in the second selecting step, the evaluating whether the dynamic topic can be selected based on the Fisher information amount based on all the obtained answer data and the item parameter information of the corresponding topic includes:
calculating a Fisher information matrix according to all acquired answer data and item parameter information of the corresponding subject;
when the Fisher information matrix obtained through calculation is reversible, determining that dynamic questions can be selected based on the Fisher information quantity, or determining that dynamic questions cannot be selected based on the Fisher information quantity.
4. The method of claim 1, wherein in the third selecting step, a third answer question is selected from the remaining questions not selected in the question bank based on a Fisher information amount method, including:
combining each of the residual questions with the answered questions respectively aiming at the unselected residual questions in the question library to obtain a question combination comprising the corresponding residual questions;
calculating the Fisher information matrix corresponding to each question combination, comparing the inverted trace of the obtained Fisher information matrix, and taking the remaining questions in the question combination corresponding to the Fisher information matrix with the smallest trace as the third answer questions selected at the time.
5. The method of claim 1, wherein each topic in the topic library is a fixed n-gram topic, where n is greater than or equal to 2.
6. The method according to claim 5, wherein in the pre-construction process of the topic library, model construction is performed based on the Luce selection axiom to obtain a topic block measurement model required for project parameter estimation;
the mathematical constraint of the block measurement model comprises that the sum of difficulty parameters of all items in each block is zero.
7. The method of claim 6, wherein each topic in the topic library is a fixed quaternary forced choice item group topic, and the responses are ordered;
in the pre-construction process of the topic library, the expression description of the constructed topic block measurement model comprises the following steps:
P j (a＞b＞c＞d|θ a ,θ b ,θ c ,θ d )＝P j (a＞b,c,d|θ a ,θ b ,θ c ,θ d )P j (b＞c,d|θ b ,θ c ,θ d )P j (c＞d|θ c ,θ d )
wherein,
a, b, c, d represent four items in the question block j,
θ a ,θ b ,θ c ,θ d corresponding to the trait parameters representing the dimensions tested for by items a, b, c, d,
P j (a＞b＞c＞d|θ a ,θ b ,θ c ,θ d ) The probability that the subject selects a > b > c > d when answering block j is indicated,
P j probability of selection, Q j (*)＝1-P j The probability of not choosing is indicated,
α j* item representing the topic block jIs used for the degree of distinction parameter of (a),
β j* the difficulty parameter representing an item in the question block j.
8. The method according to claim 6, wherein in the pre-construction process of the question bank, model parameter estimation is performed by using an iStEM algorithm based on the question measurement model according to the obtained test response data for the question bank, so as to obtain item parameter information of each question.
9. A choice question realizing device for multidimensional forced selection test, comprising:
the first selecting and processing module is used for not repeatedly selecting a plurality of questions from a pre-constructed question library to serve as first answer questions based on random selection in a first selecting process, and acquiring answer data obtained by answering each first answer question by a target subject, wherein the measured dimensions of all the first answer questions cover all the dimensions to be measured;
the second selecting and processing module is used for selecting the questions from the rest questions which are not selected in the question library in a preset mode as second answer questions and obtaining answer data of corresponding second answer questions of a target subject, evaluating whether dynamic selection questions can be carried out based on Fisher information quantity or not based on all the obtained answer data and item parameter information of the corresponding questions, if yes, skipping to execute a third selecting step, otherwise, continuing to execute the second selecting step;
and the third selecting and processing module is used for selecting a third answer question from the rest questions which are not selected in the question library based on the Fisher information amount method in the third selecting process, and acquiring answer data of the corresponding third answer questions of the target subject until the question answer condition of the target subject meets the preset requirement.
10. An electronic device, comprising:
a memory having an executable program stored thereon;
a processor for executing the executable program in the memory to implement the steps of the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311655474.5A CN117670269B (en) | 2023-12-05 | 2023-12-05 | Method and device for realizing questions for multidimensional forced selection test and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311655474.5A CN117670269B (en) | 2023-12-05 | 2023-12-05 | Method and device for realizing questions for multidimensional forced selection test and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117670269A true CN117670269A (en) | 2024-03-08 |
| CN117670269B CN117670269B (en) | 2024-06-21 |
Family
ID=90069347
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311655474.5A Active CN117670269B (en) | 2023-12-05 | 2023-12-05 | Method and device for realizing questions for multidimensional forced selection test and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117670269B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100280985A1 (en) * | 2008-01-14 | 2010-11-04 | Aptima, Inc. | Method and system to predict the likelihood of topics |
| US20160019803A1 (en) * | 2014-07-21 | 2016-01-21 | New York University | System, method and computer-accessible medium for scalable testing and evaluation |
| CN106871909A (en) * | 2017-02-20 | 2017-06-20 | 中国人民解放军国防科学技术大学 | Pulsar satellite selection method based on Fisher information matrix under a kind of multi spacecraft system |
| CN108346030A (en) * | 2017-12-29 | 2018-07-31 | 北京北森云计算股份有限公司 | Computer adaptive ability testing method and device |
| CN110287103A (en) * | 2019-05-22 | 2019-09-27 | 深圳壹账通智能科技有限公司 | Software product assessment processing method, device, computer equipment and storage medium |
| CN110428911A (en) * | 2019-07-24 | 2019-11-08 | 北京智鼎优源管理咨询有限公司 | Adaptive assessment method and equipment |
| CN111554143A (en) * | 2020-03-31 | 2020-08-18 | 北京学格科技有限公司 | Evaluation method and device based on CO-MIRT algorithm model |
| US20210278548A1 (en) * | 2020-02-13 | 2021-09-09 | Mitsubishi Electric Research Laboratories, Inc. | System and Method for Integer-Less GNSS Positioning |
| CN114691856A (en) * | 2022-04-20 | 2022-07-01 | 平安科技（深圳）有限公司 | Topic recommendation method, device, equipment and medium |
| CN116109454A (en) * | 2023-02-09 | 2023-05-12 | 科大讯飞股份有限公司 | Method, device, equipment and storage medium for determining question difficulty in capability evaluation |
| CN116562836A (en) * | 2023-06-27 | 2023-08-08 | 北森云计算有限公司 | Method, device, electronic equipment and storage medium for multidimensional forced choice question character test |
-
2023
- 2023-12-05 CN CN202311655474.5A patent/CN117670269B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100280985A1 (en) * | 2008-01-14 | 2010-11-04 | Aptima, Inc. | Method and system to predict the likelihood of topics |
| US20160019803A1 (en) * | 2014-07-21 | 2016-01-21 | New York University | System, method and computer-accessible medium for scalable testing and evaluation |
| CN106871909A (en) * | 2017-02-20 | 2017-06-20 | 中国人民解放军国防科学技术大学 | Pulsar satellite selection method based on Fisher information matrix under a kind of multi spacecraft system |
| CN108346030A (en) * | 2017-12-29 | 2018-07-31 | 北京北森云计算股份有限公司 | Computer adaptive ability testing method and device |
| CN110287103A (en) * | 2019-05-22 | 2019-09-27 | 深圳壹账通智能科技有限公司 | Software product assessment processing method, device, computer equipment and storage medium |
| CN110428911A (en) * | 2019-07-24 | 2019-11-08 | 北京智鼎优源管理咨询有限公司 | Adaptive assessment method and equipment |
| US20210278548A1 (en) * | 2020-02-13 | 2021-09-09 | Mitsubishi Electric Research Laboratories, Inc. | System and Method for Integer-Less GNSS Positioning |
| CN111554143A (en) * | 2020-03-31 | 2020-08-18 | 北京学格科技有限公司 | Evaluation method and device based on CO-MIRT algorithm model |
| CN114691856A (en) * | 2022-04-20 | 2022-07-01 | 平安科技（深圳）有限公司 | Topic recommendation method, device, equipment and medium |
| CN116109454A (en) * | 2023-02-09 | 2023-05-12 | 科大讯飞股份有限公司 | Method, device, equipment and storage medium for determining question difficulty in capability evaluation |
| CN116562836A (en) * | 2023-06-27 | 2023-08-08 | 北森云计算有限公司 | Method, device, electronic equipment and storage medium for multidimensional forced choice question character test |
Non-Patent Citations (5)
| Title |
|---|
| JUAN LIU等: "A Multi-Unidimensional Pairwise-Preference Model for RANK Response Format Data", 《WWW.PSYARXIV.COM/RYGU8/DOWNLOAD》, 31 December 2021 (2021-12-31), pages 4 - 5 * |
| 吴金翔;任山章;: "生物学教材习题中科学推理能力的定量分析", 生物学通报, no. 03, 20 March 2020 (2020-03-20) * |
| 徐玲玲: "多维计算机化多阶段自适应测验研究：自动组卷算法及其路由规则", 《中国优秀硕士论文》, 15 March 2020 (2020-03-15), pages 16 * |
| 毛秀珍;辛涛;: "多维计算机化自适应测验:模型、技术和方法", 心理科学进展, no. 05, 15 May 2015 (2015-05-15) * |
| 毛秀珍;辛涛;: "计算机化自适应测验选题策略述评", 心理科学进展, no. 10, 15 October 2011 (2011-10-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117670269B (en) | 2024-06-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111651676B (en) | Method, device, equipment and medium for performing occupation recommendation based on capability model | |
| CN107203467A (en) | The reference test method and device of supervised learning algorithm under a kind of distributed environment | |
| CN110046706A (en) | Model generating method, device and server | |
| CN111310918B (en) | Data processing method, device, computer equipment and storage medium | |
| KR20180061998A (en) | System and method for diagnosing mass attributes of leaner | |
| CN113850512A (en) | User ability grading method, device, equipment and medium based on self-adaptive evaluation | |
| CN116562836B (en) | Method, device, electronic equipment and storage medium for multidimensional forced choice question character test | |
| CN117670269B (en) | Method and device for realizing questions for multidimensional forced selection test and electronic equipment | |
| CN113361780A (en) | Behavior data-based crowdsourcing tester evaluation method | |
| Wu et al. | Toward effective automated content analysis via crowdsourcing | |
| CN116975520A (en) | Reliability evaluation method, device, equipment and storage medium for AB experiment | |
| CN114860617B (en) | Intelligent pressure testing method and system | |
| CN113393023B (en) | Mold quality evaluation method, apparatus, device and storage medium | |
| Song et al. | Efficient and robust model benchmarks with item response theory and adaptive testing | |
| CN115203556A (en) | Score prediction model training method and device, electronic equipment and storage medium | |
| CN115129834A (en) | Question answering method and system | |
| CN113827981A (en) | Game loss user prediction method and system based on naive Bayes | |
| CN111242235B (en) | Similar characteristic test data set generation method | |
| CN110069783A (en) | A kind of answer content evaluating method and device | |
| US20220237459A1 (en) | Generation method, computer-readable recording medium storing generation program, and information processing apparatus | |
| US20220040532A1 (en) | Utilizing machine learning and cognitive state analysis to track user performance | |
| US20240029882A1 (en) | Diagnostic classification device and method | |
| US20220237463A1 (en) | Generation method, computer-readable recording medium storing generation program, and information processing apparatus | |
| US20220222582A1 (en) | Generation method, computer-readable recording medium storing generation program, and information processing apparatus | |
| CN111507639B (en) | Financing risk analysis method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |