US20050125196A1 - Method and system for computer-assisted test construction performing specification matching during test item selection - Google Patents

Method and system for computer-assisted test construction performing specification matching during test item selection Download PDF

Info

Publication number
US20050125196A1
US20050125196A1 US10731776 US73177603A US2005125196A1 US 20050125196 A1 US20050125196 A1 US 20050125196A1 US 10731776 US10731776 US 10731776 US 73177603 A US73177603 A US 73177603A US 2005125196 A1 US2005125196 A1 US 2005125196A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
test
item
items
specifications
selected
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10731776
Inventor
Len Swanson
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Educational Testing Service
Original Assignee
Educational Testing Service
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management
    • G06F11/3684Test management for test design, e.g. generating new test cases
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00Electrically-operated teaching apparatus or devices working with questions and answers
    • G09B7/02Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00Electrically-operated teaching apparatus or devices working with questions and answers
    • G09B7/02Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
    • G09B7/04Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation

Abstract

A method and system for constructing a test using a computer system that performs specification matching during the test creation process is disclosed. A test developer determines one or more test item databases from which to select test items. The test item databases are organized based on psychometric and/or content specifications. The developer can examine the textual passages, artwork or statistical information pertaining to a test item before selecting it by clicking on a designation of the test item in a database. The developer can then add the test item to a list of test items for the test. The test development system updates pre-designated psychometric and content specification information as the developer adds each test item to the test. The test developer can use the specification information to determine whether to add to, subtract from, or modify the list of test items selected for the test.

Description

    FIELD OF THE INVENTION
  • [0001]
    The present invention generally relates to the field of test construction. The present invention particularly relates to a method and system for constructing a test using a computer system. Specifically, the present invention relates to a method and system for constructing a test using a computer system that performs specification matching during the test creation process.
  • BACKGROUND OF THE INVENTION
  • [0002]
    Testing services administer a variety of standardized tests. For example, the Graduate Management Admission Test® (GMAT®) evaluates graduate business school applicants by measuring general verbal, mathematical, and analytical writing skills. The Graduate Record Examinations® (GRE®) assists graduate schools and departments in graduate admissions activities. Tests offered include the General Test, which measures developed verbal, quantitative, and analytical abilities, and the Subject Tests, which measure achievement in 14 different fields of study. The Scholastic Assessment Test® (SAT®) Program includes the SAT I: Reasoning Test and SAT II: Subject Tests. The SAT I is a three-hour test, primarily multiple-choice, that measures verbal and mathematical reasoning abilities. The SAT II: Subject Tests are one-hour, mostly multiple-choice, tests in specific subjects. These tests measure knowledge of particular subjects and the ability to apply that knowledge. Colleges and universities typically use the SAT® Program as a factor in determining admission or placement of prospective students. Individual states also administer tests to determine whether and to what extent students meet state standards for educational achievement.
  • [0003]
    Many tests, such as the above-mentioned tests, are offered multiple times during a year and/or are administered over multiple years. It is important, in the case of tests that are offered multiple times during a year, that the different administrations of each test be approximately equal in difficulty in order to properly rate examinees from different testing dates against one another. For tests that are administered over multiple years, it is important that each test be of a known difficulty level to accurately assess an examinee's performance and progress. Moreover, it may be important to evaluate other psychometric specifications and statistical properties for a given test prior to its administration.
  • [0004]
    Some current methods for constructing tests, including those using a computer interface, permit a test developer to view and select test items. Other methods can display a match between content specifications and the content properties of the selected test items. For example, such test construction systems typically keep track of metrics such as the number of questions that test a particular subject. On the SAT I, for example, questions are divided into mathematics and verbal questions. Additionally, the test construction system could also keep track of the number of questions that are devoted to a sub-topic (such as geometry or algebra) or that are presented in a certain format (such as an analogy completion, sentence completion or word problem). By identifying the number of questions of a particular type included in the developed test, the test developer may be alerted if an incorrect number of questions or an incorrect number of questions of a particular type are included in the test.
  • [0005]
    However, systems implementing these methods do not combine all of the features listed above to permit the test developer to develop tests more quickly, while at the same time including the ability to determine if the selected test items meet psychometric specifications for a test and also permitting a test developer to examine content or psychometric specifications during the test development process so that the test developer can add, remove or replace test items to adjust for deficiencies with respect to test specifications during the test item selection process.
  • [0006]
    Thus, a need exists for an evaluation tool that determines whether defined content and psychometric specifications for a test are met by a particular question set.
  • [0007]
    A further need exists for providing psychometric and statistical information to a test creator during the test creation process to permit evaluation and adjustment of the selected test items during the test creation process.
  • SUMMARY OF PREFERRED EMBODIMENTS
  • [0008]
    Before the present methods, systems, and materials are described, it is to be understood that this invention is not limited to the particular methodologies, systems and materials described, as these may vary. It is also to be understood that the terminology used in the description is for the purpose of describing the particular versions or embodiments only, and is not intended to limit the scope of the present invention which will be limited only by the appended claims.
  • [0009]
    It must also be noted that as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Thus, for example, reference to a “test item” is a reference to one or more test items and equivalents thereof known to those skilled in the art, and so forth. Unless defined otherwise, all technical and scientific terms used herein have the same meanings as commonly understood by one of ordinary skill in the art. Although any methods, materials, and devices similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, the preferred methods, materials, and devices are now described. All publications mentioned herein are incorporated by reference. Nothing herein is to be construed as an admission that the invention is not entitled to antedate such disclosure by virtue of prior invention.
  • [0010]
    The present invention includes a database of test items and information regarding test items and tests. The information may include the content structure and statistical properties of test items and the content and psychometric specifications for tests to be constructed from the database. Psychometric specifications include specifications related to the measurement of human characteristics. Psychometric specifications may be used to develop tests in areas such as intelligence testing, personality testing and vocational testing.
  • [0011]
    The present invention may further include a systematic procedure for selecting items for a test using a computer system connected to the database described above. As the test is being constructed, the extent to which the developing test matches the content and psychometric specifications for the test may be displayed so that a developer may adjust the set of items selected to best match those specifications.
  • [0012]
    In a preferred embodiment, a method of constructing a test includes selecting a test item for inclusion in a set of selected test items, updating at least one evaluation statistic based on the selected test item, and revising the set of selected test items to substantially correlate the at least one evaluation statistic with at least one specification for a test. The test item may be selected at least in part based on a subject matter for the test item. The at least one evaluation statistic may be selected from content specifications and psychometric specifications. The at least one specification may also be selected from content specifications and psychometric specifications. The content specifications may include a number of test items to be presented in each of one or more pre-determined formats, a total number of test items to be included in the set of selected test items, a number of test items for testing each of one or more pre-determined subject matters, a key distribution, a percentage of test items having one or more pre-defined characteristics, a gender or racial orientation of test items, and a language in which the test items are presented. The psychometric specifications may include an overall test difficulty rating, a correlation between a correct response for a selected test item and a particular cognitive or behavioral trait, an orientation of the presentation of questions and answers for the set of selected test items, a number of pages of text for a test, a mean point-biserial, a mean r-biserial, and an arrangement of the set of selected test items.
  • [0013]
    In a preferred embodiment, a method for constructing a test includes selecting a portion of a test item database from which to select a set of test items for a test having one or more test specifications, displaying information concerning a plurality of test items in the selected portion of the test item database, examining a test item on a display device, selecting the test item for the test, and updating a value for at least one test specification based on specified properties for the selected test item. Selecting a portion of a test item database may be based on the subject matter of the test items contained within the portion of the test item database. Examining a test item may include viewing an image of the test item, statistical properties of the test item, text passages associated with the test item, an answer key, detailed content specifications, reviewers' comments, scoring guidelines, and artwork associated with the test item. The statistical properties may include one or more of a percentage of correct responses for the test item, t-biserials, r-biserials, item response theory parameters, gender-based response statistics, race-based response statistics, a percentage of responses choosing each distractor, and a frequency of previous usage for the test item. Updating a value for at least one test specification may be performed using item response theory. In an embodiment, the method further includes comparing current values for the one or more test specifications with required values for the one or more test specifications. In an embodiment, the method further includes replacing one or more test items in the set of selected test items based on the one or more updated specifications. In an embodiment, the method further includes adding one or more test items to the set of selected test items based on the one or more updated specifications. In an embodiment, the method further includes removing one or more test items from the set of selected test items based on the one or more updated specifications.
  • [0014]
    In a preferred embodiment, a system for constructing a test includes a processor, a computer-readable medium operably connected to the processor, and a display. The computer-readable medium contains one or more databases each having a plurality of test items. Each test item includes a textual question and one or more answers for the test item, a content structure of the test item, and one or more statistical properties for the test item. The statistical properties may include a percentage of correct responses for the test item, t-biserials, r-biserials, item response theory parameters, gender-based response statistics, race-based response statistics, a percentage of responses choosing each distractor, and a frequency of previous usage for the test item. The computer-readable medium may further include content specifications for a test, and psychometric specifications for the test. In an embodiment, the processor evaluates the content specifications and psychometric specifications for a test while the test is being created and determines a correlation value between the properties of the plurality of test items for the test and the content specifications and psychometric specifications for the test. The display displays the correlation value to the test developer. In an embodiment, the computer-readable medium further contains instructions for performing a method of constructing a test including selecting a test item for inclusion in a set of selected test items, updating at least one evaluation statistic based on the selected test item, and revising the set of selected test items to substantially correlate the at least one evaluation statistic with at least one specification for a test.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0015]
    The accompanying drawings, which are incorporated in and form a part of the specification, illustrate preferred embodiments of the present invention and, together with the description serve to explain the principles of the invention. The embodiments illustrated in the drawings should not be read to constitute limiting requirements, but instead are intended to assist the reader in understanding the invention.
  • [0016]
    FIG. 1 depicts an exemplary process flow for creating a test according to an embodiment of the present invention.
  • [0017]
    FIG. 2 depicts an exemplary system for creating a test according to an embodiment of the present invention.
  • DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
  • [0018]
    The present invention relates to a method and system for constructing a test using a computer system. Specifically, the present invention relates to a method and system for constructing a test using a computer system that performs specification matching during the test creation process.
  • [0019]
    FIG. 1 depicts an exemplary process flow for creating a test according to an embodiment of the present invention. First, the test developer may determine 105 the portion of the test item database or a particular test item database from which to select a test item for the test. The test item database may be a single repository containing all selectable test items for one or more tests. Alternatively, a test developer may choose questions from a plurality of test item databases. The test items contained within a portion of the test item database or within a particular test item database may possess distinguishing characteristics. For example, a particular test item database may contain test items pertaining only to questions for testing knowledge of geometric principles. The characteristics used to distinguish test items within a portion of the test item database or a particular test item database from other test items or test item databases may correspond to psychometric specifications or content specifications for a test. When the test item database or databases are organized in this manner, the test developer may more quickly modify the set of selected test items to match specifications that are not satisfied during test construction.
  • [0020]
    The test developer may then view 110 a spreadsheet displaying information about the test items within the portion of the test item database or the particular test item database. A computer system may be used to display the spreadsheet. The test developer may examine 115 an individual test item on a display device of the computer system prior to adding the test item to the test under development. Examining 115 an individual test item may include viewing the test item image, information about the test item and information about related entities such as text passages or artwork associated with the test item. Test item information may include, for example, an answer key, detailed content specifications, reviewers' comments, scoring guidelines (for constructed-response test items), statistical properties for the test item, such as a percentage of correct answers received if the test item was previously administered, t-biserials, r-biserials, item response theory parameters, gender-based response statistics, race-based response statistics, a percentage of responses choosing each distractor, and the frequency with which the question or a similar variant has been included on previous test administrations.
  • [0021]
    Upon reviewing the information, the test developer may select 120 individual test items for inclusion in the test. As the developer selects test items, the resulting correlation between content and psychometric specifications for the test and the corresponding characteristics of the user-selected test may be updated 125. If the selected test items do not meet one or more specifications, the developer may revise 130 his or her test item selection until a best matching between the test item properties and the content and psychometric specifications is achieved. In an exemplary embodiment, item response theory may be used in the matching of test item properties to test specifications. In an alternate embodiment, the percentage of examinees that answer a test item correctly may be used in the matching of test item properties to test specifications. These methods of determining test item properties are merely exemplary and are not meant to be limiting. Additional methods for determining test item properties may be performed singly or in combination with the above-listed methods and are intended to be encompassed within the scope of the present invention without limitation.
  • [0022]
    FIG. 2 depicts an exemplary system for creating a test according to an embodiment of the present invention. The system may include a computer system 200 containing a processor 205, a display 210 and a computer-readable medium 215, such as a hard drive, a floppy disk, a CD, a DVD, RAM, ROM, EPROM, EEPROM or other memory or memory storage device. The computer-readable medium 215 may contain a database 220 including potential test items and information about test items and tests.
  • [0023]
    The information about the test items may include content structure and statistical properties of the test items. The content structure may denote the format of the question, the information being tested, the style of question, and similar content-related information. The statistical properties may include the percentage of examinees that select a particular response, the correlation between selecting a particular response and exhibiting a particular personality trait (in the case of behavioral or psychological testing), and the like.
  • [0024]
    The information regarding the tests may include content specifications and psychometric specifications for tests to be constructed from the database 220. The content specifications may list requirements for the test such as requiring a certain percentage of test items to be in multiple-choice format, to test verbal skills, or to be of a specified length. Content specifications may also include, without limitation, specifying the overall test length, the number of test items presented on a particular topic, a key distribution, the percentage of test items with particular characteristics, a gender or racial orientation of items and the language in which the test is presented. Psychometric specifications may include, without limitation, a preferred overall test difficulty, a correlation between a correct response for a test item and a particular cognitive or behavioral trait, the orientation of the presentation of questions and answers for test items, mean point-biserials, mean r-biserials and the visual presentation of the testing materials.
  • [0025]
    The above-listed specifications are merely representative of specifications and properties that may be included in the database 220. It will be evident to one of skill in the art that more or fewer properties and specifications may be included in the database and still be within the scope of the invention.
  • [0026]
    The computer-readable medium 215 or a second computer-readable medium 225 operably connected to the processor 205 may contain a computer program for implementing a systematic procedure for selecting items for a test. The program may display the extent to which a test under development matches the content and psychometric specifications as a user constructs the test. In this way, the user may replace, remove, or add one or more test items to the set of selected test items to best match those specifications in an efficient manner.
  • [0027]
    Although the invention has been described with reference to the preferred embodiments, it will be apparent to one skilled in the art that variations and modifications are contemplated within the spirit and scope of the invention. The drawings and description of the preferred embodiments are made by way of example rather than to limit the scope of the invention, and it is intended to cover within the spirit and scope of the invention all such changes and modifications.

Claims (20)

  1. 1. A method of constructing a test, comprising:
    selecting a test item for inclusion in a set of selected test items;
    updating at least one evaluation statistic based on the selected test item; and
    revising the set of selected test items to substantially correlate the at least one evaluation statistic with at least one specification for a test.
  2. 2. The method of claim 1 wherein the test item is selected at least in part based on a subject matter for the test item.
  3. 3. The method of claim 1 wherein the at least one evaluation statistic is selected from a group including content specifications and psychometric specifications.
  4. 4. The method of claim 1 wherein the at least one specification is selected from a group including content specifications and psychometric specifications.
  5. 5. The method of claim 4 wherein the content specifications comprise one or more of the following:
    a number of test items to be presented in each of one or more pre-determined formats;
    a total number of test items to be included in the set of selected test items;
    a number of test items for testing each of one or more pre-determined subject matters;
    a key distribution;
    a percentage of test items having one or more pre-defined characteristics;
    a gender or racial orientation of test items; and
    a language in which the test items are presented.
  6. 6. The method of claim 4 wherein the psychometric specifications comprise one or more of the following:
    an overall test difficulty rating;
    a correlation between a correct response for a selected test item and a particular cognitive or behavioral trait;
    an orientation of the presentation of questions and answers for the set of selected test items;
    a number of pages of text for a test;
    a mean point-biserial;
    a mean r-biserial; and
    an arrangement of the set of selected test items.
  7. 7. A method for constructing a test, comprising:
    selecting a portion of a test item database from which to select a set of test items for a test, wherein the test includes one or more test specifications;
    displaying information concerning a plurality of test items in the selected portion of the test item database;
    examining a test item on a display device;
    selecting the test item for the test; and
    updating a value for at least one test specification based on specified properties for the selected test item.
  8. 8. The method of claim 7 wherein selecting a portion of a test item database comprises selecting a portion of a test item database based on a subject matter for test items contained within the portion of the test item database.
  9. 9. The method of claim 7 wherein examining a test item comprises viewing one or more of the following:
    an image of the test item;
    statistical properties of the test item;
    text passages associated with the test item;
    an answer key;
    detailed content specifications;
    reviewers' comments;
    scoring guidelines; and
    artwork associated with the test item.
  10. 10. The method of claim 7 wherein the statistical properties comprise one or more of the following:
    a percentage of correct responses for the test item;
    t-biserials;
    r-biserials;
    item response theory parameters;
    gender-based response statistics;
    race-based response statistics;
    a percentage of responses choosing each distractor; and
    a frequency of previous usage for the test item.
  11. 11. The method of claim 7 wherein updating a value for at least one test specification includes using item response theory.
  12. 12. The method of claim 7, further comprising:
    comparing current values for the one or more test specifications with required values for the one or more test specifications.
  13. 13. The method of claim 7, further comprising:
    replacing one or more test items in the set of selected test items based on the one or more updated specifications.
  14. 14. The method of claim 7, further comprising:
    adding one or more test items to the set of selected test items based on the one or more updated specifications.
  15. 15. The method of claim 7, further comprising:
    removing one or more test items from the set of selected test items based on the one or more updated specifications.
  16. 16. A system for constructing a test, comprising:
    a processor;
    a computer-readable medium operably connected to the processor; and
    a display,
    wherein the computer-readable medium contains one or more databases each having a plurality of test items, wherein each test item comprises:
    a textual question and one or more answers for the test item,
    a content structure of the test item, and
    one or more statistical properties for the test item.
  17. 17. The system of claim 16 wherein the statistical properties comprise one or more of the following:
    a percentage of correct responses for the test item;
    t-biserials;
    r-biserials;
    item response theory parameters;
    gender-based response statistics;
    race-based response statistics;
    a percentage of responses choosing each distractor; and
    a frequency of previous usage for the test item.
  18. 18. The system of claim 16 wherein the computer-readable medium further comprises:
    content specifications for a test; and
    psychometric specifications for the test.
  19. 19. The system of claim 18 wherein the processor evaluates the content specifications and psychometric specifications for a test while the test is being created, wherein the processor determines a correlation value between the properties of the plurality of test items for the test and the content specifications and psychometric specifications for the test, and wherein the display displays the correlation value.
  20. 20. The method of claim 16 wherein the computer-readable medium further contains instructions for performing a method of constructing a test comprising:
    selecting a test item for inclusion in a set of selected test items;
    updating at least one evaluation statistic based on the selected test item; and
    revising the set of selected test items to substantially correlate the at least one evaluation statistic with at least one specification for a test.
US10731776 2003-12-09 2003-12-09 Method and system for computer-assisted test construction performing specification matching during test item selection Abandoned US20050125196A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US10731776 US20050125196A1 (en) 2003-12-09 2003-12-09 Method and system for computer-assisted test construction performing specification matching during test item selection

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US10731776 US20050125196A1 (en) 2003-12-09 2003-12-09 Method and system for computer-assisted test construction performing specification matching during test item selection
PCT/US2004/041449 WO2005057377A3 (en) 2003-12-09 2004-12-09 Method and system for computer-assisted test construction performing specification matching during test item selection
US11299363 US7165012B2 (en) 2003-12-09 2005-12-09 Method and system for computer-assisted test construction performing specification matching during test item selection
US11623369 US7536273B2 (en) 2003-12-09 2007-01-16 Method and system for computer-assisted test construction performing specification matching during test item selection
US12453651 US8027806B2 (en) 2003-12-09 2009-05-18 Method and system for computer-assisted test construction performing specification matching during test item selection

Publications (1)

Publication Number Publication Date
US20050125196A1 true true US20050125196A1 (en) 2005-06-09

Family

ID=34634424

Family Applications (4)

Application Number Title Priority Date Filing Date
US10731776 Abandoned US20050125196A1 (en) 2003-12-09 2003-12-09 Method and system for computer-assisted test construction performing specification matching during test item selection
US11299363 Active US7165012B2 (en) 2003-12-09 2005-12-09 Method and system for computer-assisted test construction performing specification matching during test item selection
US11623369 Active 2024-04-18 US7536273B2 (en) 2003-12-09 2007-01-16 Method and system for computer-assisted test construction performing specification matching during test item selection
US12453651 Active 2024-05-15 US8027806B2 (en) 2003-12-09 2009-05-18 Method and system for computer-assisted test construction performing specification matching during test item selection

Family Applications After (3)

Application Number Title Priority Date Filing Date
US11299363 Active US7165012B2 (en) 2003-12-09 2005-12-09 Method and system for computer-assisted test construction performing specification matching during test item selection
US11623369 Active 2024-04-18 US7536273B2 (en) 2003-12-09 2007-01-16 Method and system for computer-assisted test construction performing specification matching during test item selection
US12453651 Active 2024-05-15 US8027806B2 (en) 2003-12-09 2009-05-18 Method and system for computer-assisted test construction performing specification matching during test item selection

Country Status (2)

Country Link
US (4) US20050125196A1 (en)
WO (1) WO2005057377A3 (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060161371A1 (en) * 2003-12-09 2006-07-20 Educational Testing Service Method and system for computer-assisted test construction performing specification matching during test item selection
US20060228686A1 (en) * 2005-04-08 2006-10-12 Act, Inc. Method and system for scripted testing
US20090164406A1 (en) * 2007-08-07 2009-06-25 Brian Benson Item banking system for standards-based assessment
US20110200979A1 (en) * 2007-09-04 2011-08-18 Brian Benson Online instructional dialogs
US20130224702A1 (en) * 2012-02-24 2013-08-29 National Assoc. Of Boards Of Pharmacy Test Pallet Assembly
US8529270B2 (en) 2003-12-12 2013-09-10 Assessment Technology, Inc. Interactive computer system for instructor-student teaching and assessment of preschool children
US20160147635A1 (en) * 2014-11-26 2016-05-26 Winfried Schwarzmann Unit Test Infrastructure Generator
US20170004723A1 (en) * 2015-06-30 2017-01-05 Act, Inc. Identifying evidence of justification and explanation skills in computer automated scoring

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8128414B1 (en) 2002-08-20 2012-03-06 Ctb/Mcgraw-Hill System and method for the development of instructional and testing materials
US20040253569A1 (en) * 2003-04-10 2004-12-16 Paul Deane Automated test item generation system and method
US8170466B2 (en) * 2005-05-27 2012-05-01 Ctb/Mcgraw-Hill System and method for automated assessment of constrained constructed responses
US20070031801A1 (en) * 2005-06-16 2007-02-08 Ctb Mcgraw Hill Patterned response system and method
US20070048723A1 (en) * 2005-08-19 2007-03-01 Caveon, Llc Securely administering computerized tests over a network
US7513775B2 (en) * 2005-10-05 2009-04-07 Exam Innovations, Inc. Presenting answer options to multiple-choice questions during administration of a computerized test
US20080187893A1 (en) * 2007-02-02 2008-08-07 Network For Instructional Tv, Inc. Determining developmental progress for preschool children
US8376755B2 (en) * 2008-05-09 2013-02-19 Location Inc. Group Corporation System for the normalization of school performance statistics
US20110045452A1 (en) * 2009-08-24 2011-02-24 Bejar Isaac I Computer-Implemented Systems and Methods for Generating an Adaptive Test

Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US2261823A (en) * 1940-04-15 1941-11-04 Ballard Robley Dunglison Roller skate wheel or caster
US5261823A (en) * 1991-01-16 1993-11-16 Brother Kogyo Kabushiki Kaisha Electronic learning machine which is capable of giving learning problems matching the student's scholastic ability
US5810605A (en) * 1994-03-24 1998-09-22 Ncr Corporation Computerized repositories applied to education
US5829983A (en) * 1994-09-02 1998-11-03 Fujitsu Limited System for carrying out educational management
US5987443A (en) * 1998-12-22 1999-11-16 Ac Properties B. V. System, method and article of manufacture for a goal based educational system
US6000945A (en) * 1998-02-09 1999-12-14 Educational Testing Service System and method for computer based test assembly
US6018617A (en) * 1997-07-31 2000-01-25 Advantage Learning Systems, Inc. Test generating and formatting system
US6259890B1 (en) * 1997-03-27 2001-07-10 Educational Testing Service System and method for computer based test creation
US6381604B1 (en) * 1999-07-30 2002-04-30 Cisco Technology, Inc. Test information management system
US20020160348A1 (en) * 2000-03-08 2002-10-31 National Science Council Of Republic Of China Method for automation of dynamic test item collection and assessment
US20020182579A1 (en) * 1997-03-27 2002-12-05 Driscoll Gary F. System and method for computer based creation of tests formatted to facilitate computer based testing
US20030138765A1 (en) * 2001-11-13 2003-07-24 Prometric, Inc. Method and system for computer based testing using an amalgamated resource file
US20030157469A1 (en) * 2002-02-15 2003-08-21 Susan Embretson Computer program for generating educational and psychological test items
US20030182289A1 (en) * 1999-02-11 2003-09-25 Anderson John C. Internet test-making method
US6648651B1 (en) * 1998-09-24 2003-11-18 Lewis Cadman Consulting Pty Ltd. Apparatus for conducting a test
US6704741B1 (en) * 2000-11-02 2004-03-09 The Psychological Corporation Test item creation and manipulation system and method
US6705870B2 (en) * 2000-09-08 2004-03-16 Margaret B. Penno Psychometric assessment testing method
US20040115597A1 (en) * 2002-12-11 2004-06-17 Butt Thomas Giles System and method of interactive learning using adaptive notes
US20040219504A1 (en) * 2003-05-02 2004-11-04 Auckland Uniservices Limited System, method and computer program for student assessment
US6847962B1 (en) * 1999-05-20 2005-01-25 International Business Machines Corporation Analyzing, optimizing and rewriting queries using matching and compensation between query and automatic summary tables

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5565316A (en) 1992-10-09 1996-10-15 Educational Testing Service System and method for computer based testing
JP3693691B2 (en) 1993-12-30 2005-09-07 株式会社リコー Image processing apparatus
US7845950B2 (en) * 1997-03-27 2010-12-07 Educational Testing Service System and method for computer based creation of tests formatted to facilitate computer based testing
WO2002009391A3 (en) 2000-07-10 2002-10-10 Educational Testing Service System and methods for computer-based testing using network-based synchronization of information
US20050125196A1 (en) * 2003-12-09 2005-06-09 Len Swanson Method and system for computer-assisted test construction performing specification matching during test item selection

Patent Citations (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US2261823A (en) * 1940-04-15 1941-11-04 Ballard Robley Dunglison Roller skate wheel or caster
US5261823A (en) * 1991-01-16 1993-11-16 Brother Kogyo Kabushiki Kaisha Electronic learning machine which is capable of giving learning problems matching the student's scholastic ability
US5810605A (en) * 1994-03-24 1998-09-22 Ncr Corporation Computerized repositories applied to education
US5829983A (en) * 1994-09-02 1998-11-03 Fujitsu Limited System for carrying out educational management
US20020182579A1 (en) * 1997-03-27 2002-12-05 Driscoll Gary F. System and method for computer based creation of tests formatted to facilitate computer based testing
US6259890B1 (en) * 1997-03-27 2001-07-10 Educational Testing Service System and method for computer based test creation
US6442370B1 (en) * 1997-03-27 2002-08-27 Educational Testing Service System and method for computer based test creation
US6018617A (en) * 1997-07-31 2000-01-25 Advantage Learning Systems, Inc. Test generating and formatting system
US6000945A (en) * 1998-02-09 1999-12-14 Educational Testing Service System and method for computer based test assembly
US6648651B1 (en) * 1998-09-24 2003-11-18 Lewis Cadman Consulting Pty Ltd. Apparatus for conducting a test
US5987443A (en) * 1998-12-22 1999-11-16 Ac Properties B. V. System, method and article of manufacture for a goal based educational system
US20030182289A1 (en) * 1999-02-11 2003-09-25 Anderson John C. Internet test-making method
US6847962B1 (en) * 1999-05-20 2005-01-25 International Business Machines Corporation Analyzing, optimizing and rewriting queries using matching and compensation between query and automatic summary tables
US6381604B1 (en) * 1999-07-30 2002-04-30 Cisco Technology, Inc. Test information management system
US20020160348A1 (en) * 2000-03-08 2002-10-31 National Science Council Of Republic Of China Method for automation of dynamic test item collection and assessment
US6705870B2 (en) * 2000-09-08 2004-03-16 Margaret B. Penno Psychometric assessment testing method
US6704741B1 (en) * 2000-11-02 2004-03-09 The Psychological Corporation Test item creation and manipulation system and method
US20030138765A1 (en) * 2001-11-13 2003-07-24 Prometric, Inc. Method and system for computer based testing using an amalgamated resource file
US20030157469A1 (en) * 2002-02-15 2003-08-21 Susan Embretson Computer program for generating educational and psychological test items
US20040115597A1 (en) * 2002-12-11 2004-06-17 Butt Thomas Giles System and method of interactive learning using adaptive notes
US20040219504A1 (en) * 2003-05-02 2004-11-04 Auckland Uniservices Limited System, method and computer program for student assessment

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8027806B2 (en) 2003-12-09 2011-09-27 Educational Testing Service Method and system for computer-assisted test construction performing specification matching during test item selection
US20060161371A1 (en) * 2003-12-09 2006-07-20 Educational Testing Service Method and system for computer-assisted test construction performing specification matching during test item selection
US7165012B2 (en) 2003-12-09 2007-01-16 Educational Testing Service Method and system for computer-assisted test construction performing specification matching during test item selection
US20090233262A1 (en) * 2003-12-09 2009-09-17 Educational Testing Service Method and system for computer-assisted test construction performing specification matching during test item selection
US8529270B2 (en) 2003-12-12 2013-09-10 Assessment Technology, Inc. Interactive computer system for instructor-student teaching and assessment of preschool children
US8784114B2 (en) 2003-12-12 2014-07-22 Assessment Technology, Inc. Interactive computer system for instructor-student teaching and assessment of preschool children
US8834173B2 (en) * 2005-04-08 2014-09-16 Act, Inc. Method and system for scripted testing
US20060228686A1 (en) * 2005-04-08 2006-10-12 Act, Inc. Method and system for scripted testing
US20090164406A1 (en) * 2007-08-07 2009-06-25 Brian Benson Item banking system for standards-based assessment
US8630577B2 (en) * 2007-08-07 2014-01-14 Assessment Technology Incorporated Item banking system for standards-based assessment
US20110200979A1 (en) * 2007-09-04 2011-08-18 Brian Benson Online instructional dialogs
US20130224702A1 (en) * 2012-02-24 2013-08-29 National Assoc. Of Boards Of Pharmacy Test Pallet Assembly
US20160147635A1 (en) * 2014-11-26 2016-05-26 Winfried Schwarzmann Unit Test Infrastructure Generator
US9626276B2 (en) * 2014-11-26 2017-04-18 Sap Se Generating a test version of a method to be called during runtime and fulfilling a collaboration contract
US20170004723A1 (en) * 2015-06-30 2017-01-05 Act, Inc. Identifying evidence of justification and explanation skills in computer automated scoring

Also Published As

Publication number Publication date Type
WO2005057377A2 (en) 2005-06-23 application
US20080071498A1 (en) 2008-03-20 application
US8027806B2 (en) 2011-09-27 grant
US7165012B2 (en) 2007-01-16 grant
US20060161371A1 (en) 2006-07-20 application
WO2005057377A3 (en) 2005-08-18 application
US20090233262A1 (en) 2009-09-17 application
US7536273B2 (en) 2009-05-19 grant

Similar Documents

Publication Publication Date Title
Boyle et al. A structural model of the dimensions of teacher stress
Shaneyfelt et al. Instruments for evaluating education in evidence-based practice: a systematic review
Ewell et al. Indicators of" good practice" in undergraduate education: A handbook for development and implementation.
Smither et al. Applicant reactions to selection procedures
Johnson Grade inflation: A crisis in college education
Holmboe et al. Methods for evaluating the clinical competence of residents in internal medicine: a review
Chan et al. Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions.
Bejar et al. A feasibility study of on‐the‐fly item generation in adaptive testing
Douglas Understanding language testing
Finn et al. Assessing the social validity of behavioral interventions: A review of treatment acceptability measures.
Mayer et al. Three facets of visual and verbal learners: Cognitive ability, cognitive style, and learning preference.
Fink et al. How to assess and interpret survey psychometrics
Boone et al. Rasch analysis in the human sciences
Lohr Assessing health status and quality-of-life instruments: attributes and review criteria
Suhre et al. Impact of degree program satisfaction on the persistence of college students
Altun et al. Undergraduate students' academic achievement, field dependent/independent cognitive styles and attitude toward computers
Schoen et al. An investigation of the self-efficacy of male and female academics
Gist et al. The influence of mathematical skills and other factors on minority student performance in principles of accounting
Kurz et al. An exploration of the relationship among teacher efficacy, collective teacher efficacy, and goal consensus
Agarwal et al. The evolving relationship between general and specific computer self-efficacy—An empirical assessment
Boulet et al. Setting defensible performance standards on OSCEs and standardized patient examinations
Robertson et al. Simulation-based crisis team training for multidisciplinary obstetric providers
Harris et al. How teacher evaluation methods matter for accountability: A comparative analysis of teacher effectiveness ratings by principals and teacher value-added measures
Kember et al. Establishing the validity and reliability of course evaluation questionnaires
Foster et al. Validity issues in clinical assessment.

Legal Events

Date Code Title Description
AS Assignment

Owner name: EDUCATIONAL TESTING SERVICE, NEW JERSEY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SWANSON, LEN;REEL/FRAME:015143/0076

Effective date: 20031215