CN108985379B - Method and device for evaluating performance of classifier and computer readable storage medium - Google Patents

Method and device for evaluating performance of classifier and computer readable storage medium Download PDF

Info

Publication number
CN108985379B
CN108985379B CN201810823251.8A CN201810823251A CN108985379B CN 108985379 B CN108985379 B CN 108985379B CN 201810823251 A CN201810823251 A CN 201810823251A CN 108985379 B CN108985379 B CN 108985379B
Authority
CN
China
Prior art keywords
data
classifier
category
weight
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810823251.8A
Other languages
Chinese (zh)
Other versions
CN108985379A (en
Inventor
孙胜方
刘丹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN201810823251.8A priority Critical patent/CN108985379B/en
Publication of CN108985379A publication Critical patent/CN108985379A/en
Application granted granted Critical
Publication of CN108985379B publication Critical patent/CN108985379B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G06F18/2193—Validation; Performance evaluation; Active pattern learning techniques based on specific statistical tests
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Abstract

The disclosure relates to a method and a device for evaluating the performance of a classifier and a computer-readable storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: determining the weight of each data category according to the number of data contained in each data category in the training data set, wherein the larger the number of data contained in each data category is, the larger the corresponding weight is; training the classifier by using a training data set; testing the trained classifier through a test data set to obtain the classification accuracy of the classifier on each data category; and carrying out weighted summation on the classification accuracy of each data category by using the weight of each data category so as to determine the classification performance of the classifier. The technical scheme of the disclosure can improve the accuracy of performance evaluation of the classifier.

Description

Method and device for evaluating performance of classifier and computer readable storage medium
Technical Field
The present disclosure relates to the field of artificial intelligence technologies, and in particular, to a method and an apparatus for evaluating performance of a classifier, and a computer-readable storage medium.
Background
Artificial intelligence technology is widely used in various fields, for example, automatic artificial intelligence response systems based on natural language processing. The effectiveness of an artificial intelligence response system depends to a large extent on the classification performance of the classifier it employs. Therefore, it is important to accurately evaluate the classification performance of the classifier.
The related technologies mainly include: a hardout evaluation method for randomly dividing a data set into a training data set and a testing data set; a K-cross validation evaluation method which divides the data set into a plurality of subsets to carry out a plurality of times of training and testing, and the like.
Disclosure of Invention
The inventors of the present disclosure found that the following problems exist in the above-described related art: the method is only suitable for performance evaluation of a large training set, and the accuracy of the performance evaluation is poor under the condition that the training set is small.
In view of this, the present disclosure provides a technical solution of a performance evaluation method for a classifier, which can improve accuracy of performance evaluation on the classifier.
According to some embodiments of the present disclosure, there is provided a performance evaluation method of a classifier, including: determining the weight of each data category according to the number of data included in each data category in a training data set, wherein the larger the number of data included in a data category is, the larger the weight of the data category is; training a classifier using the training data set; testing the trained classifier through a test data set to obtain the classification accuracy of the classifier on each data category; and carrying out weighted summation on the classification accuracy of each data category by using the weight of each data category so as to evaluate the classification performance of the classifier.
In some embodiments, the weight of each data category is determined according to a difference between the number of data included in each data category in the training data set and the number of data included in the training data set.
In some embodiments, the weight w of the data class ii＝NiN, i is a positive integer, NiFor the data class i, the number of data is included, and for the training data set, N, the number of data is included.
In some embodiments, the classifier's classification accuracy R for a data class ii＝mi/MiI is a positive integer, miNumber of data for which the classifier achieved correct classification of data class i under test, MiAnd the data category i in the test data set contains the number of data.
In some embodiments, the number of data classes in the test data set is the same as the number of data classes in the training data set; the data types in the test data set contain the same amount of data.
According to other embodiments of the present disclosure, there is provided a performance evaluation apparatus of a classifier including: a weight determining unit, configured to determine a weight of each data category according to a number of data included in each data category in a training data set, where the larger the number of data included in the data category is, the larger the weight of the data category is; a training unit for training a classifier using the training data set; the testing unit is used for testing the trained classifier through a testing data set so as to obtain the classification accuracy of the classifier on each data category; and the performance evaluation unit is used for carrying out weighted summation on the classification accuracy of each data category by using the weight of each data category so as to evaluate the classification performance of the classifier.
In some embodiments, the weight determining unit determines the weight of each data category according to a difference between the number of data included in each data category in the training data set and the number of data included in the training data set.
In some embodiments, the weight determination unit determines the weight w of the data class ii＝NiN, i is a positive integer, NiFor the data class i, the number of data is included, and for the training data set, N, the number of data is included.
In some embodiments, the test unit determines a classification accuracy R of the classifier for a data class ii＝mi/MiI is a positive integer, miNumber of data for which the classifier achieved correct classification of data class i under test, MiAnd the data category i in the test data set contains the number of data.
In some embodiments, the number of data classes in the test data set is the same as the number of data classes in the training data set; the data types in the test data set contain the same amount of data.
According to still further embodiments of the present disclosure, there is provided a performance evaluation apparatus of a classifier, including: a memory; and a processor coupled to the memory, the processor configured to perform one or more steps of the method for performance evaluation of a classifier in any of the above embodiments based on instructions stored in the memory device.
According to still further embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements one or more steps of the performance evaluation method of the classifier in any of the above embodiments.
In the above embodiment, the weight is determined according to the number of data included in each data category in the training data set, and the test result of each data category is weighted and summed according to the weight to evaluate the classification performance of the classifier. Therefore, the influence of the classification accuracy of the data categories on the performance evaluation can be adjusted according to the weight and the importance degree of the data categories, the performance of the classifier can be accurately evaluated under the condition of a small training set, and the accuracy of the performance evaluation is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
The present disclosure may be more clearly understood from the following detailed description, taken with reference to the accompanying drawings, in which:
FIG. 1 illustrates a flow diagram of some embodiments of a performance evaluation method of a classifier of the present disclosure;
FIG. 2 illustrates a block diagram of some embodiments of a performance evaluation apparatus of a classifier of the present disclosure;
FIG. 3 shows a block diagram of further embodiments of a performance evaluation apparatus of a classifier of the present disclosure;
fig. 4 shows a block diagram of further embodiments of a performance evaluation apparatus of a classifier of the present disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
As mentioned above, the classification accuracy of each data class in the related art has the same effect on the classifier evaluation. Therefore, in the case of a smaller training set, the classification result of the unimportant data category (e.g., the data category containing a smaller amount of data) also greatly affects the performance evaluation result of the classifier, thereby causing inaccurate performance evaluation.
In order to solve the problems, the performance of the classifier is evaluated in a mode of setting weight for classification accuracy. For example, the following embodiments may be employed.
Fig. 1 illustrates a flow diagram of some embodiments of a performance evaluation method of a classifier of the present disclosure.
As shown in fig. 1, the method includes: step 110, determining the weight of each data category; step 120, training the classifier; step 130, testing the classifier; and step 140, evaluating the classification performance of the classifier.
In step 110, the weight of each data category is determined based on the number of data included in each data category in the training data set. The greater the amount of data contained in a data category, the greater the corresponding weight. For example, the weight of each data class is determined based on the difference between the amount of data included in each data class in the training data set and the amount of data included in the training data set. The weights may take a variety of forms that reflect the difference between the quantities, such as differences, ratios, and the like.
In some embodiments of the present invention, the,n data in the training data set comprise I data categories, and the data category I comprises NiAnd (4) data. In this case, the weight w of the data class i may be determinedi＝Ni/N。
Therefore, the classification result of the data category occupying a larger part in the training data set can be ensured to have larger influence on the performance evaluation of the classifier, and the performance evaluation accuracy is improved.
In step 120, the classifier is trained using a training data set. For example, the classifier can be an LR (Logistic Regression) classifier, a bayesian classifier, or the like.
In step 130, the trained classifier is tested through the test data set to obtain the classification accuracy of the classifier for each data class.
In some embodiments, the test data set includes I data classes as the training data set, data class I containing MiAnd (4) data. For example, each data class in the test data set includes MiData, at which time a common I M in the data set is testediThe data can be classified accurately by the classifier under the unified standard, and the accuracy of performance evaluation can be improved by combining the weight.
In some embodiments, the classifier is tested using a test data set, and the classifier correctly classifies the data class i by the number of m data classesi. Classification accuracy R of classifier for data class ii＝mi/Mi。
In step 140, the classification accuracy of each data class is weighted and summed by using the weight of each data class to evaluate the classification performance of the classifier.
In some embodiments, performance parameters of the classifier may be set
Figure BDA0001741843920000051
The larger the H, the better the classifier.
In the above embodiment, the weight is determined according to the number of data included in each data category in the training data set, and the test result of each data category is weighted and summed according to the weight to evaluate the classification performance of the classifier. Therefore, the influence of the classification accuracy of the data categories on the performance evaluation can be adjusted according to the weight and the importance degree of the data categories, the performance of the classifier can be accurately evaluated under the condition of a small training set, and the accuracy of the performance evaluation is improved.
Fig. 2 illustrates a block diagram of some embodiments of a performance evaluation apparatus of a classifier of the present disclosure.
As shown in fig. 2, the performance evaluation apparatus 2 includes a weight determination unit 21, a training unit 22, a test unit 23, and a performance evaluation unit 24.
The weight determination unit 21 determines the weight of each data type based on the number of data included in each data type in the training data set. The greater the amount of data contained in a data category, the greater the corresponding weight.
In some embodiments, the weight determining unit 21 determines the weight of each data category according to the difference between the number of data included in each data category in the training data set and the number of data included in the training data set. For example, the weight W of the data class ii＝Ni/N。
The training unit 22 trains the classifier with a training data set.
The testing unit 23 tests the trained classifier through the test data set to obtain the classification accuracy of the classifier for each data category. For example, each data category in the test data set contains the same amount of data. For example, the test unit 23 determines the classification accuracy R of the classifier for the data class ii＝mi/Mi。
The performance evaluation unit 24 performs weighted summation on the classification accuracy of each data class by using the weight of each data class to evaluate the classification performance of the classifier.
In the above embodiment, the weight is determined according to the number of data included in each data category in the training data set, and the test result of each data category is weighted and summed according to the weight to evaluate the classification performance of the classifier. Therefore, the influence of the classification accuracy of the data categories on the performance evaluation can be adjusted according to the weight and the importance degree of the data categories, the performance of the classifier can be accurately evaluated under the condition of a small training set, and the accuracy of the performance evaluation is improved.
Fig. 3 shows a block diagram of further embodiments of a performance evaluation apparatus of a classifier of the present disclosure.
As shown in fig. 3, the performance evaluation device 3 of the classifier of this embodiment includes: a memory 31 and a processor 32 coupled to the memory 31, the processor 32 being configured to perform one or more steps of a method of evaluating the performance of a classifier in any one of the embodiments of the present disclosure based on instructions stored in the memory 31.
The memory 31 may include, for example, a system memory, a fixed nonvolatile storage medium, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader (Boot Loader), a database, and other programs.
Fig. 4 shows a block diagram of further embodiments of a performance evaluation apparatus of a classifier of the present disclosure.
As shown in fig. 4, the performance evaluation device 4 of the classifier of this embodiment includes: a memory 410 and a processor 420 coupled to the memory 410, the processor 420 configured to execute a method of performance evaluation of a classifier according to any of the foregoing embodiments based on instructions stored in the memory 410.
The memory 410 may include, for example, system memory, fixed non-volatile storage media, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader (Boot Loader), and other programs.
The performance evaluation device 4 of the classifier may further include an input-output interface 430, a network interface 440, a storage interface 450, and the like. These interfaces 430, 440, 450 and the connection between the memory 410 and the processor 420 may be, for example, via a bus 460. The input/output interface 430 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 440 provides a connection interface for various networking devices. The storage interface 440 provides a connection interface for external storage devices such as an SD card and a usb disk.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
So far, the performance evaluation method of the classifier, the apparatus of the classifier, and the computer-readable storage medium according to the present disclosure have been described in detail. Some details that are well known in the art have not been described in order to avoid obscuring the concepts of the present disclosure. It will be fully apparent to those skilled in the art from the foregoing description how to practice the presently disclosed embodiments.
The method and system of the present disclosure may be implemented in a number of ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present disclosure may also be embodied as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the foregoing examples are for purposes of illustration only and are not intended to limit the scope of the present disclosure. It will be appreciated by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims (10)

1. A performance evaluation method of a classifier includes:
determining the weight of each data category according to the quantity of data contained in each data category in a training data set, wherein the larger the quantity of data contained in each data category is, the larger the weight of the data category is, the data in the training data set is natural language data, and the data category is semantic classification in an automatic artificial intelligence response system;
training a classifier using the training data set;
testing the trained classifier through a test data set to obtain the classification accuracy of the classifier on each data category, wherein the data in the test data set are natural language data, and the classifier is used for the automatic artificial intelligence response system to perform semantic classification on the natural language data;
carrying out weighted summation on the classification accuracy of each data category by using the weight of each data category so as to evaluate the classification performance of the classifier,
the number of data classes in the test data set is the same as the number of data classes in the training data set;
the data types in the test data set contain the same amount of data.
2. The performance evaluation method of claim 1, wherein the determining the weight of each data category comprises:
and determining the weight of each data category according to the difference between the number of the data included in each data category in the training data set and the number of the data included in the training data set.
3. The performance evaluation method of claim 2, wherein the determining the weight of each data category comprises:
weight w of data class ii＝NiN, i is a positive integer, NiContaining the number of data for data class i and N for the training data setThe number of the cells.
4. The performance evaluation method of claim 1, wherein the obtaining the classification accuracy of the classifier for the data classes comprises:
the classification accuracy R of the classifier on the data class ii＝mi/MiI is a positive integer, miNumber of data for which the classifier achieved correct classification of data class i under test, MiAnd the data category i in the test data set contains the number of data.
5. A performance evaluation apparatus of a classifier, comprising:
the weight determining unit is used for determining the weight of each data category according to the number of data included in each data category in a training data set, the larger the number of data included in the data category is, the larger the weight of the data category is, the data in the training data set is natural language data, and the data category is semantic classification in an automatic artificial intelligence response system;
a training unit for training a classifier using the training data set;
the testing unit is used for testing the trained classifier through a testing data set so as to obtain the classification accuracy of the classifier on each data category, the data in the testing data set is natural language data, and the classifier is used for performing semantic classification on the natural language data by the automatic artificial intelligence response system;
a performance evaluation unit for performing weighted summation on the classification accuracy of each data class by using the weight of each data class to evaluate the classification performance of the classifier,
the number of data classes in the test data set is the same as the number of data classes in the training data set;
the data types in the test data set contain the same amount of data.
6. The performance evaluation apparatus according to claim 5,
the weight determining unit determines the weight of each data type according to the difference between the number of data included in each data type in the training data set and the number of data included in the training data set.
7. The performance evaluation apparatus according to claim 5,
the weight determination unit determines the weight w of the data class ii＝NiN, i is a positive integer, NiFor the data class i, the number of data is included, and for the training data set, N, the number of data is included.
8. The performance evaluation apparatus according to claim 5,
the test unit determines the classification accuracy R of the classifier for a data class ii＝mi/MiI is a positive integer, miNumber of data for which the classifier achieved correct classification of data class i under test, MiAnd the data category i in the test data set contains the number of data.
9. A performance evaluation apparatus of a classifier, comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the method of performance evaluation of a classifier of any of claims 1-4 based on instructions stored in the memory device.
10. A computer-readable storage medium on which a computer program is stored which, when executed by a processor, implements the performance evaluation method of the classifier of any one of claims 1-4.
CN201810823251.8A 2018-07-25 2018-07-25 Method and device for evaluating performance of classifier and computer readable storage medium Active CN108985379B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810823251.8A CN108985379B (en) 2018-07-25 2018-07-25 Method and device for evaluating performance of classifier and computer readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810823251.8A CN108985379B (en) 2018-07-25 2018-07-25 Method and device for evaluating performance of classifier and computer readable storage medium

Publications (2)

Publication Number Publication Date
CN108985379A CN108985379A (en) 2018-12-11
CN108985379B true CN108985379B (en) 2021-04-30

Family

ID=64550843

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810823251.8A Active CN108985379B (en) 2018-07-25 2018-07-25 Method and device for evaluating performance of classifier and computer readable storage medium

Country Status (1)

Country Link
CN (1) CN108985379B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110210558B (en) * 2019-05-31 2021-10-26 北京市商汤科技开发有限公司 Method and device for evaluating performance of neural network
CN111126487A (en) * 2019-12-24 2020-05-08 北京安兔兔科技有限公司 Equipment performance testing method and device and electronic equipment

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8972307B1 (en) * 2011-09-15 2015-03-03 Google Inc. Method and apparatus for machine learning
CN106203485A (en) * 2016-07-01 2016-12-07 北京邮电大学 A kind of parallel training method and device of support vector machine
CN106202044A (en) * 2016-07-07 2016-12-07 武汉理工大学 A kind of entity relation extraction method based on deep neural network
CN106897940B (en) * 2017-01-03 2021-04-13 内蒙古电力(集团)有限责任公司 Wind power prediction index evaluation method and system for power grid peak regulation capacity limit value

Also Published As

Publication number Publication date
CN108985379A (en) 2018-12-11

Similar Documents

Publication Publication Date Title
US11899800B2 (en) Open source vulnerability prediction with machine learning ensemble
CN106796585B (en) Conditional validation rules
US9218572B2 (en) Technique for classifying data
Choi et al. End-to-end prediction of buffer overruns from raw source code via neural memory networks
CN108596410B (en) Automatic wind control event processing method and device
JP2019021303A (en) Software program fault position specification
Zheng et al. Learning criteria weights of an optimistic Electre Tri sorting rule
US10437587B2 (en) Software package analyzer for increasing parallelization of code editing
CN110472802B (en) Data characteristic evaluation method, device and equipment
CN106708729B (en) The prediction technique and device of aacode defect
US10754744B2 (en) Method of estimating program speed-up in highly parallel architectures using static analysis
CN111160959B (en) User click conversion prediction method and device
CN108985379B (en) Method and device for evaluating performance of classifier and computer readable storage medium
US20190026108A1 (en) Recommendations based on the impact of code changes
CN114556360A (en) Generating training data for machine learning models
CN104504334B (en) System and method for assessing classifying rules selectivity
EP4189547A1 (en) Machine-learning based software testing technique
US10719482B2 (en) Data comparison
US20200250560A1 (en) Determining pattern similarities using a multi-level machine learning system
US20130013244A1 (en) Pattern based test prioritization using weight factors
US20200174760A1 (en) Automatic code generation
CN110544166A (en) Sample generation method, device and storage medium
CN111382052A (en) Code quality evaluation method and device and electronic equipment
US11797775B1 (en) Determining emebedding vectors for an unmapped content item using embedding inferenece
US11138099B2 (en) Method for testing software, and computing device and computer-readable storage medium thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant