CN113505730A - Model evaluation method, device, equipment and storage medium based on mass data - Google Patents
Model evaluation method, device, equipment and storage medium based on mass data Download PDFInfo
- Publication number
- CN113505730A CN113505730A CN202110842391.1A CN202110842391A CN113505730A CN 113505730 A CN113505730 A CN 113505730A CN 202110842391 A CN202110842391 A CN 202110842391A CN 113505730 A CN113505730 A CN 113505730A
- Authority
- CN
- China
- Prior art keywords
- macro
- sample
- picture
- precision
- recall
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011156 evaluation Methods 0.000 title claims abstract description 64
- 238000012360 testing method Methods 0.000 claims abstract description 52
- 238000012549 training Methods 0.000 claims abstract description 44
- 239000011159 matrix material Substances 0.000 claims abstract description 42
- 238000000034 method Methods 0.000 claims abstract description 18
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 15
- 238000012795 verification Methods 0.000 claims abstract description 12
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000009795 derivation Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 208000029152 Small face Diseases 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003997 social interaction Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a model evaluation method, a device, equipment and a storage medium based on mass data, wherein the method comprises the steps of obtaining mass picture information from a cache, and sorting the mass picture information into a sample data set; dividing the sample data set into a training set and a test set in a uniformly distributed mode; performing model training on the training set, performing random n-time verification on the test set, and finally returning n-time test results; calculating precision ratio and recall ratio of each confusion matrix according to the n times of test results; macro precision, macro recall, and macro F1 are calculated based on the precision and recall of each of the confusion matrices. According to the method, the trained data clustering model is evaluated according to the macro precision, the macro recall ratio and the macro F1, so that the clustering algorithm model with the most balanced fitting degree is selected according to the evaluation result, and the accuracy rate of the face after clustering is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a model evaluation method, a model evaluation device, model evaluation equipment and a storage medium based on mass data.
Background
In order to adapt to the new trend of the digital technology fully integrating social interaction and daily life, the construction of the intelligent community has increasingly become a new innovation power for improving the livelihood of the country. The face recognition technology is widely applied in the fields of identity authentication, public security control and the like. The face recognition under the community scene has the characteristics of large data volume and more similar repeated faces, and if the face is not classified and clustered, the calculation amount of the face recognition and the face comparison is exponentially increased.
The face clustering is a solution based on machine learning, and the purpose of classifying similar faces is achieved by taking face data as a training data set and a verification data set and using a machine learning algorithm. The final effect of reducing the comparison times is achieved. Although the current face recognition and face clustering algorithms tend to be mature, the clustering algorithm model obtained after training needs to be effectively verified, otherwise, a training error (training error) or an empirical error (empirical error) is caused. In the prior art, a method for effectively evaluating a clustering algorithm model obtained after training is not available, so that the selection and performance evaluation of the model are inaccurate, and the accuracy of the face after clustering is directly influenced.
Disclosure of Invention
The invention aims to provide a model evaluation method, a device, equipment and a storage medium based on mass data aiming at the defects that the selection and performance evaluation of a clustering algorithm model in the prior art are inaccurate and the accuracy rate of face clustering is influenced, so that the accuracy rate of face clustering is improved.
In order to achieve the above object, the present invention provides a model evaluation method based on mass data, which comprises the following steps:
acquiring mass picture information from a cache, and sorting the mass picture information into a sample data set;
dividing the sample data set into a training set and a test set in a uniformly distributed mode;
training a mass data clustering model according to the training set, randomly verifying the trained data clustering model for n times according to the test set, and finally returning n times of test results, wherein n is an integer greater than or equal to 1;
calculating precision ratio and recall ratio of each confusion matrix according to the n times of test results;
calculating macro precision, macro recall and macro F1 according to the precision and recall of each confusion matrix;
and evaluating the trained data clustering model according to the macro precision ratio, the macro recall ratio and the macro F1.
Preferably, before acquiring the massive picture information from the cache and sorting the massive picture information into the sample data set, the method further includes:
acquiring a target community face snapshot picture stream;
carrying out structural processing on the face snapshot picture stream to obtain structural picture data;
cleaning an invalid picture without a human face recognized in the structured picture data to obtain a cleaned picture;
and storing the cleaned pictures to a cache according to the target community classification.
Preferably, the storing the cleaned picture to a cache according to the target community classification specifically includes:
adjusting the cleaned picture into a uniform picture format and size to obtain an adjusted picture;
and acquiring the gender characteristics of the face in the adjusted picture, and classifying and storing the adjusted picture to a cache according to the gender characteristics and the target community.
Preferably, the obtaining of the gender feature of the face in the adjusted picture, and classifying and storing the adjusted picture to a cache according to the gender feature in the target community specifically include:
acquiring the cell number, the gender characteristics of the face and a timestamp in the adjusted picture;
generating a corresponding face identifier for each adjusted picture according to the timestamp;
and classifying the adjusted picture by taking the cell number as a first dimension, the gender characteristic of the face as a second dimension and the face identification as a third dimension, and storing the classification result to a cache.
Preferably, the automatically generating a face identifier for each of the adjusted pictures according to the timestamp specifically includes:
and generating a corresponding face identifier for each adjusted picture according to the timestamp and the random number of the preset digit.
Preferably, the verifying the trained data clustering model for n times randomly according to the test set specifically includes:
step S1, splitting the sample data set D into 10 sample subsets D1 to D10, respectively;
step S2, starting to test the sample m (m is more than or equal to 1 and less than or equal to n), randomly taking a male sample from each cell of the buffer database by the sample subset D1, and marking the sample as unavailable;
step S3, the sample subset D2 randomly takes a female sample from each cell of the buffer database and marks the sample as unavailable;
step S4, repeating steps S2 and S3 for the remaining sample subsets, and after one round of sampling is performed on 10 sample subsets, determining whether the sample data set D has available samples, and if so, starting the loop from the beginning again until all samples in the sample data set D are marked as unavailable;
step S5, taking a sample set S as a training set, wherein the sample set S is D1U D2U … … U Di (i is not more than 1 and not more than n, and i is not more than n-m +1), and training the sample set S by using a clustering algorithm to obtain a clustering algorithm model;
step S6, according to the clustering algorithm model obtained in the step S5, the sample data set D is subjected to(n-m+1)Testing to obtain a multi-class confusion matrix as a test result;
repeating the steps S2 to S6 n times to obtain n multi-class confusion matrices.
Preferably, the calculating macro precision, macro precision and macro F1 according to precision and recall of each confusion matrix specifically includes:
step S11, starting to execute the cycle of the ith (i is more than or equal to 1 and less than or equal to n), and converting the ith multi-class confusion matrix into a two-class confusion matrix;
step S12, carrying out formula derivation on the two-classification confusion matrix to obtain precision ratio and recall ratio of the two-classification confusion matrix;
step S13, the loop step is S11 and S12 to obtain precision and recall of all n confusion matrixes;
and calculating the average value of all precision ratios and recall ratios according to a preset formula to obtain the macro precision ratio, the macro recall ratio and the corresponding macro F1.
In addition, in order to achieve the above object, the present invention further provides a model evaluation device based on mass data, including:
the sorting module is used for acquiring mass picture information from the cache and sorting the mass picture information into a sample data set;
the dividing module is used for dividing the sample data set into a training set and a test set in a uniformly distributed mode;
the training verification module is used for training the mass data clustering model according to the training set, randomly verifying the trained data clustering model for n times according to the test set, and finally returning n times of test results, wherein n is an integer greater than or equal to 1;
the calculation module is used for calculating the precision ratio and the recall ratio of each confusion matrix according to the n times of test results;
the calculating module is further configured to calculate a macro precision, a macro recall and a macro F1 according to the precision and the recall of each confusion matrix;
and the evaluation module is used for evaluating the trained data clustering model according to the macro precision ratio, the macro recall ratio and the macro F1.
In addition, in order to achieve the above object, the present invention further provides a model evaluation device based on mass data, including: the system comprises a memory, a processor and a model evaluation program based on mass data, wherein the model evaluation program based on mass data is stored on the memory and can run on the processor, and when being executed by the processor, the model evaluation program based on mass data realizes the steps of the model evaluation method based on mass data.
In addition, in order to achieve the above object, the present invention further provides a storage medium, on which a model evaluation program based on mass data is stored, and the model evaluation program based on mass data realizes the steps of the model evaluation method based on mass data as described above when being executed by a processor.
In the invention, massive picture information is acquired from a cache and is arranged into a sample data set; dividing the sample data set into a training set and a test set in a uniformly distributed mode; performing model training on the training set, performing random n-time verification on the test set, and finally returning n-time test results; calculating precision ratio and recall ratio of each confusion matrix according to the n times of test results; calculating macro precision, macro recall and macro F1 according to the precision and recall of each confusion matrix; and evaluating the trained data clustering model according to the macro precision, the macro recall ratio and the macro F1, so that a clustering algorithm model with the most balanced fitting degree is selected according to the evaluation result, and the accuracy of the face after clustering is improved.
Drawings
Fig. 1 is a schematic structural diagram of a model evaluation device based on mass data in a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a first embodiment of a model evaluation method based on mass data according to the present invention;
FIG. 3 is a schematic diagram of a distributed face clustering model evaluation system in a first embodiment of the mass data-based model evaluation method of the present invention;
FIG. 4 is a schematic flow chart of performing n-time verification;
figure 5 is a schematic diagram of a three-classification matrix,
FIG. 6 is a schematic diagram of a binary matrix;
FIG. 7 is a diagram illustrating TP, FP, FN and FP;
fig. 8 is a block diagram illustrating a first embodiment of a model evaluation apparatus according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a model evaluation device based on mass data in a hardware operating environment according to an embodiment of the present invention.
As shown in fig. 1, the model evaluation device based on mass data may include: a processor 1001, such as a Central Processing Unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), and the optional user interface 1003 may further include a standard wired interface and a wireless interface, and the wired interface for the user interface 1003 may be a USB interface in the present invention. The network interface 1004 may optionally include a standard wired interface, a WIreless interface (e.g., a WIreless-FIdelity (WI-FI) interface). The Memory 1005 may be a Random Access Memory (RAM) Memory or a Non-volatile Memory (NVM), such as a disk Memory. The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration shown in fig. 1 does not constitute a limitation of a model evaluation device based on mass data, and may include more or less components than those shown, or some components in combination, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a model evaluation program based on mass data.
In the model evaluation device based on mass data shown in fig. 1, the network interface 1004 is mainly used for connecting a background server and performing data communication with the background server; the user interface 1003 is mainly used for connecting user equipment; the model evaluation device based on mass data calls a model evaluation program based on mass data stored in the memory 1005 through the processor 1001, and executes the model evaluation method based on mass data provided by the embodiment of the present invention.
Based on the hardware structure, the embodiment of the model evaluation method based on mass data is provided.
Referring to fig. 2, a first embodiment of the model evaluation method based on mass data is provided.
In a first embodiment, the method for evaluating a model based on mass data includes the following steps:
step S10: and acquiring mass picture information from the cache, and sorting the mass picture information into a sample data set.
It should be understood that the execution subject of the present embodiment is the model evaluation device based on mass data, wherein the model evaluation device based on mass data may be an electronic device such as a personal computer or a server, and the present embodiment is not limited thereto. Referring to fig. 3, fig. 3 is a schematic diagram of a distributed face clustering model evaluation system, where the distributed face clustering model evaluation system includes a snapshot device and a model evaluation device based on mass data, and a snapshot module obtains a target community face snapshot image stream through the snapshot device; a data access module in the model evaluation equipment based on mass data accesses picture data and structures the picture data; the cleaning service filters out invalid pictures without identifying human faces; the distributed storage module unifies the format and the size of the pictures for storage; the analysis service acquires the gender characteristics of the human face in the photo, and classifies and stores the gender characteristics in a cache according to the cell where the snapshot device is located; the cache module provides structured high-quality data based on the memory database.
In this embodiment, before the step S10, the method further includes: acquiring a target community face snapshot picture stream; carrying out structural processing on the face snapshot picture stream to obtain structural picture data; cleaning an invalid picture without a human face recognized in the structured picture data to obtain a cleaned picture; and storing the cleaned pictures to a cache according to the target community classification.
Further, in this embodiment, the storing the cleaned picture to a cache according to the target community classification specifically includes:
adjusting the cleaned picture into a uniform picture format and size to obtain an adjusted picture;
and acquiring the gender characteristics of the face in the adjusted picture, and classifying and storing the adjusted picture to a cache according to the gender characteristics and the target community.
Further, in this embodiment, the obtaining of the gender feature of the face in the adjusted picture, and classifying and storing the adjusted picture to a cache according to the gender feature in the target community specifically include:
acquiring the cell number, the gender characteristics of the face and a timestamp in the adjusted picture;
generating a corresponding face identifier for each adjusted picture according to the timestamp;
and classifying the adjusted picture by taking the cell number as a first dimension, the gender characteristic of the face as a second dimension and the face identification as a third dimension, and storing the classification result to a cache.
Further, in this embodiment, the automatically generating a face identifier for each of the adjusted pictures according to the timestamp specifically includes:
and generating a corresponding face identifier for each adjusted picture according to the timestamp and the random number of the preset digit.
It should be noted that, the snapshot device shoots the face snapshot picture stream of the community active personnel and the corresponding snapshot information in the target community, and the snapshot information may include: snapshot time, snapshot location, and snapshot device ID. And the data access module accesses the picture data stream from the snapshot equipment and transmits the structured picture information to the cleaning module. The cleaning module identifies effective pictures with face features based on a face identification algorithm, and the face features and the gender features are structured. The distributed storage module unifies the pictures into a png picture format, and the picture size is within 5M. And after storage, providing the relative path of the picture address to other modules.
In a specific implementation, the analysis module integrates a cell number and a face gender field of the structured data, and automatically generates a FaceId, that is, the face identifier, for each picture, and if the preset number of bits may be 5 bits, a rule is generated: the timestamp year, month, day, hour, minute, millisecond, microsecond +5 bit random number, for example 2021040520000011122257893.
The cell number is used as a key for caching, namely the picture information is classified according to the first dimension of the cell; the gender of the face is taken as a key for caching, namely the picture information is classified according to the second dimension of the gender of the face; and taking the faceId as a key of the cache, namely classifying the picture information in a faceId third dimension. Examples are 420100000009:01: 2021040520000011122257893. Where 01 denotes male, 02 denotes female, and 00 denotes unknown sex. And comparing the data set proportions of all the first dimensions and the second dimensions, and filling the cells with small face acquisition quantity in the mode of copying and rotating the existing pictures in order to ensure that the picture quantity of each cell is consistent and the male-female proportion of each cell is consistent.
And the cache module deploys Redis cluster expansion based on the memory database Redis. And the model evaluation module and the model selection module at the upper layer are used for rapidly inquiring the picture data.
Step S20: and dividing the sample data set into a training set and a testing set in a uniformly distributed mode.
Step S30: and training the mass data clustering model according to the training set, randomly verifying the trained data clustering model for n times according to the test set, and finally returning n times of test results, wherein n is an integer greater than or equal to 1.
It should be noted that, the model evaluation module takes the massive picture information from the cache and arranges the massive picture information into a sample data set, and the full-scale sample data set is divided into a training set and a test set in a uniformly distributed manner. And then carrying out model training on the training set, carrying out random n-time verification on the test set, and finally returning n-time test results.
Step S40: and calculating the precision ratio and the recall ratio of each confusion matrix according to the n times of test results.
Step S50: macro precision, macro recall, and macro F1 are calculated based on the precision and recall of each of the confusion matrices.
Step S60: and evaluating the trained data clustering model according to the macro precision ratio, the macro recall ratio and the macro F1.
In specific implementation, the model selection module acquires the test results of n times from the evaluation module, calculates precision and recall of each confusion matrix, and calculates an average value to obtain macro precision (macro-P), macro recall (macro-R) and a corresponding macro F1 (macro-F1).
In this embodiment, massive picture information is obtained from a cache, and the massive picture information is sorted into a sample data set; dividing the sample data set into a training set and a test set in a uniformly distributed mode; performing model training on the training set, performing random n-time verification on the test set, and finally returning n-time test results; calculating precision ratio and recall ratio of each confusion matrix according to the n times of test results; calculating macro precision, macro recall and macro F1 according to the precision and recall of each confusion matrix; and evaluating the trained data clustering model according to the macro precision, the macro recall ratio and the macro F1, so that a clustering algorithm model with the most balanced fitting degree is selected according to the evaluation result, and the accuracy of the face after clustering is improved.
With continued reference to fig. 2, a second embodiment of the model evaluation method based on mass data according to the present invention is provided.
In this embodiment, the randomly verifying the trained data clustering model n times according to the test set specifically includes:
step S1, splitting the sample data set D into 10 sample subsets D1 to D10, respectively;
step S2, starting to test the sample m (m is more than or equal to 1 and less than or equal to n), randomly taking a male sample from each cell of the buffer database by the sample subset D1, and marking the sample as unavailable;
step S3, the sample subset D2 randomly takes a female sample from each cell of the buffer database and marks the sample as unavailable;
step S4, repeating steps S2 and S3 for the remaining sample subsets, and after one round of sampling is performed on 10 sample subsets, determining whether the sample data set D has available samples, and if so, starting the loop from the beginning again until all samples in the sample data set D are marked as unavailable;
step S5, taking a sample set S as a training set, wherein the sample set S is D1U D2U … … U Di (i is not more than 1 and not more than n, and i is not more than n-m +1), and training the sample set S by using a clustering algorithm to obtain a clustering algorithm model;
step S6, according to the clustering algorithm model obtained in the step S5, the sample data set D is subjected to(n-m+1)Testing to obtain a multi-class confusion matrix as a test result;
repeating the steps S2 to S6 n times to obtain n multi-class confusion matrices.
It should be noted that the main function of the model evaluation module is to return the test results n times. In order to ensure the randomness of the sample distribution, the model evaluation module needs to perform n times of verification (n can be 5, 10, 20, etc. generally), and perform n times of verification as shown in fig. 4, which includes the following steps:
1. assuming that the full sample set is D, preparing to split the sample set D into 10 sample subsets, respectively D1 through D10;
2. the m (1-m-n) sample test is started. The sample subset D1 randomly takes a male sample from each cell of the buffer database and marks the sample as unavailable;
3. the sample subset D2 randomly takes a female sample from each cell of the buffer database and marks the sample as unavailable;
4. repeating the steps 2 and 3 for the rest sample subsets according to the rule, wherein after one round of sampling is performed on 10 sample subsets, if the sample set D has available samples, the cycle is started from the beginning again until all the samples in the sample set D are marked as unavailable;
5. taking a sample set S ═ D1 ═ D2 ≠ … … ═ Di (i is not less than 1 and not more than n, and i ≠ n-m +1) as a training set, and training the sample set S by using a clustering algorithm to obtain a clustering algorithm model;
6. testing the sample set D (n-m +1) according to the clustering algorithm model obtained in the step 5 to obtain a test result which is a multi-class confusion matrix;
7. repeating the steps 2 to 6 for n times to obtain n multi-class confusion matrixes.
Further, in this embodiment, the calculating a macro precision ratio, a macro recall ratio, and a macro F1 according to the precision ratio and the recall ratio of each confusion matrix specifically includes:
step S11, starting to execute the cycle of the ith (i is more than or equal to 1 and less than or equal to n), and converting the ith multi-class confusion matrix into a two-class confusion matrix;
step S12, carrying out formula derivation on the two-classification confusion matrix to obtain precision ratio and recall ratio of the two-classification confusion matrix;
step S13, the loop step is S11 and S12 to obtain precision and recall of all n confusion matrixes;
and calculating the average value of all precision ratios and recall ratios according to a preset formula to obtain the macro precision ratio, the macro recall ratio and the corresponding macro F1.
In a specific implementation, the model selection module obtains n multi-class confusion matrices from the model evaluation module, and calculates macro precision, macro recall and corresponding macro F1. The method comprises the following steps:
1. the loop of the ith (1 ≦ i ≦ n) is started, and the ith multi-class confusion matrix is converted into a two-class confusion matrix (with one of the cluster labels as a positive example). The conversion method is described by taking the example of converting the three-classification matrix into the two-classification matrix, as shown in fig. 5 and 6;
2. carrying out formula derivation on the two-classification confusion matrix obtained in the step 1 to obtain precision ratio and recall ratio of the two-classification confusion matrix, wherein the formula is as follows (TP, FP, FN and FP are shown in fig. 7);
the formula I is as follows: precision ratio Pi＝TP/(TP+FP)
The formula II is as follows: recall ratio Ri＝TP/(TP+FN)
3. Circularly obtaining the precision ratios Pi and the recall ratios Ri of all n confusion matrixes in the steps 1 and 2;
4. calculating the average value of all precision ratios and recall ratios to obtain macro precision ratio (macro-P), macro recall ratio (macro-R) and corresponding macro F1(macro-F1), wherein the preset formula is as follows:
micro-F1 is suitable for the case of multi-classification data distribution balance (is easily influenced by common classes); macro-F1 is not affected by multi-classification data distribution imbalance, but is susceptible to classes with high identifiability (rare classes); macro-F1 is more common and generally smaller values are better. A fixed recall is generally specified, preferably the model with the highest precision.
In this embodiment, the model evaluation module performs n-time verification, thereby ensuring randomness of sample distribution and improving accuracy of model evaluation.
In addition, an embodiment of the present invention further provides a storage medium, where a model evaluation program based on mass data is stored on the storage medium, and the model evaluation program based on mass data implements the steps of the model evaluation method based on mass data as described above when being executed by a processor.
In addition, referring to fig. 8, an embodiment of the present invention further provides a model evaluation device based on mass data, where the model evaluation device based on mass data includes:
the sorting module 10 is configured to acquire a large amount of picture information from the cache, and sort the large amount of picture information into a sample data set;
a dividing module 20, configured to divide the sample data set into a training set and a test set in a uniformly distributed manner;
the training verification module 30 is used for training the mass data clustering model according to the training set, randomly verifying the trained data clustering model for n times according to the test set, and finally returning n times of test results, wherein n is an integer greater than or equal to 1;
the calculation module 40 is used for calculating the precision ratio and the recall ratio of each confusion matrix according to the n times of test results;
the calculating module 40 is further configured to calculate a macro precision, a macro recall and a macro F1 according to the precision and the recall of each confusion matrix;
and the evaluation module 50 is used for evaluating the trained data clustering model according to the macro precision ratio, the macro recall ratio and the macro F1.
Other embodiments or specific implementation manners of the model evaluation device based on mass data according to the present invention may refer to the above method embodiments, and are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third and the like do not denote any order, but rather the words first, second and the like may be interpreted as indicating any order.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A model evaluation method based on mass data is characterized by comprising the following steps:
acquiring mass picture information from a cache, and sorting the mass picture information into a sample data set;
dividing the sample data set into a training set and a test set in a uniformly distributed mode;
training a mass data clustering model according to the training set, randomly verifying the trained data clustering model for n times according to the test set, and finally returning n times of test results, wherein n is an integer greater than or equal to 1;
calculating precision ratio and recall ratio of each confusion matrix according to the n times of test results;
calculating macro precision, macro recall and macro F1 according to the precision and recall of each confusion matrix;
and evaluating the trained data clustering model according to the macro precision ratio, the macro recall ratio and the macro F1.
2. The method for evaluating a model based on mass data according to claim 1, wherein before acquiring the mass image information from the cache and sorting the mass image information into the sample data set, the method further comprises:
acquiring a target community face snapshot picture stream;
carrying out structural processing on the face snapshot picture stream to obtain structural picture data;
cleaning an invalid picture without a human face recognized in the structured picture data to obtain a cleaned picture;
and storing the cleaned pictures to a cache according to the target community classification.
3. The model evaluation method based on mass data according to claim 2, wherein the storing the cleaned picture to a cache according to the target community classification specifically comprises:
adjusting the cleaned picture into a uniform picture format and size to obtain an adjusted picture;
and acquiring the gender characteristics of the face in the adjusted picture, and classifying and storing the adjusted picture to a cache according to the gender characteristics and the target community.
4. The model evaluation method based on mass data according to claim 3, wherein the obtaining of the gender feature of the face in the adjusted picture, and the storing of the adjusted picture into a cache according to the gender feature by classification according to the target community specifically comprise:
acquiring the cell number, the gender characteristics of the face and a timestamp in the adjusted picture;
generating a corresponding face identifier for each adjusted picture according to the timestamp;
and classifying the adjusted picture by taking the cell number as a first dimension, the gender characteristic of the face as a second dimension and the face identification as a third dimension, and storing the classification result to a cache.
5. The model evaluation method based on mass data according to claim 4, wherein the automatically generating a face identifier for each of the adjusted pictures according to the timestamp specifically comprises:
and generating a corresponding face identifier for each adjusted picture according to the timestamp and the random number of the preset digit.
6. The model evaluation method based on mass data according to claim 1, wherein the verifying the trained data clustering model for n times at random according to the test set specifically comprises:
step S1, splitting the sample data set D into 10 sample subsets D1 to D10, respectively;
step S2, starting to test the sample m (m is more than or equal to 1 and less than or equal to n), randomly taking a male sample from each cell of the buffer database by the sample subset D1, and marking the sample as unavailable;
step S3, the sample subset D2 randomly takes a female sample from each cell of the buffer database and marks the sample as unavailable;
step S4, repeating steps S2 and S3 for the remaining sample subsets, and after one round of sampling is performed on 10 sample subsets, determining whether the sample data set D has available samples, and if so, starting the loop from the beginning again until all samples in the sample data set D are marked as unavailable;
step S5, taking a sample set S as a training set, wherein the sample set S is D1U D2U … … U Di (i is not more than 1 and not more than n, and i is not more than n-m +1), and training the sample set S by using a clustering algorithm to obtain a clustering algorithm model;
step S6, according to the clustering algorithm model obtained in the step S5, the sample data set D is subjected to(n-m+1)Testing to obtain a multi-class confusion matrix as a test result;
repeating the steps S2 to S6 n times to obtain n multi-class confusion matrices.
7. The model evaluation method based on mass data according to any one of claims 1 to 6, wherein the calculating macro precision, macro recall and macro F1 according to the precision and recall of each confusion matrix specifically comprises:
step S11, starting to execute the cycle of the ith (i is more than or equal to 1 and less than or equal to n), and converting the ith multi-class confusion matrix into a two-class confusion matrix;
step S12, carrying out formula derivation on the two-classification confusion matrix to obtain precision ratio and recall ratio of the two-classification confusion matrix;
step S13, the loop step is S11 and S12 to obtain precision and recall of all n confusion matrixes;
and calculating the average value of all precision ratios and recall ratios according to a preset formula to obtain the macro precision ratio, the macro recall ratio and the corresponding macro F1.
8. A model evaluation device based on mass data is characterized in that the model evaluation device based on mass data comprises:
the sorting module is used for acquiring mass picture information from the cache and sorting the mass picture information into a sample data set;
the dividing module is used for dividing the sample data set into a training set and a test set in a uniformly distributed mode;
the training verification module is used for training the mass data clustering model according to the training set, randomly verifying the trained data clustering model for n times according to the test set, and finally returning n times of test results, wherein n is an integer greater than or equal to 1;
the calculation module is used for calculating the precision ratio and the recall ratio of each confusion matrix according to the n times of test results;
the calculating module is further configured to calculate a macro precision, a macro recall and a macro F1 according to the precision and the recall of each confusion matrix;
and the evaluation module is used for evaluating the trained data clustering model according to the macro precision ratio, the macro recall ratio and the macro F1.
9. A model evaluation device based on mass data is characterized in that the model evaluation device based on mass data comprises: a memory, a processor and a mass data based model evaluation program stored on the memory and executable on the processor, the mass data based model evaluation program when executed by the processor implementing the steps of the mass data based model evaluation method according to any one of claims 1 to 7.
10. A storage medium, characterized in that the storage medium stores thereon a mass data-based model evaluation program, which when executed by a processor implements the steps of the mass data-based model evaluation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110842391.1A CN113505730A (en) | 2021-07-26 | 2021-07-26 | Model evaluation method, device, equipment and storage medium based on mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110842391.1A CN113505730A (en) | 2021-07-26 | 2021-07-26 | Model evaluation method, device, equipment and storage medium based on mass data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113505730A true CN113505730A (en) | 2021-10-15 |
Family
ID=78014563
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110842391.1A Pending CN113505730A (en) | 2021-07-26 | 2021-07-26 | Model evaluation method, device, equipment and storage medium based on mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113505730A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112861898A (en) * | 2019-11-28 | 2021-05-28 | 初速度（苏州）科技有限公司 | Training sample obtaining method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109344618A (en) * | 2018-02-08 | 2019-02-15 | 中国人民解放军陆军炮兵防空兵学院郑州校区 | A kind of malicious code classification method based on depth forest |
| CN109934293A (en) * | 2019-03-15 | 2019-06-25 | 苏州大学 | Image-recognizing method, device, medium and obscure perception convolutional neural networks |
| CN110059852A (en) * | 2019-03-11 | 2019-07-26 | 杭州电子科技大学 | A kind of stock yield prediction technique based on improvement random forests algorithm |
| CN110909977A (en) * | 2019-10-12 | 2020-03-24 | 郑州电力高等专科学校 | Power grid fault diagnosis method based on ADASYN-DHSD-ET |
| CN111861487A (en) * | 2020-07-10 | 2020-10-30 | 中国建设银行股份有限公司 | Financial transaction data processing method, and fraud monitoring method and device |
| CN112351429A (en) * | 2020-10-22 | 2021-02-09 | 珠海高凌信息科技股份有限公司 | Harmful information detection method and system based on deep learning |
| CN112613536A (en) * | 2020-12-08 | 2021-04-06 | 燕山大学 | Near infrared spectrum diesel grade identification method based on SMOTE and deep learning |
| CN113011742A (en) * | 2021-03-18 | 2021-06-22 | 恒睿(重庆)人工智能技术研究院有限公司 | Clustering effect evaluation method, system, medium and device |
-
2021
- 2021-07-26 CN CN202110842391.1A patent/CN113505730A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109344618A (en) * | 2018-02-08 | 2019-02-15 | 中国人民解放军陆军炮兵防空兵学院郑州校区 | A kind of malicious code classification method based on depth forest |
| CN110059852A (en) * | 2019-03-11 | 2019-07-26 | 杭州电子科技大学 | A kind of stock yield prediction technique based on improvement random forests algorithm |
| CN109934293A (en) * | 2019-03-15 | 2019-06-25 | 苏州大学 | Image-recognizing method, device, medium and obscure perception convolutional neural networks |
| CN110909977A (en) * | 2019-10-12 | 2020-03-24 | 郑州电力高等专科学校 | Power grid fault diagnosis method based on ADASYN-DHSD-ET |
| CN111861487A (en) * | 2020-07-10 | 2020-10-30 | 中国建设银行股份有限公司 | Financial transaction data processing method, and fraud monitoring method and device |
| CN112351429A (en) * | 2020-10-22 | 2021-02-09 | 珠海高凌信息科技股份有限公司 | Harmful information detection method and system based on deep learning |
| CN112613536A (en) * | 2020-12-08 | 2021-04-06 | 燕山大学 | Near infrared spectrum diesel grade identification method based on SMOTE and deep learning |
| CN113011742A (en) * | 2021-03-18 | 2021-06-22 | 恒睿(重庆)人工智能技术研究院有限公司 | Clustering effect evaluation method, system, medium and device |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112861898A (en) * | 2019-11-28 | 2021-05-28 | 初速度（苏州）科技有限公司 | Training sample obtaining method and device |
| CN112861898B (en) * | 2019-11-28 | 2022-06-10 | 魔门塔(苏州)科技有限公司 | Training sample acquisition method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220043827A1 (en) | Systems and methods for configuring system memory for extraction of latent information from big data | |
| CN112016623B (en) | Face clustering method, device, equipment and storage medium | |
| CN111539451B (en) | Sample data optimization method, device, equipment and storage medium | |
| CN112306829B (en) | Method and device for determining performance information, storage medium and terminal | |
| CN104766062A (en) | Face recognition system and register and recognition method based on lightweight class intelligent terminal | |
| CN112132279B (en) | Convolutional neural network model compression method, device, equipment and storage medium | |
| CN111159115A (en) | Similar file detection method, device, equipment and storage medium | |
| Thai et al. | Estimation of primary quantization steps in double-compressed JPEG images using a statistical model of discrete cosine transform | |
| CN113505730A (en) | Model evaluation method, device, equipment and storage medium based on mass data | |
| CN115795329A (en) | Power utilization abnormal behavior analysis method and device based on big data grid | |
| CN109697155B (en) | IT system performance evaluation method, device, equipment and readable storage medium | |
| CN110489175A (en) | Service processing method, device, server and storage medium | |
| CN107688744B (en) | Malicious file classification method and device based on image feature matching | |
| CN110209260B (en) | Power consumption abnormality detection method, device, equipment and computer readable storage medium | |
| CN111091194B (en) | Operation system identification method based on CAVWBB _ KL algorithm | |
| CN112215176A (en) | Method and device for releasing face image based on differential privacy | |
| CN112488140A (en) | Data association method and device | |
| CN114519520B (en) | Model evaluation method, device and storage medium | |
| CN116010216A (en) | Method, device, equipment and storage medium for evaluating health degree of data asset | |
| CN113115107B (en) | Handheld video acquisition terminal system based on 5G network | |
| CN115423201A (en) | Method, device and equipment for predicting power generation capacity data and computer readable storage medium | |
| CN112866142B (en) | Mobile internet real flow identification method and device | |
| CN114357082A (en) | Cloud computing-based big data analysis method and system | |
| CN113627542A (en) | Event information processing method, server and storage medium | |
| CN112328779A (en) | Training sample construction method and device, terminal equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211015 |