CN117251725A

CN117251725A - Method and device for identifying data based on machine learning

Info

Publication number: CN117251725A
Application number: CN202310769618.3A
Authority: CN
Inventors: 张清森; 陈俊灵; 易启邦; 吕旭明
Original assignee: Shenzhen Lvlian Technology Co Ltd
Current assignee: Shenzhen Lvlian Technology Co Ltd
Priority date: 2023-06-27
Filing date: 2023-06-27
Publication date: 2023-12-19

Abstract

The invention discloses a method and a device for identifying data based on machine learning, wherein the method comprises the following steps: the acquired training data set aiming at the preset data recognition requirement is input into the machine learning type data recognition model matched with the preset data recognition requirement, the data recognition model is trained, an accurate and reliable data recognition model can be trained, and the trained target data recognition model is deployed into a corresponding framework for application, so that the recognition accuracy and recognition precision of data to be recognized can be improved; and the mobility of the target data identification model can be improved by deploying the target data identification model into a preset frame, so that the target data identification model is better suitable for related working environments.

Description

Method and device for identifying data based on machine learning

Technical Field

The invention relates to the technical field of artificial intelligence, in particular to a method and a device for identifying data based on machine learning.

Background

In the prior art, the conventional OCR or SDK of a third party manufacturer is often required to be used for image and text recognition development so as to realize the functions of recognizing data such as characters, registering images and the like. However, in relatively complex environments, such as recognition of relatively blurred pictures or text, the recognition accuracy of conventional OCR or third party vendors' SDKs is low. Therefore, it is important to provide a technical scheme capable of improving the accuracy of data identification.

Disclosure of Invention

The invention provides a method and a device for identifying data based on machine learning, which can be beneficial to improving the identification precision of identifying the data and the range of applicable data, and improve the accuracy of identifying the data.

To solve the above technical problem, a first aspect of the present invention discloses a method for identifying data based on machine learning, the method comprising: acquiring a training data set aiming at preset data identification requirements, wherein the type of the training data set comprises a text type and/or an image type;

inputting the training data set into a predetermined data identification model matched with the preset data identification requirement, training the data identification model to obtain a trained target data identification model, wherein the data identification model is a model of a machine learning type;

deploying the target data identification model into a preset framework;

and identifying the data to be identified through the target data identification model deployed in the preset framework to obtain target information corresponding to the preset data identification requirement.

As an optional implementation manner, in the first aspect of the present invention, a model framework of a data identification model is determined according to the preset data identification requirement and the parameters of the training data set, where the model framework includes one of a convolutional neural network framework, a cyclic neural network framework and a transfer learning framework;

Determining a batch size of the data recognition model according to parameters of the training data set, wherein the parameters of the training data set comprise one or more of a data type, a data size and a data quantity;

determining an initial learning rate of the data recognition model according to a model framework of the data recognition model, parameters of the training data set and a batch size of the data recognition model;

and generating the data identification model according to the model framework, the batch size and the initial learning rate, and taking the data identification model as the data identification model matched with the preset data identification requirement.

As an optional implementation manner, in the first aspect of the present invention, a verification data set is input into the target data identification model, and a performance parameter set of the target data identification model is calculated, wherein a type of the verification data set includes a text type and/or an image type;

adding a weight coefficient for each performance parameter in the performance parameter set according to the data identification effect corresponding to the preset data identification requirement;

calculating the comprehensive performance value of the target data identification model according to the performance parameter set of the target data identification model and the weight coefficient of each performance parameter in the performance parameter set;

Evaluating whether the comprehensive performance value of the target data identification model meets the standard according to a preset comprehensive performance threshold value to obtain an evaluation result;

and when the evaluation result shows that the comprehensive performance value of the target data identification model meets the standard, executing the operation of deploying the target data identification model into a preset framework.

As an optional implementation manner, in the first aspect of the present invention, before the inputting the verification data set into the target data identification model, the method further includes, before calculating the performance parameter set of the target data identification model:

determining a data set matched with the preset data identification requirement and the application scene of the target data identification model as the verification data set according to the preset data identification requirement and the application scene of the target data identification model;

determining parameters of the verification data set, and determining a performance parameter type to be calculated according to the preset data identification requirement and the parameters of the verification data set, wherein the performance parameter type comprises at least one of accuracy rate, precision rate, recall rate, F1 value, ROC curve, AUC value and confusion matrix;

Wherein inputting a validation data set into the target data identification model, calculating a set of performance parameters of the target data identification model, comprises:

inputting the verification data set and the performance parameter type into the target data identification model, and calculating a parameter set matched with the performance parameter type to be used as the performance parameter set of the target data identification model.

As an optional implementation manner, in the first aspect of the present invention, after the acquiring the training data set for the preset data identification requirement, the method further includes:

according to the preset data identification requirement and parameters of the training data set, carrying out noise reduction treatment on data in the training data set to obtain a first training data set;

performing data enhancement operations on the first training data set to obtain a second training data set, wherein the data enhancement operations comprise at least one of cutting, rotating, scaling and translating;

labeling the second training data set according to the preset data identification requirement to obtain a target training data set;

inputting the training data set into a predetermined data identification model matched with the preset data identification requirement, training the data identification model to obtain a trained target data identification model, and comprising the following steps:

And inputting the target training data set into a predetermined data identification model matched with the preset data identification requirement, and training the data identification model to obtain a trained target data identification model.

As an optional implementation manner, in the first aspect of the present invention, when the evaluation result indicates that the comprehensive performance value of the target data identification model does not reach the standard, the method further includes:

determining scene parameters of an application scene required to be applied by the target data identification model, wherein the scene parameters of the application scene comprise one or more of data amount of data to be identified in the application scene, data type of the data to be identified, concurrency amount of the data to be identified and data feedback duration requirements of the data to be identified;

analyzing scene parameters of the application scene to obtain data identification requirement parameters matched with the application scene;

determining a data type of the target data identification model;

optimizing the target data identification model according to the data identification requirement parameters and the data type of the target data identification model, updating the optimized target data identification model into the target data identification model, and triggering and executing the operation of deploying the target data identification model into a preset framework.

In a first aspect of the present invention, the optimizing the target data recognition model according to the data recognition requirement parameter and the data type of the target data recognition model, and updating the optimized target data recognition model to the target data recognition model, includes:

determining redundancy coefficients of each layer of the target data identification model according to the data identification demand parameters;

comparing the redundancy coefficient of each layer of the target data identification model with a preset first redundancy coefficient threshold value;

when the redundancy coefficient of a certain layer of the target data identification model is larger than a preset first redundancy coefficient threshold value, adding the layer into a redundancy layer set;

determining the number of layers to be pruned by a network according to the difference value between the redundancy coefficient of each layer in the redundancy layer set and a preset second redundancy coefficient threshold value and the data identification demand parameter, wherein the second redundancy coefficient threshold value is larger than the first redundancy coefficient threshold value;

performing network pruning on layers with redundancy coefficients larger than the second redundancy coefficient threshold value in the redundancy layer set according to the number of layers needing to be network pruned;

After the network pruning operation is finished on the layer with the redundancy coefficient larger than the second redundancy coefficient threshold value in the redundancy layer set, carrying out model quantization on the target data identification model according to the data type of the target data identification model, and updating the target data identification model after the model quantization is finished into the target data identification model.

The second aspect of the invention discloses a device for identifying data based on machine learning, the device comprises:

the acquisition module is used for acquiring a training data set aiming at preset data identification requirements, wherein the type of the training data set comprises a text type and/or an image type;

the training module is used for inputting the training data set into a predetermined data identification model matched with the preset data identification requirement, training the data identification model to obtain a trained target data identification model, wherein the data identification model is a model of a machine learning type;

the deployment module is used for deploying the target data identification model into a preset frame;

the data identification module is used for identifying the data to be identified through the target data identification model deployed in the preset frame to obtain target information corresponding to the preset data identification requirement.

As an alternative embodiment, in the second aspect of the present invention, the apparatus further includes:

the first determining module is used for determining a model framework of a data identification model according to the preset data identification requirement and parameters of the training data set, wherein the model framework comprises one of a convolutional neural network framework, a cyclic neural network framework and a transfer learning framework;

the first determining module is further configured to determine a batch size of the data identification model according to parameters of the training data set, where the parameters of the training data set include one or more of a data type, a data size, and a data amount;

the first determining module is further configured to determine an initial learning rate of the data recognition model according to a model framework of the data recognition model, parameters of the training dataset, and a batch size of the data recognition model;

and the generation module is used for generating the data identification model according to the model framework, the batch size and the initial learning rate to serve as a data identification model matched with the preset data identification requirement.

A computing module for inputting a verification data set into the target data identification model, computing a performance parameter set of the target data identification model, wherein the type of the verification data set comprises a text type and/or an image type;

the adding module is used for adding a weight coefficient to each performance parameter in the performance parameter set according to the data identification effect corresponding to the preset data identification requirement;

the calculation module is further used for calculating the comprehensive performance value of the target data identification model according to the performance parameter set of the target data identification model and the weight coefficient of each performance parameter in the performance parameter set;

the evaluation module is used for evaluating whether the comprehensive performance value of the target data identification model meets the standard according to a preset comprehensive performance threshold value to obtain an evaluation result;

and when the evaluation result shows that the comprehensive performance value of the target data identification model meets the standard, triggering the deployment module to execute the operation of deploying the target data identification model into a preset framework.

the second determining module is used for determining a data set matched with the preset data identification requirement and the application scene of the target data identification model as the verification data set according to the preset data identification requirement and the application scene of the target data identification model before the computing module inputs the verification data set into the target data identification model and computes the performance parameter set of the target data identification model;

The second determining module is further configured to determine parameters of the verification data set, and determine a performance parameter type to be calculated according to the preset data identification requirement and the parameters of the verification data set, where the performance parameter type includes at least one of an accuracy rate, a recall rate, an F1 value, an ROC curve, an AUC value, and a confusion matrix;

the computing module inputs the verification data set into the target data identification model, and the computing mode of the performance parameter set of the target data identification model specifically comprises the following steps:

the noise reduction module is used for carrying out noise reduction processing on data in the training data set according to the preset data identification requirement and parameters of the training data set after the training data set aiming at the preset data identification requirement is acquired by the acquisition module, so as to obtain a first training data set;

The data enhancement module is used for executing data enhancement operation on the first training data set to obtain a second training data set, and the data enhancement operation comprises at least one of cutting, rotating, scaling and translating;

the labeling module is used for labeling the second training data set according to the preset data identification requirement to obtain a target training data set;

the training module inputs the training data set to a predetermined data identification model matched with the preset data identification requirement, trains the data identification model, and the method for obtaining the trained target data identification model specifically comprises the following steps:

the third determining module is used for determining scene parameters of an application scene required to be applied by the target data identification model when the evaluation result shows that the comprehensive performance value of the target data identification model does not reach the standard, wherein the scene parameters of the application scene comprise one or more of data quantity of data to be identified in the application scene, data type of the data to be identified, concurrency quantity of the data to be identified and data feedback duration requirements of the data to be identified;

The analysis module is used for analyzing scene parameters of the application scene to obtain data identification requirement parameters matched with the application scene;

the third determining module is further configured to determine a data type of the target data identification model;

the optimizing module is used for optimizing the target data identification model according to the data identification requirement parameters and the data type of the target data identification model, updating the optimized target data identification model into the target data identification model, and triggering the deployment module to execute the operation of deploying the target data identification model into a preset framework.

In a second aspect of the present invention, the optimizing module optimizes the target data recognition model according to the data recognition requirement parameter and the data type of the target data recognition model, and updates the optimized target data recognition model to the target data recognition model specifically includes:

Another apparatus for identifying data based on machine learning is disclosed in a third aspect of the present invention, the apparatus comprising:

a memory storing executable program code;

A processor coupled to the memory;

the processor invokes the executable program code stored in the memory to perform the method of identifying data based on machine learning disclosed in the first aspect of the invention.

A fourth aspect of the invention discloses a computer storage medium storing computer instructions that, when invoked, are adapted to perform the method of identifying data based on machine learning disclosed in the first aspect of the invention.

Compared with the prior art, the embodiment of the invention has the following beneficial effects:

according to the embodiment of the invention, the acquired training data set aiming at the preset data identification requirement is input into the machine learning data identification model matched with the preset data identification requirement, the data identification model is trained, an accurate and reliable data identification model can be trained, and the trained target data identification model is deployed into a corresponding frame for application, so that the identification accuracy and identification precision of data to be identified can be improved; and the mobility of the target data identification model can be improved by deploying the target data identification model into a preset frame, so that the target data identification model is better suitable for related working environments.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an application framework of a method for identifying data based on machine learning according to an embodiment of the present invention;

FIG. 2 is a flow chart of a method for identifying data based on machine learning according to an embodiment of the present invention;

FIG. 3 is a flow chart of another method for identifying data based on machine learning according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of an apparatus for identifying data based on machine learning according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of another apparatus for identifying data based on machine learning according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of another apparatus for recognizing data based on machine learning according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The terms first, second and the like in the description and in the claims and in the above-described figures are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, article, or article that comprises a list of steps or elements is not limited to only those listed but may optionally include other steps or elements not listed or inherent to such process, method, article, or article.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

The invention discloses a method and a device for identifying data based on machine learning, which can be beneficial to improving the identification precision of identifying the data and the range of applicable data, and improve the accuracy of identifying the data. The following will describe in detail.

In order to better understand the method and apparatus for identifying data based on machine learning described in the present invention, first, a description is given of a structure of an application framework of the method for identifying data based on machine learning, specifically, the structure of the application framework of the method for identifying data based on machine learning described in the present invention may be shown in fig. 1, and fig. 1 is a schematic structural diagram of an application framework of the method for identifying data based on machine learning disclosed in the embodiment of the present invention. As shown in fig. 1, the application framework may include a Vision framework and/or an OpenCV framework, and the application framework can be quickly matched with a device of the framework to adapt and import a model into the device (such as a apple phone), and after data is acquired by a data acquisition device (such as a camera) of the device, text detection, bar code identification, image registration or general feature tracking operation is performed on the data by the trained Core ML-based model mlmode in the application framework, and the features are analyzed and the result is output.

It should be noted that, the schematic structural diagram of the application framework of the method for identifying data based on machine learning shown in fig. 1 is only for illustrating the structure of the application framework to which the present invention is applied, and the specific framework and structure, such as the Vision framework, are also only schematically illustrated, which is not limited by the schematic structural diagram shown in fig. 1. The structure diagram of the application framework of the method for identifying data based on machine learning is described above, and the method and the device for identifying data based on machine learning are described in detail below.

Example 1

Referring to fig. 2, fig. 2 is a flowchart of a method for identifying data based on machine learning according to an embodiment of the present invention. The method for identifying data based on machine learning described in fig. 2 may be applied to a device for identifying data based on machine learning, where the device for identifying data based on machine learning may include any one of an intelligent server or an intelligent platform for controlling the device for identifying data based on machine learning, where the intelligent server includes a local server or a cloud server, and embodiments of the present invention are not limited. As shown in fig. 2, the machine learning based method of identifying data may include the operations of:

101. A training data set aiming at preset data identification requirements is obtained.

In the embodiment of the present invention, optionally, the preset data identification requirement may be a preset identification purpose, and the preset identification purpose may include one or more of text detection, barcode identification, image registration and general feature tracking. The data in the training data set can comprise text type and/or image type, the data in the training data set can be random data independent of each other, and can also be data pairs or data groups which are arranged in pairs or groups and can play a role in comparing recognition effects, and the invention is not limited.

In the embodiment of the present invention, optionally, the data in the training data set needs to satisfy a preset data condition, where the data condition may include a data quality condition and/or a data sufficiency condition; taking text detection as a preset data identification requirement as an example, the data in the training data set needs to meet a certain definition requirement, and the data in the training data set needs to include data containing texts and data without texts, wherein the types of the texts in the data containing texts can include one or more of literal characters, digital characters, related punctuation marks and the like of different languages, and the invention is not limited.

102. And inputting the training data set into a data identification model which is determined in advance and matched with the preset data identification requirement, and training the data identification model to obtain a trained target data identification model.

In an embodiment of the present invention, optionally, the data recognition model may be a machine learning model, and specifically, the data recognition model may be a Core ML-based neural network OCR (Optical Character Recognition ) recognition model.

103. And deploying the target data identification model into a preset framework.

In the embodiment of the present invention, optionally, the preset frame may be a Vision frame, or may be an OpenCV frame, and in the embodiment of the present invention, the Vision frame is preferred.

104. And identifying the data to be identified through the target data identification model deployed in the preset framework to obtain target information corresponding to the preset data identification requirement.

In the embodiment of the invention, optionally, the data to be identified can be prepared data in advance, or can be data acquired on site through intelligent equipment, such as image data obtained by photographing through a mobile phone; the type of the data to be identified may be a text type and/or an image type, and the present invention is not limited thereto.

In the embodiment of the present invention, optionally, the target information corresponding to the preset data identification requirement may include one or more combinations of text information, barcode information, two-dimensional code information, image registration information and general feature tracking information. Specifically, when the preset data identification requirement is text detection, the target information is text information, for example, when the preset data identification requirement is an identification card number required to identify an identification card or a bank card number required to identify a bank card, the target information is a corresponding identification card number or bank card number, which is not limited in this embodiment.

As can be seen, implementing the method for recognizing data based on machine learning described in fig. 2 can obtain a training data set for a preset data recognition requirement, input the training data set into a data recognition model which is determined in advance and matches with the preset data recognition requirement, train the data recognition model to obtain a trained target data recognition model, improve accuracy and reliability of recognizing data by the target data recognition model, deploy the target data recognition model into the preset frame, improve mobility of the target data recognition model, be better suitable for a related working environment, recognize data to be recognized by deploying the target data recognition model into the preset frame, obtain target information corresponding to the preset data recognition requirement, improve recognition accuracy of recognizing data and range of applicable data, and improve accuracy of recognizing data.

In an alternative embodiment, the method for identifying data based on machine learning may further comprise the operations of:

inputting a verification data set into the target data identification model, calculating a performance parameter set of the target data identification model, wherein the type of the verification data set comprises a text type and/or an image type;

adding a weight coefficient for each performance parameter in the performance parameter set according to a data identification effect corresponding to a preset data identification requirement;

and when the evaluation result shows that the comprehensive performance value of the target data identification model meets the standard, performing the operation of deploying the target data identification model into a preset framework.

In this optional embodiment, optionally, the type of the verification data set may include a text type and/or an image type, and the data recognition effect may include one of a recognition accuracy of a character, a recognition accuracy of a barcode, and an accuracy of image registration, which is not limited in this embodiment.

In this optional embodiment, optionally, the weight coefficient of each performance parameter may be determined according to a data recognition effect corresponding to a preset data recognition requirement, may be determined according to a history training record or a history recognition record corresponding to a preset data recognition requirement, may be adjusted according to a weight coefficient in history training data of a data recognition model, and the preset comprehensive performance threshold may be determined according to a data recognition effect corresponding to a preset data recognition requirement, and may be determined according to a history training record or a history recognition record corresponding to a preset data recognition requirement.

Therefore, by implementing the optional embodiment, whether the target data recognition model meets the preset requirement can be calculated according to the performance parameter set of the target data recognition model, and the target data recognition model is deployed into the preset framework when the target data recognition model meets the preset requirement, so that the data recognition accuracy and reliability of the target data recognition model are improved.

In another alternative embodiment, the method of identifying data based on machine learning may further comprise, prior to inputting the validation data set into the target data identification model, calculating a set of performance parameters for the target data identification model, the operations of:

According to the application scenes of the preset data identification requirements and the target data identification model, determining a data set matched with the application scenes of the preset data identification requirements and the target data identification model as a verification data set;

determining parameters of a verification data set, and determining a performance parameter type to be calculated according to preset data identification requirements and the parameters of the verification data set, wherein the performance parameter type comprises at least one of accuracy, precision, recall rate, F1 value, ROC curve, AUC value and confusion matrix;

wherein inputting the validation data set into the target data identification model, calculating the set of performance parameters of the target data identification model may comprise:

inputting the verification data set and the performance parameter type into the target data identification model, and calculating a parameter set matched with the performance parameter type to serve as the performance parameter set of the target data identification model.

In this optional embodiment, optionally, the application scenario of the preset data recognition requirement and the target data recognition model may include one of text detection, barcode recognition, image registration and general feature tracking, and when the application scenario is text detection, the dataset matched with the application scenario may include text data, image data with text information, and the like; when the application scene is bar code recognition, the data set matched with the application scene can comprise bar code data, two-dimensional code data and other forms of information coding data capable of carrying information; when the application scene is image registration, the data set matched with the application scene may include image data or text data arranged in pairs or groups and capable of playing a role in matching or contrasting with each other; when the application scene is general feature tracking, the data set matched with the application scene may include image data after feature enhancement and feature filtering, which is not limited in this embodiment.

In this alternative embodiment, optionally, the parameters of the verification data set may include one or more of data type, data size, data amount, resolution of image data, and the like, and the performance parameter type includes at least one of Accuracy (Accuracy), precision (Precision), recall (Recall), F1 value (F1-score), ROC curve (Receiver Operating Characteristic Curve, recipient operating characteristic curve), AUC value (Area Under the Curve, area under curve), and Confusion Matrix (fusion Matrix), wherein the higher the Accuracy of the target data identification model is used to represent the proportion of the model that is correctly classified in all samples, the better the Accuracy represents the identification effect of the model; the accuracy of the target data recognition model is used for representing the proportion of the model predicted to be positive in the samples of the positive class in the classification problem (positive class and negative class) to be truly positive; the recall rate of the target data identification model is used for representing the proportion of the samples truly in the positive class, which is predicted by the model to be in the positive class; the F1 value of the target data identification model is a harmonic average value of the accuracy rate and the recall rate of the model; the ROC curve of the target data recognition model may be used to evaluate the classification performance of the model; the AUC value of the target data identification model can be used for evaluating the sequencing capability and classification accuracy of model prediction; the confusion matrix of the target data recognition model may show a correspondence between a classification result of the model and a real label, including a real case, a false positive case, a true negative case, and a false negative case, which is not limited in this embodiment.

Therefore, the optional embodiment can determine the data set matched with the preset data identification requirement and the application scene of the target data identification model according to the preset data identification requirement and the application scene of the target data identification model, determine the parameters of the verification data set as the verification data set, determine the performance parameter types required to be calculated according to the preset data identification requirement and the parameters of the verification data set, input the verification data set and the performance parameter types into the target data identification model, calculate the parameter set matched with the performance parameter types and serve as the performance parameter set of the target data identification model, so that the determination accuracy of the performance parameter set of the target data identification model is improved, the accuracy of evaluating the target data identification model is further improved, the performance of the target data identification model is ensured, and the identification accuracy and the response speed of the target data identification model are improved.

In yet another alternative embodiment, after acquiring the training data set for the preset data identification requirement, the method for identifying data based on machine learning may further include the operations of:

according to preset data identification requirements and parameters of a training data set, carrying out noise reduction treatment on data in the training data set to obtain a first training data set;

Performing data enhancement operations on the first training data set to obtain a second training data set, the data enhancement operations including at least one of clipping, rotation, scaling, and translation;

inputting the training data set into a predetermined data recognition model matched with the preset data recognition requirement, training the data recognition model, and obtaining a trained target data recognition model can comprise the following operations:

inputting the target training data set into a predetermined data recognition model matched with the preset data recognition requirement, and training the data recognition model to obtain a trained target data recognition model

In this optional embodiment, optionally, the noise reduction processing on the data in the training data set may include identifying noise, blur, distortion, and other interference factors in the data, and performing operations such as targeted denoising, sharpness improvement, recognition rate, and anti-distortion on the interference factors, where the embodiment is not limited.

In this optional embodiment, optionally, the data enhancement operation may include generating new training data samples by means of data clipping, data rotation, data scaling, data color transformation, and so on, so as to improve robustness and generalization capability of the model, where the data enhancement operation may be performed according to a preset condition or may be performed randomly, and this embodiment is not limited.

In this embodiment of the present invention, optionally, labeling the second training data set may include adding a correct label to each training sample, for example, when the training sample is an identification card and the data identification purpose of the target data identification model is a number for identifying the identification card, a corresponding identification card number may be added to the training sample, so as to train the model by using a supervised learning method in the training process; in addition, the training data set can be further segmented, and the identity card image can be segmented into an identity card number area and other areas, so that the concentration of the model on the identification task of the identity card number is improved, the interference of other areas on training is reduced, and the embodiment is not limited.

Therefore, according to the implementation of the optional embodiment, the data processing operations such as noise reduction, data enhancement and labeling can be performed on the training data set according to the preset data identification requirement and parameters of the training data set, so that the matching degree of data in the training data set in the training process of the target data identification model and the richness, the accuracy and the reliability of the data in the training data set are improved, and the data identification capability and the accuracy of the target data identification model can be improved.

In yet another alternative embodiment, when the evaluation result indicates that the comprehensive performance value of the target data recognition model does not reach the standard, the method for recognizing data based on machine learning may further include the following operations:

determining scene parameters of an application scene required to be applied by the target data identification model, wherein the scene parameters of the application scene comprise one or more of data quantity of data to be identified in the application scene, data type of the data to be identified, concurrency quantity of the data to be identified and data feedback duration requirements of the data to be identified;

determining the data type of the target data identification model;

In this optional embodiment, optionally, the scene parameters of the application scene may include one or more combinations of a data amount of data to be identified in the application scene, a data type of the data to be identified, a data concurrency amount of the target data identification model in the application scene, and a data feedback duration requirement of the data to be identified, and when the evaluation result indicates that the comprehensive performance value of the target data identification model does not reach the standard, the comprehensive performance value of the target data identification model can reach the standard by performing iterative training on the target data identification model, which is not limited in this embodiment.

In this optional embodiment, optionally, the data identification requirement parameter matched with the application scenario may include a data parameter requirement, a data concurrency requirement, a data feedback duration requirement, a structure requirement of the target data identification model, and the like of the application scenario, which is not limited in this embodiment.

Therefore, when the evaluation result indicates that the comprehensive performance value of the target data identification model does not reach the standard, the implementation of the optional embodiment can optimize the target data identification model according to the data identification requirement parameter of the application scene of the target data identification model and the data type of the target data identification model, so that the data identification accuracy and reliability of the target data identification model can be ensured.

In yet another alternative embodiment, optimizing the target data recognition model according to the data recognition requirement parameter and the data type of the target data recognition model, and updating the optimized target data recognition model to the target data recognition model may include the operations of:

determining the number of layers to be pruned by a network according to the difference value between the redundancy coefficient of each layer in the redundancy layer set and a preset second redundancy coefficient threshold value and the data identification requirement parameter, wherein the second redundancy coefficient threshold value is larger than the first redundancy coefficient threshold value;

performing network pruning on layers with redundancy coefficients larger than a second redundancy coefficient threshold value in the redundancy layer set according to the number of layers required to be network pruned;

after the network pruning operation is finished on the layer with the redundancy coefficient larger than the second redundancy coefficient threshold value in the redundancy layer set, carrying out model quantization on the target data recognition model according to the data type of the target data recognition model, and updating the target data recognition model after the model quantization into the target data recognition model.

In this optional embodiment, optionally, the redundancy coefficient of each layer of the target data identification model may be used to represent the acting weight of the layer on the data identification task in the target data identification model, the preset first redundancy coefficient threshold is used to divide each layer into a necessary layer and an unnecessary layer, specifically, when the redundancy coefficient of a certain layer is greater than the preset first redundancy coefficient threshold, that is, the layer is determined to be an unnecessary layer, the preset second redundancy coefficient threshold may be used to determine the number of layers that may be pruned by the network, specifically, the number of layers that need to be pruned by the network may be determined according to the difference between the redundancy coefficient of each layer in the redundancy layer set and the preset second redundancy coefficient threshold and the data identification requirement parameter, and the second redundancy coefficient threshold is greater than the first redundancy coefficient threshold.

In this alternative embodiment, optionally, the model quantization is to convert model parameters from floating point type to fixed point type, so as to reduce storage and calculation costs of the model, specifically, the model quantization may be implemented by reducing the number of bits of the model and/or using symmetric quantization techniques, which is not limited in this embodiment.

In this optional embodiment, optionally, hardware optimization may be further performed on hardware to which the target data identification model is applied to improve the identification speed and the calculation efficiency of the model, and specifically, technologies such as a hardware accelerator, distributed calculation, and parallel calculation may be used to improve the calculation efficiency of the model, which is not limited in this embodiment.

Therefore, by implementing the alternative embodiment, the redundancy coefficient of each layer of the target data identification model can be determined according to the data identification requirement parameter, the layers in the redundancy layer set are subjected to network pruning according to the preset first redundancy coefficient threshold value and the preset second redundancy coefficient threshold value, then the target data identification model is subjected to model quantization, the target data identification model after the model quantization is updated to the target data identification model, and the calculation efficiency of the model and the data identification effect of the model can be improved by simplifying the structure of the model and changing the data type of the model.

Example two

Referring to fig. 3, fig. 3 is a flowchart of a method for identifying data based on machine learning according to an embodiment of the present invention. The method for identifying data based on machine learning described in fig. 3 may be applied to a device for identifying data based on machine learning, where the device for identifying data based on machine learning may include any one of an intelligent server or an intelligent platform for controlling the device for identifying data based on machine learning, where the intelligent server includes a local server or a cloud server, and embodiments of the present invention are not limited. As shown in fig. 3, the machine learning based method of identifying data may include the following operations:

201. a training data set aiming at preset data identification requirements is obtained.

202. And determining a model framework of the data identification model according to the preset data identification requirement and parameters of the training data set.

In an embodiment of the present invention, optionally, the model framework of the data identification model may include one of a convolutional neural network framework (Convolutional Neural Network, CNN), a cyclic neural network framework (Recurrent Neural Network, RNN), and a transfer learning framework (Transfer Learning), where the convolutional neural network framework may be used for feature extraction and classification of images, the cyclic neural network framework may process sequence data, such as identifying digital sequences in an identification card number, may capture long-term dependencies in the sequence using RNN variants such as LSTM (Long Short Term Memory, long-term memory recurrent neural network) or GRU (Gated Recurrent Unit, gate-controlled cyclic unit), and transfer learning may use a pre-trained model to extract features in the images, and then input the features into a custom classifier for identification, which is not limited in this embodiment.

203. The batch size of the data recognition model is determined based on parameters of the training dataset.

In the embodiment of the present invention, optionally, parameters of the training data set may include one or more of a data type, a data size, a data amount, a resolution of image data, and the like, which is not limited in this embodiment.

204. And determining the initial learning rate of the data recognition model according to the model framework of the data recognition model, the parameters of the training data set and the batch size of the data recognition model.

In the embodiment of the present invention, optionally, after each training iteration of the data identification model, the learning rate of the data identification model may be changed, which is not limited in this embodiment.

205. And generating a data identification model according to the model frame, the batch size and the initial learning rate, and taking the data identification model as the data identification model matched with the preset data identification requirement.

In the embodiment of the invention, optionally, the data identification model may be generated according to a batch size and an initial learning rate, that is, parameters of the data identification model may include a batch size and an initial learning rate, parameters of the data identification model may further include one or more combinations of a network structure, a convolution kernel size and a step length, a pooling layer, an activation function, a batch standardization layer, a full connection layer, a loss function, an optimizer, a learning rate, and the like, where the network structure of the data identification model may include any one of a LeNet, a VGG, and a ResNet, and depth and width of the network may be adjusted according to complexity of preset data identification requirements and available computing resources; selecting the size and the step length of a convolution kernel matched with the preset data identification requirement, wherein a smaller convolution kernel can capture local features of an image, a larger convolution kernel can capture features in a larger range, and the size of the step length can influence the size of a feature map and the calculated amount of a model; the pooling layer of the data recognition model may include a maximum pooling layer or an average pooling layer to reduce the size of the feature map and extract the most salient features; the activation function of the data identification model may include one of ReLU, sigmoid, or Tanh; the batch standardization layer of the data identification model can accelerate the convergence of the model and improve the robustness of the model; a full connection layer of the data recognition model may be added after the convolution layer to map the extracted features to categories of target data; the loss function of the data identification model can be a cross entropy loss function, and is used for measuring the difference between the model output and the real label; the data recognition model may include Adam and/or SGD, which is not limited in this embodiment.

206. And inputting the training data set into a data identification model which is determined in advance and matched with the preset data identification requirement, and training the data identification model to obtain a trained target data identification model.

207. And deploying the target data identification model into a preset framework.

208. And identifying the data to be identified through the target data identification model deployed in the preset framework to obtain target information corresponding to the preset data identification requirement.

In the embodiment of the present invention, for other descriptions of step 201 and step 206 to step 208, please refer to the detailed descriptions of step 101 to step 104 in the first embodiment, and the present invention is not limited thereto.

It can be seen that the method for recognizing data based on machine learning described in fig. 3 can acquire a training data set for a preset data recognition requirement based on machine learning, determine a model frame of the data recognition model according to the preset data recognition requirement and parameters of the training data set, determine a batch size of the data recognition model according to parameters of the training data set, determine an initial learning rate of the data recognition model according to a model frame and the batch size of the data recognition model, further generate the data recognition model, improve the frame structure of the data recognition model and the rationality of the parameters, further improve the accuracy of the data recognition model in recognizing data and the robustness of the data recognition model, input the training data set into the data recognition model which is matched with the preset data recognition requirement determined in advance, train the data recognition model to obtain a trained target data recognition model, improve the accuracy and reliability of recognizing the data by the target data recognition model, improve the mobility of the target data recognition model according to the model frame and the batch size, further generate the data recognition model, further improve the accuracy of recognizing the data and the data recognition requirement by arranging the target recognition model in the preset frame, and the accuracy of the target recognition model.

Example III

Referring to fig. 4, fig. 4 is a schematic structural diagram of an apparatus for recognizing data based on machine learning according to an embodiment of the present invention. As shown in fig. 4, the apparatus for identifying data based on machine learning may include:

the acquiring module 301 is configured to acquire a training data set for a preset data identification requirement, where a type of the training data set includes a text type and/or an image type;

the training module 302 is configured to input a training data set into a predetermined data recognition model that matches with a preset data recognition requirement, train the data recognition model, and obtain a trained target data recognition model, where the data recognition model is a model of a machine learning type;

the deployment module 303 is configured to deploy the target data identification model into a preset framework;

the data recognition module 304 is configured to recognize data to be recognized through a target data recognition model deployed in a preset framework, so as to obtain target information corresponding to a preset data recognition requirement.

As can be seen, implementing the device for recognizing data based on machine learning described in fig. 4 can obtain a training data set for a preset data recognition requirement, input the training data set into a data recognition model which is determined in advance and matched with the preset data recognition requirement, train the data recognition model to obtain a trained target data recognition model, improve accuracy and reliability of recognizing data by the target data recognition model, deploy the target data recognition model into a preset frame, improve mobility of the target data recognition model, be better suitable for a related working environment, recognize data to be recognized by deploying the target data recognition model into the preset frame, obtain target information corresponding to the preset data recognition requirement, improve recognition accuracy of recognizing data and range of applicable data, and improve accuracy of recognizing data.

In an alternative embodiment, as shown in fig. 5, the apparatus for identifying data based on machine learning may further include:

a first determining module 305, configured to determine a model framework of the data identification model according to a preset data identification requirement and parameters of the training data set, where the model framework includes one of a convolutional neural network framework, a cyclic neural network framework, and a transfer learning framework;

the first determining module 305 is further configured to determine a batch size of the data identification model according to parameters of a training data set, where the parameters of the training data set include one or more of a data type, a data size, and a data amount;

the first determining module 305 is further configured to determine an initial learning rate of the data recognition model according to the model framework of the data recognition model, parameters of the training data set, and a batch size of the data recognition model;

the generating module 306 is configured to generate a data recognition model according to the model framework, the batch size and the initial learning rate, as a data recognition model matching with the preset data recognition requirement.

It can be seen that the device for recognizing data based on machine learning described in fig. 5 can acquire a training data set for a preset data recognition requirement based on machine learning, determine a model frame of the data recognition model according to the preset data recognition requirement and parameters of the training data set, determine a batch size of the data recognition model according to parameters of the training data set, determine an initial learning rate of the data recognition model according to a model frame and the batch size of the data recognition model, further generate the data recognition model, improve the frame structure of the data recognition model and the rationality of the parameters, further improve the accuracy of the data recognition model in recognizing data and the robustness of the data recognition model, input the training data set into the data recognition model which is matched with the preset data recognition requirement determined in advance, train the data recognition model to obtain a trained target data recognition model, improve the accuracy and reliability of the recognition of the target data recognition model on the data, improve the mobility of the target data recognition model, better adapt to related working environments, further improve the accuracy of the data recognition model on the target recognition, and the data recognition requirement of the target recognition model.

In another alternative embodiment, as shown in fig. 5, the apparatus for identifying data based on machine learning may further include:

a computing module 307 for inputting a validation data set into the target data recognition model, computing a set of performance parameters of the target data recognition model, the validation data set being of a type comprising a text type and/or an image type;

the adding module 308 is configured to add a weight coefficient to each performance parameter in the performance parameter set according to a data identification effect corresponding to a preset data identification requirement;

the calculating module 307 is further configured to calculate a comprehensive performance value of the target data recognition model according to the performance parameter set of the target data recognition model and the weight coefficient of each performance parameter in the performance parameter set;

the evaluation module 309 is configured to evaluate whether the comprehensive performance value of the target data identification model meets the standard according to a preset comprehensive performance threshold value, so as to obtain an evaluation result;

when the evaluation result indicates that the comprehensive performance value of the target data identification model meets the standard, the deployment module 303 is triggered to perform the operation of deploying the target data identification model into the preset framework.

Therefore, the device for recognizing data based on machine learning described in fig. 5 can calculate whether the target data recognition model meets the preset requirement according to the performance parameter set of the target data recognition model, and deploy the target data recognition model into the preset framework when the target data recognition model meets the preset requirement, so that the accuracy and reliability of data recognition of the target data recognition model are improved.

In yet another alternative embodiment, as shown in fig. 5, the apparatus for identifying data based on machine learning may further include:

a second determining module 310, configured to determine, as the verification data set, a data set matching the preset data identification requirement and the application scenario of the target data identification model according to the preset data identification requirement and the application scenario of the target data identification model before the computing module 307 inputs the verification data set into the target data identification model and computes the performance parameter set of the target data identification model;

the second determining module 310 is further configured to determine parameters of the verification data set, and determine a performance parameter type to be calculated according to the preset data identification requirement and the parameters of the verification data set, where the performance parameter type includes at least one of an accuracy rate, a recall rate, an F1 value, an ROC curve, an AUC value, and a confusion matrix;

wherein the computing module 307 inputs the validation data set into the target data recognition model, the specific manner of computing the performance parameter set of the target data recognition model includes:

Therefore, the device for recognizing data based on machine learning described in fig. 5 can determine the data set matched with the preset data recognition requirement and the application scene of the target data recognition model according to the preset data recognition requirement and the application scene of the target data recognition model, as the verification data set, determine the parameters of the verification data set, determine the performance parameter type to be calculated according to the preset data recognition requirement and the parameters of the verification data set, input the verification data set and the performance parameter type into the target data recognition model, calculate the parameter set matched with the performance parameter type, and serve as the performance parameter set of the target data recognition model, so that the determination accuracy of the performance parameter set of the target data recognition model is improved, the evaluation accuracy of the target data recognition model is further improved, the performance of the target data recognition model is ensured, and the recognition accuracy and response speed of the target data recognition model are improved.

the noise reduction module 311 is configured to, after the obtaining module obtains a training data set for a preset data identification requirement, perform noise reduction processing on data in the training data set according to the preset data identification requirement and parameters of the training data set, so as to obtain a first training data set;

A data enhancement module 312 for performing data enhancement operations on the first training data set to obtain a second training data set, the data enhancement operations including at least one of clipping, rotation, scaling, and translation;

the labeling module 313 is configured to label the second training data set according to a preset data identification requirement, so as to obtain a target training data set;

the training module 302 inputs the training data set into a predetermined data recognition model matched with a preset data recognition requirement, and trains the data recognition model, so as to obtain a trained target data recognition model in a specific manner comprising:

and inputting the target training data set into a data identification model which is determined in advance and matched with the preset data identification requirement, and training the data identification model to obtain a trained target data identification model.

Therefore, the device for recognizing data based on machine learning described in fig. 5 can perform data processing operations such as noise reduction, data enhancement and labeling on the training data set according to preset data recognition requirements and parameters of the training data set, so that the matching degree of data in the training data set in the training process of the target data recognition model and the richness, accuracy and reliability of the data in the training data set are improved, and the data recognition capability and accuracy of the target data recognition model can be improved.

a third determining module 314, configured to determine, when the evaluation result indicates that the comprehensive performance value of the target data identification model does not reach the standard, a scene parameter of an application scene required to be applied by the target data identification model, where the scene parameter of the application scene includes one or more of a data amount of data to be identified in the application scene, a data type of the data to be identified, a concurrency amount of the data to be identified, and a data feedback duration requirement of the data to be identified;

an analysis module 315, configured to analyze scene parameters of an application scene to obtain data identification requirement parameters matched with the application scene;

the third determining module 314 is further configured to determine a data type of the target data identification model;

the optimizing module 316 is configured to optimize the target data recognition model according to the data recognition requirement parameter and the data type of the target data recognition model, update the optimized target data recognition model to the target data recognition model, and trigger the deploying module 303 to perform the operation of deploying the target data recognition model into the preset framework.

Therefore, when the evaluation result indicates that the comprehensive performance value of the target data identification model does not reach the standard, the device for identifying the data based on the machine learning described in fig. 5 can optimize the target data identification model according to the data identification requirement parameter of the application scene of the target data identification model and the data type of the target data identification model, so that the data identification accuracy and reliability of the target data identification model can be ensured.

In yet another alternative embodiment, as shown in fig. 5, the optimizing module 316 optimizes the target data recognition model according to the data recognition requirement parameter and the data type of the target data recognition model, and the specific manner of updating the optimized target data recognition model to the target data recognition model includes:

Therefore, the device for recognizing data based on machine learning described in fig. 5 can determine the redundancy coefficient of each layer of the target data recognition model according to the data recognition requirement parameter, perform network pruning on the layers in the redundancy layer set according to the preset first redundancy coefficient threshold and the preset second redundancy coefficient threshold, perform model quantization on the target data recognition model, update the target data recognition model after the model quantization into the target data recognition model, and improve the calculation efficiency of the model and the data recognition effect of the model by simplifying the structure of the model and changing the data type of the model.

Example IV

Referring to fig. 6, fig. 6 is a schematic structural diagram of another apparatus for recognizing data based on machine learning according to an embodiment of the present invention. As shown in fig. 6, the apparatus for identifying data based on machine learning may include:

a memory 401 storing executable program codes;

a processor 402 coupled with the memory 401;

the processor 402 invokes executable program code stored in the memory 401 to perform the steps in the method for identifying data based on machine learning described in the first or second embodiment of the present invention.

Example five

The embodiment of the invention discloses a computer storage medium which stores computer instructions for executing the steps in the method for identifying data based on machine learning described in the first or second embodiment of the invention when the computer instructions are called.

Example six

An embodiment of the present invention discloses a computer program product comprising a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the method for identifying data based on machine learning described in the first or second embodiment.

The apparatus embodiments described above are merely illustrative, wherein the modules illustrated as separate components may or may not be physically separate, and the components shown as modules may or may not be physical, i.e., may be located in one place, or may be distributed over a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above detailed description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product that may be stored in a computer-readable storage medium including Read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), erasable programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), one-time programmable Read-Only Memory (OTPROM), electrically erasable programmable Read-Only Memory (EEPROM), compact disc Read-Only Memory (Compact Disc Read-Only Memory, CD-ROM) or other optical disc Memory, magnetic disc Memory, tape Memory, or any other medium that can be used for computer-readable carrying or storing data.

Finally, it should be noted that: the embodiment of the invention discloses a method and a device for identifying data based on machine learning, which are disclosed as preferred embodiments of the invention, and are only used for illustrating the technical scheme of the invention, but not limiting the technical scheme; although the invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that; the technical scheme recorded in the various embodiments can be modified or part of technical features in the technical scheme can be replaced equivalently; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.

Claims

1. A method of identifying data based on machine learning, the method comprising:

acquiring a training data set aiming at preset data identification requirements, wherein the type of the training data set comprises a text type and/or an image type;

Deploying the target data identification model into a preset framework;

2. The machine learning based data identification method of claim 1, further comprising:

determining a model framework of a data identification model according to the preset data identification requirement and parameters of the training data set, wherein the model framework comprises one of a convolutional neural network framework, a cyclic neural network framework and a transfer learning framework;

3. The machine learning based data identification method of claim 1 or 2, further comprising:

inputting a verification data set into the target data identification model, and calculating a performance parameter set of the target data identification model, wherein the type of the verification data set comprises a text type and/or an image type;

4. A method of machine learning based data identification as claimed in claim 3, wherein prior to said inputting a validation data set into said target data identification model, calculating a set of performance parameters for said target data identification model, said method further comprises:

5. The method of machine learning based data identification of claim 1 or 2, wherein after the acquiring of the training data set for preset data identification requirements, the method further comprises:

6. The method of machine learning based data identification of claim 3, wherein when the evaluation result indicates that the composite performance value of the target data identification model does not meet the standard, the method further comprises:

determining a data type of the target data identification model;

7. The machine learning based data recognition method of claim 6, wherein optimizing the target data recognition model based on the data recognition requirement parameter and the data type of the target data recognition model, and updating the optimized target data recognition model to the target data recognition model, comprises:

8. An apparatus for identifying data based on machine learning, the apparatus comprising:

9. An apparatus for identifying data based on machine learning, the apparatus comprising:

a memory storing executable program code;

a processor coupled to the memory;

the processor invokes the executable program code stored in the memory to perform the method of identifying data based on machine learning as claimed in any one of claims 1 to 7.

10. A computer storage medium storing computer instructions which, when invoked, are operable to perform a method of identifying data based on machine learning as claimed in any one of claims 1 to 7.