CN115878701A

CN115878701A - Target detection model training method and device based on ETL tool

Info

Publication number: CN115878701A
Application number: CN202211516257.3A
Authority: CN
Inventors: 黑俊铭; 蔺川
Original assignee: Inspur General Software Co Ltd
Current assignee: Inspur General Software Co Ltd
Priority date: 2022-11-30
Filing date: 2022-11-30
Publication date: 2023-03-31

Abstract

The invention provides a target detection model training method and device based on an ETL tool. The method comprises the following steps: when a json request sent by a back-end program of the ETL tool is received, acquiring model training information input by a user in the front-end program of the ETL tool from the json request; selecting a corresponding target detection algorithm according to the type of the target detection task, and setting a hyper-parameter of the target detection algorithm; acquiring path information from a training set table of a relational database accessed by a back-end program of the ETL tool according to the picture setting information, acquiring a training picture from a picture folder according to the path information and acquiring marking information corresponding to the training picture from a marking folder; and training the target detection algorithm with the set hyper-parameters according to the obtained training picture and the labeling information to obtain a target detection model. The invention provides a scheme for realizing target detection model training based on an ETL tool.

Description

Target detection model training method and device based on ETL tool

Technical Field

The invention relates to the technical field of deep learning target detection model training, in particular to a target detection model training method and device based on an ETL tool.

Background

When image processing tasks such as industrial part identification, industrial defect detection and the like are handled, because the traditional visual algorithm is low in identification speed and identification accuracy, the deep learning target detection technology based on the convolutional neural network is often adopted in the current industrial field. Target detection, which may also be referred to as target extraction, combines the tasks of positioning and identifying targets in an image, requiring real-time and accuracy.

The ETL tool is an English full name Extract-Transform-Load and is responsible for extracting various unevenly distributed and multi-source heterogeneous data, cleaning, converting and integrating the data according to a certain rule, and finally loading the data into a data warehouse, so that a data basis is provided for subsequent data analysis and intelligent decision. However, no scheme for realizing target detection model training based on the ETL tool exists at present.

Disclosure of Invention

In view of at least one of the above technical problems, embodiments of the present invention provide a method and an apparatus for training a target detection model based on an ETL tool.

According to a first aspect, in the target detection model training method based on the ETL tool provided in the embodiments of the present invention, a backend program of the ETL tool is installed in an application server, an image folder and a labeling folder are stored in the application server, and labeling information of each image in the image folder is stored in the labeling folder; a back-end program of the ETL tool is accessed into a relational database, a training set table and a verification set table are stored in the relational database, a plurality of field information of pictures are stored in the training set table and the verification set table, the plurality of field information comprises path field information, and the path field information is path information for storing the pictures and the labeled information;

the method is performed by the application server, the method comprising:

when a json request sent by a back-end program of the ETL tool is received, acquiring model training information input by a user in a front-end program of the ETL tool from the json request; the model training information comprises the type of a target detection algorithm, the hyper-parameter of the target detection algorithm and picture setting information;

selecting a target detection algorithm of a corresponding type according to the type of the target detection algorithm, and carrying out hyper-parameter setting on the target detection algorithm according to the hyper-parameter;

acquiring path information from a training set table of a relational database accessed by a back-end program of the ETL tool according to the picture setting information, acquiring a training picture from the picture folder according to the path information and acquiring marking information corresponding to the training picture from the marking folder;

and training the target detection algorithm with the set hyper-parameters according to the obtained training picture and the obtained labeling information to obtain a target detection model.

According to a second aspect, in the target detection model training apparatus based on the ETL tool provided in the embodiments of the present invention, a backend program of the ETL tool is installed in an application server, an image folder and a labeling folder are stored in the application server, and labeling information of each image in the image folder is stored in the labeling folder; a back-end program of the ETL tool is accessed into a relational database, a training set table and a verification set table are stored in the relational database, a plurality of field information of pictures are stored in the training set table and the verification set table, the plurality of field information comprises path field information, and the path field information is path information for storing the pictures and the labeled information;

the training device is deployed on the application server, the training device comprising:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring model training information input by a user in a front-end program of the ETL tool from a json request when the json request sent by the back-end program of the ETL tool is received; the model training information comprises the type of a target detection algorithm, the hyper-parameters of the target detection algorithm and picture setting information;

the first setting module is used for selecting a target detection algorithm of a corresponding type according to the type of the target detection algorithm and carrying out hyper-parameter setting on the target detection algorithm according to the hyper-parameter;

the second acquisition module is used for acquiring path information from a training set table of a relational database accessed by a back-end program of the ETL tool according to the picture setting information, acquiring a training picture from the picture folder according to the path information and acquiring marking information corresponding to the training picture from the marking folder;

and the model training module is used for training the target detection algorithm with the set hyper-parameters according to the obtained training picture and the labeling information to obtain a target detection model.

According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when executed in a computer, causes the computer to execute a method for implementing the method provided by the first aspect.

According to a fourth aspect, the computing device provided by the embodiment of the present invention includes a memory and a processor, where the memory stores executable codes, and the processor executes the executable codes to implement the method provided by the first aspect.

The target detection model training method and device based on the ETL tool provided by the embodiment of the invention have the following beneficial effects in each or the combination:

(1) The method comprises the steps that model training information is input by a user in a front-end program of an ETL tool, a json request is generated by a back-end program of the ETL tool according to the model training information set by the user, then the json request is sent to an application server, the application server obtains the model training information from the json request, then a corresponding target detection algorithm is selected according to the type in the model training information, the hyperparameter of the target detection algorithm is set according to the hyperparameter in the model training information, then path information is obtained from a training set table of a relational database accessed by the back-end program of the ETL tool according to picture setting information in the model training information, and then a training picture is obtained from a picture folder according to the path information and annotation information corresponding to the training picture is obtained from the annotation folder; and finally, training the target detection algorithm with the set hyper-parameters according to the obtained training picture and the labeling information to obtain a target detection model. Therefore, the embodiment of the invention provides a scheme for realizing target detection model training based on an ETL tool. In the process, model training is realized according to the ETL tool, a back-end program of the ETL tool is accessed into a relational database, and path information of pictures is stored in the relational database, so that the ETL tool supports accessing picture data, and can be used for training a target detection model aiming at the pictures.

(2) In one embodiment, the target detection model is evaluated according to three parameters of recall rate, precision and average accuracy, so that the evaluation of the target detection model is more comprehensive. Moreover, the user can pay more attention to the result of a certain parameter by setting the weight according to the preference or task requirement of the user, so that the trained target detection model can better meet the preference or task requirement of the user.

(3) In one embodiment, the user can preview the pictures and the annotation information through the front-end program of the ETL tool.

Drawings

FIG. 1 is a schematic deployment diagram of a system architecture according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart illustrating a target detection model training method based on an ETL tool according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a P-R curve in accordance with an embodiment of the present invention.

Detailed Description

In a first aspect, an embodiment of the present invention provides a target detection model training method based on an ETL tool, and referring to fig. 1, a backend program of the ETL tool is installed in an application server, a picture folder and a markup folder are stored in the application server, and markup information of each picture in the picture folder is stored in the markup folder; the back-end program of the ETL tool is accessed into a relational database, a training set table and a verification set table are stored in the relational database, a plurality of pieces of field information of pictures are stored in the training set table and the verification set table, the plurality of pieces of field information comprise path field information, and the path field information is path information for storing the pictures and marking information.

In fig. 1, the ETL software application is a backend program of the ETL tool, the database is a relational database, and the image data sets are picture folders and markup folders.

That is, the application server stores a backend program of the ETL tool, a picture folder, and a markup folder. The image folder stores a large number of images including training images and verification images, and the labeling folder stores labeling information of each image. The markup folder may be a folder in XML format.

The back-end program of the ETL tool is accessed into a relational database, a training set table and a verification set table are stored in the relational database, the training set table comprises a plurality of pieces of field information, such as main key field information, host name field information and path field information, wherein the main key field information is used for representing the serial number of a training picture, the host name field information is used for representing the host name of an application server, and the path field information is used for representing each training picture and the storage path of corresponding label information. The verification set table also includes a plurality of fields, for example, a primary key field, a host name field, and a path field, wherein the primary key field is used to represent the serial number of the verification picture, the host name field is used to represent the host name of the application server, and the path field is used to represent the storage path of each verification picture and the corresponding label information.

The relational database can be deployed on a host different from the application server, and the application server can access the relational database through an IP + port.

The application server is configured with a backend program of the ETL tool, and thus the application server may be referred to as an ETL application server. The memory of the application server is not lower than 16GB, an X86 architecture multi-core processor is adopted, the storage capacity is not lower than 600GB, the main frequency is more than 3.2GHz, and the GTX 2080ti display card with 11GB memory is provided. The application server is internally provided with a python algorithm service for calling when the Java program sends a request. The ETL tool can be programmed in the python language, so the application server can also be called python algorithm application server.

The back-end program of the ETL tool supports access of multiple types of data sources, such as a relational data source, a manual data source and a sensing data source; the relational data source supports management and inspection of objects such as entity tables, code tables, stored procedures, functions, views and the like. The manual data source supports Excel data access. The sensing data source is connected with an mqtt protocol, and equipment data are accessed and persisted.

In the embodiment of the present invention, a relational data source, i.e., the above-mentioned relational database, is selected.

The method provided by the embodiment of the present invention is executed by the application server, and specifically includes the following steps S110 to S140, referring to fig. 2:

s110, when a json request sent by a back-end program of the ETL tool is received, obtaining model training information input by a user in a front-end program of the ETL tool from the json request; the model training information comprises the type of a target detection algorithm, the hyper-parameter of the target detection algorithm and picture setting information;

that is, the user inputs model training information in the front-end program of the ETL tool, then the front-end program of the ETL tool sends the model training information to the back-end program of the ETL tool, the back-end program of the ETL tool performs certain processing on the received model training information to form a json request, and then the json request is sent to the application server. And when the application server receives the json request, analyzing the json request to obtain model training information.

JSON (JavaScript Object Notation) is a lightweight data exchange format.

Wherein ETL is an abbreviation of Extract-Transform-Load in english, and is used to describe the process of extracting (Extract), converting (Transform), and loading (Load) data from a source end to a destination end. The term ETL is more commonly used in data warehouses, but its objects are not limited to data warehouses.

The model training information comprises the type of the target detection algorithm, the hyper-parameter of the target detection algorithm and picture setting information.

The types of the target detection algorithm may include a single-stage target detection algorithm and a dual-stage target detection algorithm. The user may specify a type in the model component of the front-end program of the ETL tool.

It will be appreciated that the task of the object detection model is to locate all objects of interest in the picture and to determine the category to which they belong. Different detection targets have large differences in form, color and appearance, and the targets are shielded by other objects, so that the target detection task is challenging. The target detection algorithm comprises two steps of classification and positioning, wherein the classification of a target in a picture is judged firstly, and then the position of the target is obtained.

The idea of the single-stage target detection algorithm is to directly perform one-step regression processing on an input picture, namely, two tasks of classification and positioning are completed simultaneously, and the idea of the double-stage target detection algorithm is to firstly generate a candidate region in an image and then classify the candidate region. The single-stage target detection algorithm has real-time performance, while the double-stage target detection algorithm has low detection speed and higher detection accuracy, and the industry generally adopts the single-target algorithm with stronger real-time performance in combination with an optimization technology to obtain the detection accuracy.

The single-stage target detection algorithm is represented by a Yolo series algorithm which is called You Only Look one, and the meaning of the naming is that the algorithm can obtain the position and the category information of the target to be detected Only by processing the input picture Once. The detection speed of YOLOv1 is very fast, and the method has strong feature extraction capability, but the positioning accuracy of YOLOv1 is very low. To further improve the detection efficiency, YOLOv2 and YOLOv3 were developed in sequence. The YOLOv2 removes the original full connection layer in the YOLOv1, and uses the operation of setting an anchor frame for reference, so as to play a guiding role in the regression of the boundary frame, and adopts Batch standardization (namely Batch normalization) algorithm, thereby improving the convergence rate of loss. A new trunk network named Darknet53 is proposed in YOLOv3, and is combined with a feature pyramid structure to realize feature fusion, so that features can be extracted from feature maps with different sizes, the sizes of the three feature maps in an algorithm are 13 × 13, 26 × 26 and 52 × 52 in sequence, wherein small feature maps are spliced with large feature maps to realize further prediction, and the detection precision of small targets is improved by the further operation.

Among these, the hyper-parameters of the target detection algorithm, such as learning rate, optimizer parameters, etc.

The picture setting information may include a picture selection mode and/or a picture number; the picture selection mode comprises a sequential selection mode and a random selection mode. The sequential selection mode is to select pictures according to their serial numbers, and the random selection mode is to select pictures randomly. The number of pictures is the number of training pictures used in model training.

S120, selecting a target detection algorithm of a corresponding type according to the type of the target detection algorithm, and carrying out hyper-parameter setting on the target detection algorithm according to the hyper-parameter;

namely, based on the type of the target detection algorithm selected by the user, the application server selects the target detection algorithm of the type, and builds a frame of the target detection model by using the target detection algorithm of the type. And then setting the hyper-parameters in the frame based on the hyper-parameters set by the user. After the super parameters are set, the training pictures and the labeling information can be selected according to S130 and input into the frame, and the parameters of the frame are trained.

S130, acquiring path information from a training set table of a relational database accessed by a back-end program of the ETL tool according to the picture setting information, acquiring a training picture from the picture folder according to the path information, and acquiring annotation information corresponding to the training picture from the annotation folder;

that is, based on the picture setting information set by the user, the path information of the training picture and the path information of the label information of the training picture are obtained from the training set table of the relational database through the back-end program of the ETL tool, then the training picture is obtained from the picture folder according to the path information of the training picture, and the label information is obtained from the label folder according to the path information of the label information.

S140, training the target detection algorithm with the set hyper-parameters according to the obtained training picture and the obtained labeling information to obtain a target detection model.

The target detection algorithm with the set hyper-parameters is trained through the training pictures and the labeling information, and specific values of the parameters to be determined in the target detection algorithm are obtained, so that the target detection model is obtained.

In one embodiment, S140 specifically includes:

s141, after each training is finished, obtaining path information from a verification set table of a relational database accessed by a back-end program of the ETL tool, obtaining a verification picture from the picture folder according to the path information and obtaining marking information corresponding to the verification picture from the marking folder;

s142, verifying the target detection model obtained by the training according to the obtained verification picture and the obtained labeling information to obtain multiple recall rates and multiple accuracies of the target detection model obtained by the training;

and S143, evaluating the target detection model obtained by the training according to the plurality of recall rates and the plurality of accuracies, and returning an evaluation result to a front-end program of the ETL tool so that a person can determine to terminate the training or continue the training.

After the training is completed, the target detection model obtained by the training picture and the corresponding label information needs to be verified through the verification picture and the corresponding label information. Namely, the verification picture is input into the target detection model, the target detection model outputs a detection result, the detection result is compared with the labeling information, and then the recall rate and the accuracy of the target detection model are improved. And evaluating the target detection model according to the recall rate and the accuracy to obtain an evaluation result, and then feeding the evaluation result back to a front-end program of the ETL tool, so that personnel can know the training condition of the model, and if the training condition meets the requirements of the personnel, the training is stopped. And if the training condition does not meet the requirement of the user, continuing training and further adjusting the parameter value.

After the model training, a plurality of recall rates and a plurality of accuracies are obtained, because in the model training process, a set of confidence threshold values are set for each parameter to be determined of the target detection model, and one recall rate and accuracy are obtained under each confidence threshold value. Specifically, the target detection model may be evaluated by an average of the plurality of recall rates and an average of the plurality of accuracies.

The accuracy is the proportion of positive samples judged to be positive in all samples judged to be positive, which reflects the detection capability of the target detection algorithm on positive samples, namely how many samples which are found by the target detection algorithm and are considered to be real frames are real frames. The recall rate is how many real frames are detected, and the detection capability of the algorithm on the real frames is reflected.

Wherein the higher the accuracy, the better the performance of the target detection model. The higher the recall rate, the better the performance of the target detection model.

Further, in S143, according to the plurality of recall rates and the plurality of accuracies, evaluating the target detection model obtained by the training, which may specifically include:

a1, forming an average accuracy curve according to the plurality of recall rates and the plurality of accuracies; the horizontal axis coordinate of the average accuracy curve is recall rate, the vertical axis coordinate of the average accuracy curve is accuracy, and the area formed by the average accuracy curve and the first quadrant of the coordinate system is average accuracy;

and A2, evaluating the target detection model obtained by the training according to the plurality of recall rates, the plurality of accuracies and the average accuracy.

It can be understood that the ideal situation is that the target detection model has high accuracy and recall rate at the same time, but the real situation is that the target detection model and the recall rate are difficult to be combined, and personnel can only make a balance according to the characteristics of the actual detection task of the personnel to select a preference. The Precision and the Recall rate Recall are considered simultaneously, namely the average Precision AP, the average Precision is obtained by the area enclosed by the P-R curve, namely the Precision-Recall curve, and the first quadrant of the coordinate system, as shown in FIG. 3, and the curve reflects the change process of the Precision and the Recall rate Recall under different confidence thresholds.

Wherein the higher the average accuracy, the better the performance of the target detection model.

Wherein, a plurality of recall rates can be averaged, a plurality of accuracies can be averaged, and the target detection model is evaluated jointly by two averages and one average accuracy. For example, weight values are set for three values, and the three values are weighted and summed to obtain a final evaluation result. The higher the evaluation result, the better the performance of the target detection model.

In one embodiment, a real-time evaluation result table may be further stored in the relational database; correspondingly, the method provided by the embodiment of the invention can further comprise the following steps: and sending the evaluation result of the target detection model obtained by each training to a real-time evaluation result table in the relational database for storage.

That is, after each training, the evaluation result of the target detection model is sent to the real-time evaluation result table in the relational database for storage, so that each model training process can be recorded, and the front-end program of the ETL tool accesses the back-end program, so that the evaluation result in the relational database is obtained, and the performance change of the model after each training is known.

In one embodiment, the front end program of the ETL tool may be used to: the method comprises the steps that a back-end program of the ETL tool obtains a training picture and path information of corresponding marking information from a relational database, the back-end program of the ETL tool obtains the training picture and the marking information from a picture folder and a marking information folder according to the path information, and the back-end program of the ETL tool feeds the training picture and the marking information back to the front-end program, so that personnel can preview the training picture and the marking information.

That is to say, a person can call the matplotl ib interface in the front-end program of the ETL through the back-end program to obtain path information of the training pictures and the labeling information in the relational database, then the back-end program obtains the training pictures and the labeling information from the picture folder and the labeling information folder according to the path information, and the back-end program feeds the training pictures and the labeling information back to the front-end program of the ETL, so that the person can browse and view the training pictures and the labeling information from a front-end page.

In one embodiment, the pictures in the picture folder are pictures in a public data set.

It will be appreciated that the selection of the data set is important in the training of the object detection model. When constructing a data set, attention is paid to cleaning and labeling of data, and a high-quality data set can improve the quality of model training and the accuracy of prediction. In the absence of data, attempts may be made to find some public data sets, particularly data sets that are recognized as being commonly used. For common tasks, such as: the task aspects of image recognition, object detection and image segmentation all have corresponding public data sets available.

It can be appreciated that the backend program of the ETL tool is equivalent to the controller role in MVC. M refers to a business model, V refers to a user interface, and C is a controller. The main functions include: the method call entry and the main business processing method receive the field parameters transmitted in the previous step, process the field configuration of the step, and transmit the result to the next component after merging. And receiving the field parameters transmitted in the previous step, for example, receiving model training information sent by the front-end program, processing the information, for example, performing certain processing on the model training information to form a json request, and transmitting the json request to the application server.

In summary, the embodiment of the present invention provides a scheme for implementing target detection model training based on an ETL tool. In the process, model training is realized according to the ETL tool, a back-end program of the ETL tool is accessed into a relational database, and path information of pictures is stored in the relational database, so that the ETL tool supports accessing picture data, and can be used for training a target detection model aiming at the pictures.

In a second aspect, an embodiment of the present invention provides an ETL tool-based target detection model training apparatus, where a backend program of the ETL tool is installed in an application server, a picture folder and a markup folder are stored in the application server, and markup information of each picture in the picture folder is stored in the markup folder; a back-end program of the ETL tool is accessed into a relational database, a training set table and a verification set table are stored in the relational database, a plurality of field information of pictures are stored in the training set table and the verification set table, the plurality of field information comprises path field information, and the path field information is path information for storing the pictures and the labeled information;

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring model training information input by a user in a front-end program of the ETL tool from a json request when the json request sent by the rear-end program of the ETL tool is received; the model training information comprises the type of a target detection algorithm, the hyper-parameter of the target detection algorithm and picture setting information;

In one embodiment, the picture setting information includes a picture selection mode and/or a number of pictures; the picture selection mode comprises a sequential selection mode and a random selection mode.

In one embodiment, the model training module comprises:

the system comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring path information from a verification set table of a relational database accessed by a back-end program of the ETL tool after each training is finished, acquiring a verification picture from a picture folder according to the path information and acquiring marking information corresponding to the verification picture from a marking folder;

the first verification unit is used for verifying the target detection model obtained by the training according to the obtained verification picture and the obtained labeling information to obtain a plurality of recall rates and a plurality of accuracies of the target detection model obtained by the training;

and the first evaluation unit is used for evaluating the target detection model obtained by the training according to the plurality of recall rates and the plurality of accuracies, and returning an evaluation result to a front-end program of the ETL tool so that a person can determine to terminate the training or continue the training.

Further, the first evaluation unit includes:

a curve forming subunit, configured to form an average accuracy curve according to the plurality of recall rates and the plurality of accuracies; the horizontal axis coordinate of the average accuracy curve is recall rate, the vertical axis coordinate of the average accuracy curve is accuracy, and the area formed by the average accuracy curve and the first quadrant of the coordinate system is average accuracy;

and the model evaluation subunit is used for evaluating the target detection model obtained by the training according to the plurality of recall rates, the plurality of accuracies and the average accuracy.

In one embodiment, the types of object detection algorithms include a single stage object detection algorithm and a dual stage object detection algorithm.

In one embodiment, the front end program of the ETL tool is further to: and acquiring training pictures and corresponding path information of the labeled information from the relational database through a back-end program of the ETL tool, and acquiring the training pictures and the labeled information from a picture folder and a labeled information folder according to the path information so as to enable personnel to preview the training pictures and the labeled information.

In one embodiment, a real-time evaluation result table is further stored in the relational database; correspondingly, the device further comprises:

and the evaluation recording module is used for sending the evaluation result of the target detection model obtained by each training to the real-time evaluation result table in the relational database for storage.

In one embodiment, the plurality of pieces of field information further include primary key field information and host name field information, the primary key field information is a picture sequence number, and the host name field information is a host name of the application server.

It is to be understood that for the explanation, the detailed description, the beneficial effects, the examples and the like of the contents in the computer-readable medium provided in the embodiment of the present invention, reference may be made to the corresponding parts in the method provided in the first aspect, and details are not described here.

In a third aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored, and when executed by a processor, the computer instructions cause the processor to execute the method provided in the first aspect.

Specifically, a system or an apparatus equipped with a storage medium on which software program codes that realize the functions of any of the embodiments described above are stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program codes stored in the storage medium.

In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

Examples of the storage medium for supplying the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer via a communications network.

Further, it should be clear that the functions of any one of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform a part or all of the actual operations based on instructions of the program code.

Further, it is to be understood that the program code read out from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then causes a CPU or the like mounted on the expansion board or the expansion module to perform part or all of the actual operations based on instructions of the program code, thereby realizing the functions of any of the above-described embodiments.

It is to be understood that, for the explanation, the detailed description, the beneficial effects, the examples and the like of the contents in the computer-readable medium provided in the embodiment of the present invention, reference may be made to the corresponding parts in the method provided in the first aspect, and details are not described herein again.

In a fourth aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, where the memory stores executable code, and the processor executes the executable code to implement a method in any embodiment of the specification.

All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

Those skilled in the art will recognize that the functionality described in this disclosure may be implemented in hardware, software, firmware, or any combination thereof, in one or more of the examples described above. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.

The above-mentioned embodiments, objects, technical solutions and advantages of the present invention are further described in detail, it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made on the basis of the technical solutions of the present invention should be included in the scope of the present invention.

Claims

1. A target detection model training method based on an ETL tool is characterized in that a back-end program of the ETL tool is installed in an application server, a picture folder and a labeling folder are stored in the application server, and labeling information of each picture in the picture folder is stored in the labeling folder; a back-end program of the ETL tool is accessed into a relational database, a training set table and a verification set table are stored in the relational database, a plurality of field information of pictures are stored in the training set table and the verification set table, the plurality of field information comprises path field information, and the path field information is path information for storing the pictures and the labeled information;

the method is performed by the application server, the method comprising:

2. The method according to claim 1, wherein the picture setting information includes picture selection manner and/or number of pictures; the picture selection mode comprises a sequential selection mode and a random selection mode.

3. The method according to claim 1, wherein the training a target detection algorithm with well-set hyper-parameters according to the obtained training picture and the labeling information to obtain a target detection model comprises:

after each training, acquiring path information from a verification set table of a relational database accessed by a back-end program of the ETL tool, and acquiring a verification picture from the picture folder and label information corresponding to the verification picture from the label folder according to the path information;

verifying the target detection model obtained by the training according to the obtained verification picture and the obtained labeling information to obtain a plurality of recall rates and a plurality of accuracies of the target detection model obtained by the training;

and evaluating the target detection model obtained by the training according to the plurality of recall rates and the plurality of accuracies, and returning an evaluation result to a front-end program of the ETL tool so that a person can determine to terminate the training or continue the training.

4. The method of claim 3, wherein the evaluating the trained object detection model according to the plurality of recall rates and the plurality of accuracies comprises:

forming an average accuracy curve according to the plurality of recall rates and the plurality of accuracies; the horizontal axis coordinate of the average accuracy curve is recall rate, the vertical axis coordinate of the average accuracy curve is accuracy, and the area formed by the average accuracy curve and the first quadrant of the coordinate system is average accuracy;

and evaluating the target detection model obtained by the training according to the plurality of recall rates, the plurality of accuracies and the average accuracy.

5. The method of claim 1, wherein the types of target detection algorithms include a single stage target detection algorithm and a dual stage target detection algorithm.

6. The method of claim 1, wherein the front end program of the ETL tool is further configured to: the method comprises the steps that path information of training pictures and corresponding marking information is obtained from a relational database through a back-end program of an ETL tool, the training pictures and the marking information are obtained from a picture folder and a marking information folder through the back-end program of the ETL tool according to the path information, and the training pictures and the marking information are fed back to a front-end program through the back-end program of the ETL tool, so that personnel can preview the training pictures and the marking information.

7. The method according to claim 1, wherein a real-time evaluation result table is further stored in the relational database; correspondingly, the method further comprises the following steps: and sending the evaluation result of the target detection model obtained by each training to a real-time evaluation result table in the relational database for storage.

8. The method according to claim 1, wherein the plurality of field information further includes primary key field information and host name field information, the primary key field information is a picture sequence number, and the host name field information is a host name of the application server.

9. The method of claim 1, wherein the pictures in the picture folder are pictures in a public data set.

10. A target detection model training device based on an ETL tool is characterized in that a back-end program of the ETL tool is installed in an application server, a picture folder and a labeling folder are stored in the application server, and labeling information of each picture in the picture folder is stored in the labeling folder; a back-end program of the ETL tool is accessed into a relational database, a training set table and a verification set table are stored in the relational database, a plurality of field information of pictures are stored in the training set table and the verification set table, the plurality of field information comprises path field information, and the path field information is path information for storing the pictures and the labeled information;

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring model training information input by a user in a front-end program of the ETL tool from a json request when the json request sent by the back-end program of the ETL tool is received; the model training information comprises the type of a target detection algorithm, the hyper-parameter of the target detection algorithm and picture setting information;