WO2021103835A1

WO2021103835A1 - Training sample acquisition method and apparatus

Info

Publication number: WO2021103835A1
Application number: PCT/CN2020/121356
Authority: WO
Inventors: 马贤忠; 胡皓瑜; 江浩; 董维山; 任少卿; 范一磊
Original assignee: 初速度（苏州）科技有限公司
Priority date: 2019-11-28
Filing date: 2020-10-16
Publication date: 2021-06-03
Also published as: CN112861898B; CN112861898A

Abstract

Disclosed are a training sample acquisition method and apparatus. The method comprises: receiving a training sample acquisition task, wherein the acquisition task includes the number of training samples to be acquired, and the proportion of the training samples in a whole data set; acquiring a cost unit price corresponding to each data collection mode; acquiring a screening performance parameter and cost unit price corresponding to each sizer mode; acquiring a storage cost of unit data and a unit label cost; determining combination modes for original data acquisition; with regard to each of the combination modes, on the basis of the cost unit price corresponding to the data collection mode and the screening performance parameter and cost unit price corresponding to the sizer mode, performing calculation to obtain the total cost corresponding to each of the combination modes; selecting the minimum total cost from the obtained various total costs, and determining a combination mode corresponding to the minimum total cost as an original data acquisition combination scheme of the acquisition task acquiring the training samples; and sending the combination scheme to a corresponding device, so as to acquire an original data set for screening the training samples.

Description

Method and device for acquiring training samples

Technical field

The present invention relates to the field of automatic driving, and in particular to a method and device for obtaining training samples.

Background technique

In the field of autonomous driving, in the iterative training of data-driven algorithms represented by deep learning, with the explosive growth of data volume, necessary screening from massive data has become an important step in data preprocessing. However, the current focus is mainly on designing filter algorithms, and there is a lack of a scientific evaluation system for comparing different filters. In actual use, the pros and cons of filters are often measured by comparing algorithm evaluation indicators such as precision and recall; or decision makers and developers subjectively decide the choice and choice of filters based on experience.

However, simply using algorithm performance evaluation indicators (such as precision and recall) can only compare the pros and cons of the screening algorithm, ignoring that some filters can achieve high precision and recall, but the calculation process It is complicated and takes up a lot of resources, and may not be the best choice in actual application scenarios. However, subjective judgments made by developers based on experience lack objective scientific basis, and it is difficult to choose when multiple people have disagreements.

In practical applications, cost control occupies a very important position in each production link. Therefore, calculating the total cost of the combination of various data collection modes and filter modes and comparing them, and selecting the most suitable combination to obtain training samples has become an urgent problem to be solved.

Summary of the invention

The present invention provides a method and device for obtaining training samples. By calculating the total cost of the data collection mode and the filter mode, a scientific evaluation system is established, and human experience is avoided when the performance of the filter algorithm and the efficiency of the algorithm are taken into account. Subjectivity. The specific technical scheme is as follows.

In the first aspect, the present invention provides a universal filter selection method, including:

Receiving an acquisition task of training samples, where the acquisition task includes the number of training samples to be acquired and the proportion of the training samples in the overall data set;

Obtain the cost unit price corresponding to each data collection mode; wherein, the data collection mode includes special collection and crowdsourcing collection, the cost unit price of the special collection mode includes the unit time collection cost, and the cost unit price of the crowdsourcing collection mode includes Transmission cost of each sample, depreciation cost of equipment installed on crowdsourced vehicles;

Obtain the screening performance parameters and cost unit price corresponding to each filter mode; wherein, the filter modes include uniform sampling, manual screening, car-side filter, cloud filter, and combined filter, and the uniform sampling mode corresponds to the screening The performance parameters include the time interval of uniform sampling, the screening performance parameters corresponding to the manual screening mode include manual screening speed, the cost unit price corresponding to the manual screening mode includes the manual screening cost per unit time, and the screening corresponding to the car-side filter mode The performance parameters include the precision rate of the car-side filter, the recall rate of the car-side filter, and the speed of the car-side filter. The screening performance parameters corresponding to the cloud filter mode include the precision rate of the cloud filter and the cloud filter The recall rate and cloud filter speed of the cloud filter mode include the operating cost of cloud computing resources per unit time; the manual filtering speed refers to the number of data manually processed per unit time, the vehicle end The filter speed refers to the number of data processed by the vehicle-side filter per unit time, and the cloud filter speed refers to the number of data processed by the cloud filter per unit time;

The storage cost of acquiring unit data and the unit labeling cost of manually labeling a training sample;

Determine a combination mode of raw data acquisition, the combination mode includes a data collection mode and a filter mode;

For each combination method, based on the cost unit price corresponding to the data collection mode of the combination method and the screening performance parameters and cost unit price corresponding to the filter mode, the total cost corresponding to the training sample obtained in the data set by this combination method is calculated ；

Select the lowest total cost from each total cost obtained, and determine the combination of the data collection mode and the filter mode corresponding to the lowest total cost as the original data acquisition combination plan for the acquisition task to acquire training samples;

The data acquisition combination scheme is sent to the corresponding device to acquire the original data set used to screen the training samples.

Optionally, when the combination of the data collection mode and the filter mode is special collection and uniform sampling, the total cost calculation formula corresponding to the combination is:

Where C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the total data set, T is the time interval of the uniform sampling, and c _collection is the unit time collection Cost, c _store is the storage cost of the unit data, and c _label is the unit label cost.

Optionally, when the combination of the data collection mode and the filter mode is special collection and car-side filter, the total cost calculation formula corresponding to the combination is:

Where C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, R _car is the recall rate of the car-end filter, and f _car is the total cost. The speed _{of the car-end filter, P car} is the precision rate of the car-end filter, c _collection is the collection cost per unit time, c _store is the storage cost of the unit data, and c _label is the unit labeling cost .

Optionally, when the combination of the data collection mode and the filter mode is special collection and cloud filter, the total cost calculation formula corresponding to the combination is:

Wherein, C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the total data set, T is the time interval of the uniform sampling, and R _cloud is the cloud filter F _cloud is the speed of the cloud filter, P _cloud is the precision rate of the cloud filter, c _collection is the collection cost per unit time, c _store is the storage cost of the unit data, c _resource is the operating cost of the cloud computing resource per unit time, and c _label is the label cost per unit.

Optionally, when the combination of the data collection mode and the filter mode is a special collection and a first combined filter, wherein the first combined filter is a combination of uniform sampling and manual screening, and the total cost of the combined mode is The calculation formula is:

Where C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, T is the time interval of the uniform sampling, and f _person is the manual screening speed , C _collection is the collection cost per unit time, c _store is the storage cost of the unit data, c _person is the manual screening cost per unit time, and c _label is the unit labeling cost.

Optionally, when the combination of the data collection mode and the filter mode is a special collection and a second combined filter, the second combined filter is a combination of a cloud filter and a manual filter, and the combined mode corresponds to the total The cost calculation formula is:

Wherein, C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the total data set, T is the time interval of the uniform sampling, and R _cloud is the cloud filter F _cloud is the speed of the cloud filter, P _cloud is the precision rate of the cloud filter, f _person is the manual screening speed, c _collection is the collection cost per unit time, and c _store is For the storage cost of the unit data, c _resource is the operating cost of the cloud computing resource per unit time, c _person is the manual screening cost per unit time, and c _label is the unit labeling cost.

Optionally, when the combination of the data collection mode and the filter mode is a special collection and a third combined filter, the third combined filter is a combination of a car-side filter, a cloud filter, and manual screening. The total cost calculation formula corresponding to the combination method is:

Among them, C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, R _car is the recall rate of the _car -end filter, and f car is the car-end screening P _car is the precision rate of the car filter, R _cloud is the recall rate _{of the cloud filter, f cloud} is the cloud filter speed, P _cloud is the precision rate of the cloud filter, and f _person is the manual filter Speed, c _collection is the collection cost per unit time, c _store is the storage cost per unit data, c _resource is the operating cost of cloud computing resources per unit time, c _person is the manual screening cost per unit time, and c _label is the unit labeling cost.

Optionally, when the combination of the data collection mode and the filter mode is crowdsourced collection and the second combined filter, the total cost calculation formula corresponding to the combination is:

Where C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, R _cloud is the recall rate of the cloud filter, and f _cloud is the Cloud filter speed, P _cloud is the accuracy rate of the cloud filter, f _person is the manual screening speed, c _netword is the traffic cost of transmitting each sample, c _store is the storage cost of unit data, and c _resource is the cloud computing per unit time Resource operating cost, c _person is the manual screening cost per unit time, and c _label is the unit labeling cost.

Optionally, when the combination of the data collection mode and the filter mode is crowdsourced collection and the third combined filter, the total cost calculation formula corresponding to the combination is:

Where C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, R _car is the recall rate of the car-end filter, and f _car is the total cost. The car-side filter speed, P _car is the precision rate of the car-side filter, R _cloud is the recall rate _{of the cloud filter, f cloud} is the cloud filter speed, and P _cloud is the cloud The precision rate _{of the filter, f person} is the manual screening speed, c _device is the depreciation cost of the equipment installed on the crowdsourced vehicle, c _netword is the traffic cost of transmitting each sample, and c _store is the cost The storage cost of the unit data, c _resource is the operating cost of the cloud computing resource per unit time, c _person is the manual screening cost per unit time, and c _label is the unit labeling cost.

In the second aspect, the present invention provides a device for acquiring training samples, including:

A receiving module configured to receive an acquisition task of training samples, the acquisition task including the number of training samples to be acquired and the proportion of the training samples in the overall data set;

The acquisition cost acquisition module is configured to acquire the cost unit price corresponding to each data acquisition mode;

The screening performance parameter and cost acquisition module is configured to obtain the screening performance parameter and cost unit price corresponding to each filter mode;

The storage and labeling cost acquisition module is configured to acquire the storage cost of unit data and the unit labeling cost of manually labeling a training sample;

The determining module is configured to initially determine a combination mode of data acquisition, and the combination mode includes a data collection mode and a filter mode;

The calculation module is configured to, for each combination mode, based on the cost unit price corresponding to the data collection mode of the combination mode and the screening performance parameter and cost unit price corresponding to the filter mode, calculate the combination mode to obtain the training in the data set The total cost corresponding to the sample;

The comparison module is configured to select the lowest total cost from each total cost obtained, and determine the combination of the data collection mode and the filter mode corresponding to the lowest total cost as the raw data of the acquisition task to obtain training samples Get a combination plan;

The sending module is configured to send the data acquisition combination scheme to the corresponding device to acquire the original data set used to filter the training samples.

The beneficial effects of the embodiments of the present invention are as follows:

When the acquisition task of training samples is received, the number of training samples to be acquired and the proportion of training samples in the total data set are collected from the acquisition task, and the cost unit price corresponding to all data collection modes and the cost unit price corresponding to the screening mode are obtained. And screening performance parameters, according to the preset total cost calculation formula corresponding to each combination of data collection mode and filter mode, find the total cost corresponding to all combinations, so as to select the combination corresponding to the lowest total cost as this time Obtain the combination plan of the task, and send the combination plan to the corresponding device to obtain training samples. Starting from the actual screening process, according to the steps of data collection, data storage, data filtering, and data labeling, the costs of various combinations are calculated and summarized to obtain the total cost of various combinations, and select from all total costs The lowest total cost is calculated, and the combination method corresponding to the lowest total cost is used as the combination plan for obtaining the task. By calculating the total cost of the data collection mode and the filter mode, a scientific evaluation system is established. While taking into account the performance and efficiency of the filter algorithm, it can also avoid the subjectivity of human experience and overcome the difficulties in the prior art. The problem of selecting the most appropriate filter in the actual situation, and avoiding the differences in opinions caused by the developers due to subjective differences in experience. Therefore, in the iterative training of data-driven algorithms represented by deep learning in the field of autonomous driving, the required training sample data is selected from the massive data, and the cost of obtaining training sample data is reduced.

The innovative points of the embodiments of the present invention include:

1. When the acquisition task of training samples is received, the number of training samples to be acquired and the proportion of training samples in the total data set are collected from the acquisition task, and the cost unit price corresponding to all data acquisition modes and the corresponding screening mode are obtained. Cost unit price and screening performance parameters, according to the preset total cost calculation formula corresponding to the combination of each data collection mode and filter mode, find the total cost corresponding to all combinations, and select the combination corresponding to the lowest total cost as Obtain the combination plan of the task this time, and send the combination plan to the corresponding device to obtain training samples. By calculating the total cost of the data collection mode and the filter mode, a scientific evaluation system is established, which takes into account the performance and efficiency of the filter algorithm while avoiding the subjectivity of human experience. This is the innovation of the embodiment of the present invention. one.

2. Obtain the cost unit price of each production link based on the evaluation method of the cost of each production link, and obtain the performance parameter of each filter based on the evaluation method of the filter mode. Scientifically evaluate the parameters of the total cost of each production link, and then summarize them to obtain the total cost of the combination method. Based on the scientific evaluation method of the cost of each production link, the establishment of a scientific evaluation system is one of the innovations of the embodiments of the present invention .

3. Starting from the actual screening process, according to the steps of data collection, data storage, data screening, and data labeling, the costs under various screening schemes are calculated and summarized to obtain the total cost of various screening schemes. Based on the cost calculation formulas of various data collection modes and filter modes, the total cost of each combination method is obtained, and a quantitative decision-making basis is provided for the choice and selection of combination methods, while taking into account the performance of the filter algorithm and the efficiency of the algorithm. It also avoids the subjectivity of human experience, which is one of the innovative points of the embodiments of the present invention.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic flowchart of a method for obtaining training samples according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a device for acquiring training samples provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

It should be noted that the terms "including" and "having" in the embodiments of the present invention and the drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

The embodiment of the present invention discloses a method and device for obtaining training samples. By calculating the total cost of the data collection mode and the filter mode, a scientific evaluation system is established. While taking into account the performance of the filter algorithm and the efficiency of the algorithm, it also The subjectivity of human experience can be avoided. Detailed descriptions are given below.

FIG. 1 is a schematic flowchart of a method for obtaining training samples according to an embodiment of the present invention. The method specifically includes the following steps:

S102: Receive a training sample acquisition task, where the acquisition task includes the number of training samples to be acquired and the proportion of the training samples in the overall data set.

In order to compare the pros and cons of multiple combinations, the intuitive idea is to give a certain metric (such as cost) and compare the size relationship of each combination under this metric. Concretize the problem: Suppose that N training samples that meet the requirements need to be selected from the data set, and the total cost of various combinations is calculated. When the acquisition task of training samples is received, the number of training samples to be acquired and the proportion of training samples in the total data set need to be collected from the acquisition task. The number of training samples to be obtained and the proportion of training samples in the overall data set determine the total cost of data collection, data storage, data screening, and data labeling. Since the proportion of samples to be screened is used to measure the difficulty of the screening task, in practice, it is difficult to obtain an accurate value of the proportion of samples to be screened. Generally, the proportions estimated in advance based on experience are used.

S104: Obtain the cost unit price corresponding to each data collection mode.

The data collection mode includes special collection mode and crowdsourcing collection mode. Special collection is a collection mode that specifically collects vehicles for certain specific purposes. The collection process includes data collection, data storage (usually saved by hard disk) and data recovery (hard disk storage). Copy the data to a suitable place), based on the collection process of special collection, when evaluating the cost unit price of the special collection mode, set the data collection cost as the cost unit price of the special collection mode, that is, the unit time collection cost. Crowdsourcing collection is a collection mode that collects data by installing data collection equipment on outsourcing vehicles. The collection process includes data collection, data transmission (mainly through a traffic network) and data recovery (putting the received data to a suitable place). In the collection process of package collection, when evaluating the cost unit price of the crowdsourcing collection mode, the data collection cost is set as the traffic cost of transmitting each sample and the depreciation cost of the equipment installed on the crowdsourcing vehicle. The unit cost of the data collection mode determines the total cost of this link of data collection.

S106: Obtain screening performance parameters and cost unit prices corresponding to each filter mode.

Filter modes include uniform sampling, manual filtering, car-side filters, cloud filters, and combined filters. The car-end filter is installed on the car, and the data can be directly filtered through the filter during the collection process. The screening performance parameters corresponding to the car-end filter mode include the precision rate of the car-end filter and the search of the car-end filter. Full rate and car-side filter speed; cloud filter needs to recycle the data collected by the vehicle and filter it in an offline state. Various large servers can be used. The screening performance parameters corresponding to the cloud filter mode include the accuracy of the cloud filter The cost unit price corresponding to the cloud filter mode includes the operating cost of cloud computing resources per unit time; for comparison, two special screening modes, uniform sampling and manual screening, are also considered. Uniform sampling is to directly obtain sample data from the collected data through uniform sampling. Manual screening is to select training samples from the collected data through manual observation. The screening performance parameters corresponding to the manual screening mode include manual screening speed and manual screening. The cost unit price corresponding to the model includes the cost of manual screening per unit time; in addition, there are combination filters, which include the first combination filter, the second combination filter, and the third combination filter. The first combination The filter is a combination of uniform sampling and manual screening, the second combination filter is a combination of cloud filters and manual screening, and the third combination filter is a combination of car-side filters, cloud filters and manual screening. The screening performance parameters corresponding to the filter mode determine the total cost of data collection, data storage, data screening, and data labeling. The unit cost of the filter mode determines the total cost of data screening.

S108: Obtain the storage cost of unit data and the unit labeling cost of manually labeling a training sample.

The storage cost of unit data determines the total cost of data storage, and the unit labeling cost determines the total cost of data labeling.

S110: Determine a combination mode of raw data acquisition, where the combination mode includes a data collection mode and a filter mode.

The combination of data collection mode and filter mode includes the following situations: special collection and uniform sampling, special collection and car filter, special collection and cloud filter, special collection and first combination filter, special collection and second Two combination filter, special collection and third combination filter, crowdsourcing collection and second combination filter, crowdsourcing collection and third combination filter. Based on different combination methods, different total cost calculation formulas are obtained.

S112: For each combination method, the cost unit price corresponding to the data collection mode of the combination method and the screening performance parameter and cost unit price corresponding to the filter mode are calculated, and the combination method obtains the training sample corresponding to the data set. total cost.

When the combination of data collection mode and filter mode is special collection and uniform sampling, the total cost calculation formula corresponding to this combination is:

When the combination of data collection mode and filter mode is special collection and car-side filter, the total cost calculation formula corresponding to this combination is:

When the combination of the data collection mode and the filter mode is special collection and cloud filter, the total cost calculation formula corresponding to the combination is:

When the combination of data collection mode and filter mode is special collection and the first combination filter, the total cost calculation formula corresponding to this combination is:

When the combination of the data collection mode and the filter mode is the special collection and the second combined filter, the total cost calculation formula corresponding to the combination is:

When the combination of data collection mode and filter mode is special collection and the third combination filter, the total cost calculation formula corresponding to this combination is:

When the combination of the data collection mode and the filter mode is crowdsourced collection and the second combined filter, the total cost calculation formula corresponding to the combination is:

When the combination of the data collection mode and the filter mode is crowdsourced collection and the third combination filter, the total cost calculation formula corresponding to the combination method is:

Where C _total is the total cost, N is the number of training samples, p is the proportion of the training samples in the total data set, T is the time interval of the uniform sampling, and R _car is the car-end screening The recall rate _{of the filter, f car} is the speed of the car-side filter, P _car is the precision rate of the car-side filter, R _cloud is the recall rate of the cloud filter, and f _cloud is the cloud Filter speed, P _cloud is the accuracy rate of the cloud filter, f _person is the manual screening speed, c _collection is the collection cost per unit time, and c _device is the depreciation cost of the equipment installed on the crowdsourced vehicle , C _netword is the traffic cost of the transmission of each sample, c _store is the storage cost of the unit data, c _resource is the operation cost of the cloud computing resource per unit time, c _person is the manual screening cost per unit time, c _label indicates the cost of the unit.

Based on the total cost calculation formula of each combination method, after obtaining the parameters described in S102, S104, S106, and S108, the parameters appearing in the total cost calculation formula are substituted into the formula to obtain the total cost corresponding to each combination method. Cost provides a quantitative decision-making basis for the choice and selection of combination methods.

S114: Select the lowest total cost from each total cost obtained, and determine the combination mode of the data collection mode and the filter mode corresponding to the lowest total cost as the original data acquisition combination plan for the acquisition task to acquire training samples.

Comparing the total cost corresponding to each combination method, you will get the combination method corresponding to the lowest total cost. This method can be determined as the original data acquisition combination plan for obtaining the task to obtain the training sample, while taking into account the performance of the filter algorithm and the efficiency of the algorithm. , It also avoids the subjectivity of human experience.

S116: Send the data acquisition combination solution to a corresponding device to acquire the original data set used to screen the training samples.

It can be seen from the above content that this embodiment can collect the number of training samples to be acquired and the proportion of training samples in the total data set from the acquisition task when the acquisition task of the training samples is received, and acquire the corresponding data collection modes. Cost unit price and screening mode corresponding to cost unit price and screening performance parameters, according to the preset total cost calculation formula corresponding to the combination of each data collection mode and filter mode, find the total cost corresponding to all combinations, so as to select the lowest The combination method corresponding to the total cost is used as the combination plan for this acquisition task, and the combination plan is sent to the corresponding device to obtain training samples. Starting from the actual screening process, according to the steps of data collection, data storage, data filtering, and data labeling, the costs of various combinations are calculated and summarized to obtain the total cost of various combinations, and select from all total costs The lowest total cost is calculated, and the combination method corresponding to the lowest total cost is used as the combination plan for obtaining the task. By calculating the total cost of the data collection mode and the filter mode, a scientific evaluation system is established. While taking into account the performance and efficiency of the filter algorithm, it can also avoid the subjectivity of human experience and overcome the difficulties in the prior art. The problem of selecting the most appropriate filter in the actual situation, and avoiding the differences in opinions caused by the developers due to subjective differences in experience.

FIG. 2 is a schematic structural diagram of a device for acquiring training samples provided by an embodiment of the present invention. Referring to FIG. 2, an apparatus for acquiring training samples provided by an embodiment of the present invention may include:

The receiving module 202 is configured to receive an acquisition task of training samples, where the acquisition task includes the number of training samples to be acquired and the proportion of the training samples in the overall data set;

The acquisition cost acquisition module 204 is configured to acquire the cost unit price corresponding to each data acquisition mode;

The screening performance parameter and cost obtaining module 206 is configured to obtain screening performance parameters and cost unit prices corresponding to each filter mode;

The storage and labeling cost obtaining module 208 is configured to obtain the storage cost of unit data and the unit labeling cost of manually labeling a training sample;

The determining module 210 is configured to determine a combination mode for acquiring raw data, the combination mode including a data collection mode and a filter mode;

The calculation module 212 is configured to, for each combination mode, based on the cost unit price corresponding to the data collection mode of the combination mode and the screening performance parameters and cost unit price corresponding to the filter mode, calculate the combination mode to obtain the said combination in the data set The total cost corresponding to the training sample;

The comparison module 214 is configured to select the lowest total cost from each total cost obtained, and determine the combination of the data collection mode and the filter mode corresponding to the lowest total cost as the original acquisition task to obtain training samples. Data acquisition combination plan;

The sending module 216 is configured to send the data acquisition combination scheme to a corresponding device to acquire the original data set used to screen the training samples.

It can be seen from the above content that this device can collect the number of training samples to be acquired and the proportion of training samples in the total data set from the acquisition task when receiving the acquisition task of training samples, and acquire the corresponding cost of all data acquisition modes The unit price and the corresponding cost unit price and the screening performance parameters of the screening mode, according to the preset total cost calculation formula corresponding to the combination of each data collection mode and the screening mode, find the total cost corresponding to all the combinations, so as to select the lowest total The combination method corresponding to the cost is used as the combination plan for this acquisition task, and the combination plan is sent to the corresponding device to obtain training samples. Starting from the actual screening process, according to the steps of data collection, data storage, data filtering, and data labeling, the costs of various combinations are calculated and summarized to obtain the total cost of various combinations, and select from all total costs The lowest total cost is calculated, and the combination method corresponding to the lowest total cost is used as the combination plan for obtaining the task. By calculating the total cost of the data collection mode and the filter mode, a scientific evaluation system is established. While taking into account the performance and efficiency of the filter algorithm, it can also avoid the subjectivity of human experience and overcome the difficulties in the prior art. The problem of selecting the most appropriate filter in the actual situation, and avoiding the differences in opinions caused by the developers due to subjective differences in experience.

Those of ordinary skill in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily necessary for implementing the present invention.

A person of ordinary skill in the art can understand that the modules in the device in the embodiment may be distributed in the device in the embodiment according to the description of the embodiment, or may be located in one or more devices different from this embodiment with corresponding changes. The modules of the above-mentioned embodiments can be combined into one module, or further divided into multiple sub-modules.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments are modified, or some of the technical features thereof are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A method for obtaining training samples is characterized in that it includes:

Receiving an acquisition task of training samples, where the acquisition task includes the number of training samples to be acquired and the proportion of the training samples in the overall data set;

Obtain the cost unit price corresponding to each data collection mode; wherein, the data collection mode includes special collection and crowdsourcing collection, the cost unit price of the special collection mode includes the unit time collection cost, and the cost unit price of the crowdsourcing collection mode includes Transmission cost of each sample, depreciation cost of equipment installed on crowdsourced vehicles;

Obtain the screening performance parameters and cost unit price corresponding to each filter mode; wherein, the filter modes include uniform sampling, manual screening, car-side filter, cloud filter, and combined filter, and the uniform sampling mode corresponds to the screening The performance parameters include the time interval of uniform sampling, the screening performance parameters corresponding to the manual screening mode include manual screening speed, the cost unit price corresponding to the manual screening mode includes the manual screening cost per unit time, and the screening corresponding to the car-side filter mode The performance parameters include the precision rate of the car-side filter, the recall rate of the car-side filter, and the speed of the car-side filter. The screening performance parameters corresponding to the cloud filter mode include the precision rate of the cloud filter and the cloud filter The recall rate and cloud filter speed of the cloud filter mode include the operating cost of cloud computing resources per unit time; the manual filtering speed refers to the number of data manually processed per unit time, the vehicle end The filter speed refers to the number of data processed by the vehicle-side filter per unit time, and the cloud filter speed refers to the number of data processed by the cloud filter per unit time;

The storage cost of acquiring unit data and the unit labeling cost of manually labeling a training sample;

Determine a combination mode of raw data acquisition, the combination mode includes a data collection mode and a filter mode;

For each combination method, based on the cost unit price corresponding to the data collection mode of the combination method and the screening performance parameters and cost unit price corresponding to the filter mode, the total cost corresponding to the training sample obtained in the data set by this combination method is calculated ；

Select the lowest total cost from each total cost obtained, and determine the combination of the data collection mode and the filter mode corresponding to the lowest total cost as the original data acquisition combination plan for the acquisition task to acquire training samples;

The original data acquisition combination scheme is sent to a corresponding device to obtain an original data set for screening the training samples.
The method according to claim 1, wherein when the combination of the data collection mode and the filter mode is special collection and uniform sampling, the total cost calculation formula corresponding to the combination is:

Where C total is the total cost, N is the number of training samples, p is the proportion of the training samples in the total data set, T is the time interval of the uniform sampling, and c collection is the unit time collection Cost, c store is the storage cost of the unit data, and c label is the unit label cost.
The method according to claim 1, wherein when the combination of the data collection mode and the filter mode is special collection and car-side filter, the total cost calculation formula corresponding to the combination is:

Where C total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, R car is the recall rate of the car-end filter, and f car is the total cost. The speed of the car-end filter, P car is the precision rate of the car-end filter, c collection is the collection cost per unit time, c store is the storage cost of the unit data, and c label is the unit labeling cost .
The method according to claim 1, wherein when the combination of the data collection mode and the filter mode is special collection and cloud filter, the total cost calculation formula corresponding to the combination is:

Wherein, C total is the total cost, N is the number of training samples, p is the proportion of the training samples in the total data set, T is the time interval of the uniform sampling, and R cloud is the cloud filter F cloud is the speed of the cloud filter, P cloud is the precision rate of the cloud filter, c collection is the collection cost per unit time, c store is the storage cost of the unit data, c resource is the operating cost of the cloud computing resource per unit time, and c label is the label cost per unit.
The method according to claim 1, wherein when the combination of the data collection mode and the filter mode is a special collection and a first combined filter, wherein the first combined filter is uniform sampling and manual screening Combination, the total cost calculation formula corresponding to this combination method is:

Where C total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, T is the time interval of the uniform sampling, and f person is the manual screening speed , C collection is the collection cost per unit time, c store is the storage cost of the unit data, c person is the manual screening cost per unit time, and c label is the unit labeling cost.
The method according to claim 1, wherein when the combination of the data collection mode and the filter mode is a special collection and a second combined filter, wherein the second combined filter is a cloud filter and a manual filter The total cost calculation formula corresponding to this combination is:

Wherein, C total is the total cost, N is the number of training samples, p is the proportion of the training samples in the total data set, T is the time interval of the uniform sampling, and R cloud is the cloud filter F cloud is the speed of the cloud filter, P cloud is the precision rate of the cloud filter, f person is the manual screening speed, c collection is the collection cost per unit time, and c store is For the storage cost of the unit data, c resource is the operating cost of the cloud computing resource per unit time, c person is the manual screening cost per unit time, and c label is the unit labeling cost.
The method according to claim 1, wherein when the combination of the data collection mode and the filter mode is a special collection and a third combined filter, wherein the third combined filter is a car-side filter and a cloud Combination of filter and manual screening, the total cost calculation formula corresponding to this combination method is:

Among them, C total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, R car is the recall rate of the car -end filter, and f car is the car-end screening P car is the precision rate of the car filter, R cloud is the recall rate of the cloud filter, f cloud is the cloud filter speed, P cloud is the precision rate of the cloud filter, and f person is the manual filter Speed, c collection is the collection cost per unit time, c store is the storage cost per unit data, c resource is the operating cost of cloud computing resources per unit time, c person is the manual screening cost per unit time, and c label is the unit labeling cost.
The method according to claim 1, wherein when the combination of the data collection mode and the filter mode is crowdsourced collection and the second combined filter, the total cost calculation formula corresponding to the combination is:

Where C total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, R cloud is the recall rate of the cloud filter, and f cloud is the Cloud filter speed, P cloud is the accuracy rate of the cloud filter, f person is the manual screening speed, c netword is the traffic cost of transmitting each sample, c store is the storage cost of unit data, and c resource is the cloud computing per unit time Resource operating cost, c person is the manual screening cost per unit time, and c label is the unit labeling cost.
The method according to claim 1, wherein when the combination of the data collection mode and the filter mode is crowdsourced collection and the third combined filter, the total cost calculation formula corresponding to the combined mode is:

Where C total is the total cost, N is the number of training samples, p is the proportion of the training samples in the overall data set, R car is the recall rate of the car-end filter, and f car is the total cost. The car-side filter speed, P car is the precision rate of the car-side filter, R cloud is the recall rate of the cloud filter, f cloud is the cloud filter speed, and P cloud is the cloud The precision rate of the filter, f person is the manual screening speed, c device is the depreciation cost of the equipment installed on the crowdsourced vehicle, c netword is the traffic cost of transmitting each sample, and c store is the cost The storage cost of the unit data, c resource is the operating cost of the cloud computing resource per unit time, c person is the manual screening cost per unit time, and c label is the unit labeling cost.
A device for acquiring training samples is characterized in that it comprises:

A receiving module configured to receive an acquisition task of training samples, the acquisition task including the number of training samples to be acquired and the proportion of the training samples in the overall data set;

The acquisition cost acquisition module is configured to acquire the cost unit price corresponding to each data acquisition mode;

The screening performance parameter and cost acquisition module is configured to obtain the screening performance parameter and cost unit price corresponding to each filter mode;

The storage and labeling cost acquisition module is configured to acquire the storage cost of unit data and the unit labeling cost of manually labeling a training sample;

A determining module, configured to determine a combination mode of original data acquisition, the combination mode including a data collection mode and a filter mode;

The calculation module is configured to, for each combination mode, based on the cost unit price corresponding to the data collection mode of the combination mode and the screening performance parameter and cost unit price corresponding to the filter mode, calculate the combination mode to obtain the training in the data set The total cost corresponding to the sample;

The comparison module is configured to select the lowest total cost from each total cost obtained, and determine the combination of the data collection mode and the filter mode corresponding to the lowest total cost as the raw data of the acquisition task to obtain training samples Get a combination plan;

The sending module is configured to send the data acquisition combination scheme to a corresponding device to acquire the original data set used to screen the training samples.