CN114881096A

CN114881096A - Multi-label class balancing method and device

Info

Publication number: CN114881096A
Application number: CN202110164841.6A
Authority: CN
Inventors: 王俊凯; 李亚敏; 刘新春
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-02-05
Filing date: 2021-02-05
Publication date: 2022-08-09
Also published as: WO2022166325A1

Abstract

A method of multi-label class balancing, the method comprising: obtaining a sample set comprising a plurality of instances, each instance comprising one or more category labels; classifying the instances in the sample set according to the class labels to form a plurality of classes, wherein each instance is classified into one or more classes, and each class corresponds to one class; determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in each subclass and the number of all class labels in each instance, wherein the subclass is a plurality of classes of which the number is smaller than a first threshold value, the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is a sampling weight corresponding to the subclass in each instance; and carrying out weighted sampling on the examples of each subclass according to the target sampling weight of each example in the subclass. The scheme also provides a device which can avoid scene loss and indirectly increase the number of other types.

Description

Class balancing method and device for multiple labels

Technical Field

The application relates to the technical field of artificial intelligence and computer vision, in particular to a multi-label class balancing method and device.

Background

Target detection is an important research direction in the fields of computer vision and image processing, and is widely applied to the related fields of video live broadcast, security monitoring, robots, human-computer interaction and the like at present. However, most data types detected by targets in real life have long tail effects, namely, the head class occupies most samples, and the tail class has few samples. For example, the number of instances of trains and cars in the 2D data set BDD is 600 times different, and the number of instances of bicycles and cars in the 3D data set nuScenes is 4 times different. Currently, sampling techniques can address the long tail effect of data. However, sampling techniques often result in scene loss or indirectly increase the number of other classes.

Disclosure of Invention

In view of the above, there is a need to provide a multi-label class balancing method and apparatus thereof, which can avoid scene loss and indirectly increase the number of other classes.

An embodiment of the present invention provides a class balancing method for multiple tags, where the method includes: obtaining a sample set comprising a plurality of instances, each instance comprising one or more category labels; classifying the instances in the sample set according to the class labels to form a plurality of classes, wherein each instance is classified into one or more classes, and each class corresponds to one class; determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in each subclass and the number of all class labels in each instance, wherein the subclass is a plurality of classes of which the number is smaller than a first threshold value, the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is a sampling weight corresponding to the subclass in each instance; and carrying out weighted sampling on the examples of each subclass according to the target sampling weight of each example in the subclass.

According to the method and the device, the weight model is built for the subclasses, the target sampling weight of each instance of each class is calculated, the instances of each subclass are weighted and sampled to increase the instances, only the instances can be increased, the original instances can be completely reserved, the instances are not lost, meanwhile, the more the number of the target class labels in the subclasses is compared with the number of all the class labels, the more the instances are easily sampled, the influence on other classes is small, and scene loss and indirect increase of the number of other classes can be avoided.

According to some embodiments of the application, the method further comprises: and constructing the subclass by taking the target instance determined according to the weighted sampling result as a new added instance.

The application can further avoid the long tail effect by constructing the subclasses to increase the number of instances in the subclasses.

According to some embodiments of the application, the method further comprises: constructing instances in the subclass to a first number; determining whether the number of instances in each subclass reaches a second number; if the number of the instances in each subclass does not reach the second number, continuously classifying the instances in the sample set according to the class labels to form a plurality of classes, continuously determining the target sampling weight of each instance in each subclass, continuously performing weighted sampling on the instances of each subclass, and continuously constructing the subclass by taking the target instances determined according to the weighted sampling result as newly-added instances until the number of the instances in each subclass reaches the second number.

The data quantity of the subclasses can be increased in a directional mode, and based on an iterative sampling strategy, due to the fact that the new examples are added, the preset target example number, the first number and the second number can be adjusted during each sampling, the total sample quantity cannot be increased suddenly, and the number of other classes is further prevented from being increased indirectly.

According to some embodiments of the application, the method further comprises: if the number of instances in each subclass reaches a second number, the instances of all classes are output.

According to the method and the device, the training equipment can train the initial AI model according to the examples to obtain the target model by outputting the examples of all classes.

According to some embodiments of the application, prior to said constructing the first number of instances in the subclass, the method further comprises: the first number is determined.

According to the scheme, the first quantity can be adjusted during each sampling cycle by determining the first quantity.

According to some embodiments of the application, the determining the first number comprises: determining the first number to be an average of the number of current instances of all classes.

According to the scheme, the first number is the average value of the number of the current examples of all classes, so that the N is avoided _aim Too large a total number of instances in the sample set is likely to result in too large a number, and the N is avoided _aim Too small easily results in too many iterations and slow convergence.

According to some embodiments of the application, prior to the determining whether the number of instances in each subclass reaches the second number, the method further comprises: determining the second number.

By determining the second quantity, the second quantity can be adjusted during each sampling cycle.

According to some embodiments of the application, the determining the second number comprises: determining the second quantity as r N _max Where r is [0.5,0.8 ]]，N _max Is the number of instances of the largest class.

In the scheme, the second quantity is r x N _max Where r is [0.5,0.8 ]]，N _max For the number of the instances of the maximum class, it is avoided that too large of the second number easily causes a large number of repeated samples in a new sample set, easily causes overfitting in a training process, and that too small of the second number easily causes an insignificant balancing effect of each class.

According to some embodiments of the application, the weighted sampling comprises at least one of weighted random sampling WRS and naive weighted sampling.

In the scheme, the weighted sampling comprises at least one weighted random sampling method WRS and naive weighted sampling method, so that the newly added examples mainly comprise more target class labels, and the imbalance of each class can be reduced.

According to some embodiments of the present application, the weighted sampling of the instances of each subclass according to the target sampling weight of each instance in the subclass comprises: generating a random number in a preset interval aiming at each instance in the subclass; determining the sampling score of each instance in the subclass by a weighted random sampling method WRS according to the random number and the target sampling weight; and taking the target instance with the maximum sampling score as a new added instance.

According to the method, the weighted random sampling method WRS is adopted, so that the sampling score of the example with the larger target sampling weight is larger, and the example with the larger target sampling weight is easier to sample by taking the target example with the largest sampling score as a new example.

According to some embodiments of the present application, the weighted sampling of the instances of each subclass according to the target sampling weight of each instance in the subclass may include: ordering all the instances in the subclass according to the sequence of the target sampling weight from small to large; generating a random number of a preset interval; and determining a new added example by a naive weighted sampling method according to the random numbers and the ordering of the target sampling weights.

According to the scheme, through a naive weighting sampling method, the accumulated value one by one of the examples with the larger target sampling weight is larger, and the example with the larger target sampling weight is easier to sample by taking the target example with the accumulated value reaching the random number one by one as a new example.

According to some embodiments of the present application, before determining the target sampling weight for each instance in each subclass according to the number of target class labels in each instance in the subclass and the number of all class labels in each instance, the method further comprises: a subclass of a plurality of the classes is determined.

According to the scheme, the subclasses can be sampled by determining the subclasses.

According to some embodiments of the application, said determining a subclass of a plurality of said classes comprises: determining the sampling rate of each class according to the number of the current examples of each class and the preset target example number; determining the class with the sampling rate greater than zero as a subclass of the plurality of classes.

According to the applicationIn some embodiments, the determining the sampling rate of each class according to the number of current instances and the preset target number of instances of each class includes: according to the formula

Determining a sampling rate, wherein f _t Is a sampling rate of class t, N _aim For the preset target number of instances, N _i Is the number of current instances of class t.

The scheme is represented by a formula

The sampling rate, and hence the subclass, may be determined.

According to some embodiments of the present application, determining the target sampling weight for each instance in each subclass according to the number of target class labels in each instance in the subclass and the number of all class labels in each instance comprises: determining the proportion of the target class labels of each example in each subclass according to the number of the target class labels in each example in the subclass and the number of all the class labels in each example; and determining the target sampling weight of each example in each subclass according to the occupation ratio of the target class label of each example in each subclass.

According to the scheme, through a weight model, the more the number of target class labels of an instance in each subclass is relative to the number of all class labels in the instance, the higher the target sampling weight of the instance of the class is; and the fewer the number of target class labels of an instance in each subclass relative to the number of all class labels in that instance, the lower the target sampling weight for that instance of that class.

According to some embodiments of the present application, determining the percentage of the object class label of each instance in each subclass according to the number of the object class labels in each instance in the subclass and the number of all class labels in each instance comprises: according to the formula

Determining each subclassTarget class label proportion for each instance, where p _ti Proportion of object class labels, k, for instance i in subclass t _ti Number of object class labels, m, for instance i in subclass t _ti The number of all class labels in instance i in the subclass t.

The scheme is represented by a formula

Such that the greater the number of target category labels for an instance in each subclass relative to the number of all category labels in the instance, the higher the duty cycle of the target category labels for the instance of the class; the fewer the number of target class labels of an instance in each subclass relative to the number of all class labels in that instance, the lower the duty cycle of the target class labels for that instance of that class.

According to some embodiments of the present application, the determining the target sampling weight of each instance in each subclass according to the duty ratio of the target class label of each instance in each subclass comprises: according to the formula

Determining a target sampling weight for each instance in each subclass, wherein w _ti Target sampling weight, p, for instance i in subclass t _ti A target class label ratio for instance i in the subclass t, tn is the total number of all instances in the subclass t,

is the sum of the target class labels of all instances in the subclass t.

The scheme is represented by a formula

Such that the higher the duty cycle of a target class label for an instance in each subclass, the higher the target sampling weight for the instance of the class; the lower the duty cycle of a target class label for an instance in each subclass, the lower the target sampling weight for that instance of that class。

An embodiment of the present invention further provides a class balancing apparatus for multiple tags, where the apparatus includes: an obtaining unit for obtaining a sample set comprising a plurality of instances, each instance comprising one or more category labels; a sampling unit, configured to classify the instances in the sample set according to the class labels to form a plurality of classes, where each instance is classified into one or more classes, and each class corresponds to a class; the sampling unit is used for determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in each subclass and the number of all class labels in each instance, wherein the subclass is a plurality of classes of which the number is smaller than a first threshold value, the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is a sampling weight corresponding to the subclass in each instance; the sampling unit is further configured to perform weighted sampling on the instances of each subclass according to the target sampling weight of each instance in the subclass.

According to some embodiments of the application, the sampling unit is further configured to construct the subclass using the target instance determined from the weighted sampling result as a new sample.

According to some embodiments of the present application, the sampling unit is further configured to construct the instances in the subclass to a first number; the sampling unit is further configured to determine whether the number of instances in each subclass reaches a second number; the sampling unit is further configured to, if the number of instances in each subclass does not reach the second number, continue to classify the instances in the sample set according to the class labels to form a plurality of classes, continue to determine a target sampling weight for each instance in each subclass, and continue to perform weighted sampling on the instances of each subclass until the number of instances in each subclass reaches the second number.

According to some embodiments of the application, the sampling unit is further configured to output the instances of all classes if the number of instances in each of the subclasses reaches a second number.

According to some embodiments of the application, prior to said constructing the instances in the subclass to the first number: the sampling unit is further configured to determine the first number.

By determining the first number, the first number can be adjusted each time sampling is performed.

According to some embodiments of the application, the sampling unit is further configured to determine the first number to be an average of the number of current instances of all classes.

According to some embodiments of the application, prior to said determining whether the number of instances in each subclass reaches the second number: the sampling unit is further configured to determine the second number.

By determining the second number, the second number can be adjusted each time sampling is performed.

According to some embodiments of the present application, the sampling unit is further configured to determine the second number as r × N _max Where r is [0.5,0.8 ]]，N _max Is the number of instances of the maximum class.

In the scheme, the second quantity is r x N _max Where r is [0.5,0.8 ]]，N _max For the number of instances of the maximum class, it is avoided that a large second number easily causes a large number of repeated samples in a new sample set, easily causes overfitting, and that a small second number easily causes a poor equalization effect of each class.

According to some embodiments of the present application, the sampling unit is further configured to generate a random number of a preset interval for each instance in the subclass; determining the sampling score of each instance in the subclass by a weighted random sampling method WRS according to the random number and the target sampling weight; and taking the target instance with the maximum sampling score as a new added instance.

According to some embodiments of the present application, the sampling unit is further configured to sort all instances in the subclass in order of smaller to larger target sampling weights; generating a random number of a preset interval; and determining a new added example by a naive weighted sampling method according to the random numbers and the ordering of the target sampling weights.

According to some embodiments of the application, before determining the target sampling weight for each instance in each subclass according to the number of target class labels in each instance in the subclass and the number of all class labels in each instance: the sampling unit is further configured to determine a subclass of a plurality of the classes.

According to some embodiments of the present application, the sampling unit is further configured to determine a sampling rate of each class according to the number of current instances of each class and a preset target number of instances; determining the class with the sampling rate greater than zero as a subclass of the plurality of classes.

According to some embodiments of the application, the sampling unit is further configured to generate the sampling signal according to a formula

Determining a sampling rate, wherein f _t Is a sampling rate of class t, N _ai For the preset target number of instances, N _i Is the number of current instances of class t.

The scheme is represented by a formula

The sampling rate, and hence the subclass, may be determined.

According to some embodiments of the present application, the sampling unit is configured to determine a proportion of the target class label of each instance in each subclass according to the number of the target class labels in each instance in the subclass and the number of all class labels in each instance; the sampling unit is further used for determining a target sampling weight of each example in each subclass according to the proportion of the target class label of each example in each subclass.

Determining a target class label fraction for each instance in each subclass, wherein p _ti Proportion of object class labels, k, for instance i in subclass t _ti Number of object class labels, m, for instance i in subclass t _ti The number of all class labels in instance i in the subclass t.

The scheme is represented by a formula

is the sum of the target class labels of all instances in the subclass t.

The scheme is represented by a formula

Such that the higher the duty cycle of a target class label for an instance in each subclass, the higher the target sampling weight for the instance of the class; the lower the duty cycle of a target class label for an instance in each subclass, the lower the target sampling weight for that instance of that class.

An embodiment of the present invention further provides an electronic device, where the electronic device includes a processor and a memory, where the memory is used to store program instructions, and when the processor calls the program instructions, the multi-tag class balancing method as described in any of the above is implemented.

An embodiment of the invention also provides a vehicle including the electronic device.

An embodiment of the present invention also provides a computer-readable storage medium storing a program that causes a computer device to implement the multi-label class equalization method as described in any one of the above.

An embodiment of the present invention also provides a computer program product comprising computer executable instructions stored in a computer readable storage medium; the computer-executable instructions may be read from the computer-readable storage medium by at least one processor of a device, execution of which by the at least one processor causes the device to perform the multi-tag class-balancing method as described in any one of the above.

Drawings

Fig. 1 is a schematic system architecture diagram according to an embodiment of the present invention.

Fig. 2 is a flowchart illustrating a class balancing method for multiple tags according to a first embodiment of the present invention.

FIG. 3 is a schematic diagram of an example of an embodiment of the invention.

FIG. 4 is a diagram illustrating the number of instances under the corresponding category after the instances in the sample set are classified according to an embodiment of the present invention.

Fig. 5 is a schematic diagram of a category label of a category t according to an embodiment of the present invention.

Fig. 6 is a flowchart illustrating a class balancing method for multiple tags according to a second embodiment of the present invention.

FIG. 7 is a diagram illustrating an example of the subclasses constructed such that the number of examples per subclass reaches a first number, according to an embodiment of the present invention.

Fig. 8 is a schematic diagram of iteratively constructing a subclass for the subclass according to the embodiment of the present invention.

Fig. 9 is a schematic structural diagram of a multi-tag class equalization apparatus according to an embodiment of the present invention.

Fig. 10 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.

Fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Description of the main elements

System architecture 10

Data acquisition device 160

Database 130

Training device 120

Object model 101

Execution device 110

I/O interface 112

Client device 140

Data storage system 150

Calculation Module 111

Multi-label class balancing device 900

Acquisition unit 901

Sampling unit 902

Training unit 903

Electronic device 1000

Electronic device 1100

Memory

1001,1101

Processor

1002,1102

Communication interface

1003,1103

Bus

1004,1104

Detailed Description

In the description of the embodiments of the present application, words such as "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "for example" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "such as" is intended to present relevant concepts in a concrete fashion.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. It is to be understood that, in this application, unless otherwise indicated, "plurality" means two or more than two.

Existing hybrid sampling-based data equalization methods form new data sets by example classification, mean calculation, random upsampling, random downsampling, and combination. The example classification includes counting the number of examples corresponding to each category. The calculating the average value comprises accumulating the number of the instances of each category and then averaging. The random upsampling includes upsampling the class with the number of instances below the average by putting back the random samples so that the number of instances equals the average. The random downsampling includes downsampling a class for which the number of instances exceeds the average value such that the number of instances equals the average value. The combining includes recombining the category instances into a new data set. However, in random down-sampling, part of the scene is directly discarded, thereby causing scene loss. In random upsampling, since upsampling is performed by putting back random sampling, it cannot be ensured that each instance is selected certainly, and a scene may be lost. Meanwhile, since one example may include a plurality of tags, when upsampling is performed based on a certain tag, the number of other tags may be increased greatly at the same time, which may result in imbalance among the classes.

In order to better understand the method of the embodiments of the present application, some terms and concepts related to the present application will be described below.

Example (c): generally refers to picture samples (Image) in a 2D dataset and Point Cloud samples (Point Cloud) in a 3D dataset.

Sample set: a collection of instances, each element in the collection being an instance.

The category: it is a distinction between different things, and is made according to different categories.

Class (c): a subset is defined over the set of samples made up of all instances, samples of the same class being indistinguishable in some property of interest to us, i.e. having the same pattern.

Subclass: refers to a number of classes less than a first threshold.

Before explaining the embodiments of the present application in detail, an application scenario related to the embodiments of the present application will be described.

Currently, instances in the training dataset are available for the model. The model may be an AI model. The AI model is a machine learning model, which is essentially a mathematical model that includes a large number of parameters and mathematical formulas (or mathematical rules) in order to learn mathematical expressions that provide a correlation between an input value x and an output value y, and the mathematical expression that provides the correlation between x and y is a trained AI model. The initial AI model needs to be trained before detection with the AI model.

Factors that have an impact on AI model training mainly include three aspects: training data sets, initial AI models, and machine calculations. With the increase of current application scenarios of the AI model, the scenarios to which the AI model needs to face are complicated, resulting in more and more complex training AI models. Meanwhile, in order to improve the training effect of the AI model, the requirement on the data volume of the training data set is increased, and the data types are increased. This results in an increased computational effort for the training process, an increasing demand for machine computing power, and an increasing time for training. How to optimize the AI model training process to obtain a better accurate AI model in the shortest time is a major concern of the industry.

Based on this, in order to improve the performance of AI model training, after writing the initial neural network model to be trained, a developer can establish an example by the multi-label class balancing method provided by the embodiment of the present application, and can train the AI model to be trained by using the training data set including the example constructed by the present application, thereby improving the accuracy of model inference. The neural network model to be trained is an initial AI model to be trained.

The trained AI model has predictive analysis capability and can be used for various different functions. For example, data of interest (e.g., a person or object in an image/video) may be found from target data (e.g., a video stream, a picture stream, or an image) to determine the location and size of the object in the image or video. The object of interest is called a target, and the type of the target can be determined according to an application scene and a service direction. For example, in the field of monitoring security, the target may be, but is not limited to, a vehicle, a human body, a human face, etc. In the traffic field, the target may be, but is not limited to, a road, a license plate number, a road facility, a traffic sign, a pedestrian, etc. The system is applied to the field of logistics, and targets can be, but are not limited to, container numbers, express bill numbers, obstacles, package numbers and the like. In military and defense applications, the targets may be, without limitation, terrain, flying equipment, etc. In the medical field, the target may be, but is not limited to, an organ, a tissue, or the like.

Obviously, the trained AI model may also be applied to other scenarios besides the target detection scenario, such as recognition, segmentation, and the like. In the following, the trained AI model is applied to a target detection scenario as an example, but the application is not limited to the target detection scenario.

Referring to fig. 1, fig. 1 is a schematic diagram of a system architecture 10 according to an embodiment of the present invention. The data acquisition device 160 is used to acquire training data. For the target detection method (one example of application examples of the AI model) of the embodiment of the present application, the training data may include training instances, where the result of the training instance may be a result labeled in advance manually or a result labeled automatically by an existing labeling system, that is, each training instance includes one or more category labels. After the training data is collected, data collection facility 160 stores the training data in database 130. The training device 120 trains the target model 101 based on training data maintained in the database 130. How the training device 120 trains the target model 101 based on the training data will be described in more detail below, the target model 101 can find data of interest (e.g., a person or an object in an image/video) in the target data (e.g., a video stream, a picture stream, or an image) to determine the position and size of the object in the image or video.

In this embodiment, the training apparatus 120 may obtain a sample set including a plurality of instances, each instance including one or more category labels; classifying the instances in the sample set according to the class labels to form a plurality of classes, wherein each instance is classified into one or more classes, and each class corresponds to one class; determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in each subclass and the number of all class labels in each instance, wherein the subclass is a plurality of classes of which the number is smaller than a first threshold value, the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is a sampling weight corresponding to the subclass in each instance; and carrying out weighted sampling on the examples of each subclass according to the target sampling weight of each example in the subclass. The training device 120 may also train the initial AI model according to the instances to obtain the target model 101.

It should be noted that, in practical applications, the training data maintained in the database 130 may not necessarily all come from the acquisition of the data acquisition device 160, and may also be received from other devices. It should be noted that, the training device 120 does not necessarily perform the training of the target model 101 based on the training data maintained by the database 130, and may also obtain the training data from the cloud or other places for performing the model training, for example, obtain the training data from the 2D data set BDD or the 3D data set nuScenes for performing the model training, and the above description should not be taken as a limitation to the embodiments of the present application.

The target model 101 obtained by training according to the training device 120 may be applied to different systems or devices, for example, the execution device 110 shown in fig. 1, where the execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an AR/VR, a vehicle-mounted terminal, or a server or a cloud. In fig. 1, the execution device 110 is configured with an I/O interface 112 for data interaction with an external device, and a user may input data to the I/O interface 112 through the client device 140, where the input data may include, in an embodiment of the present application: the image to be processed is input by the client device.

The data storage system 150 is used for receiving and storing the parameters of the target model sent by the training device 120, and for storing the data of the target detection result obtained by the target model 101, and may also include program codes (or instructions) required for the data storage system 150 to operate normally. The data storage system 150 may be a distributed storage cluster configured by one device or multiple devices disposed outside the execution device 110, in this case, when the execution device 110 needs to use data on the data storage system 150, the data storage system 150 may send the data needed by the execution device 110 to the execution device 110, and accordingly, the execution device 110 receives and stores (or caches) the data. Of course, the data storage system 150 may also be deployed in the execution device 110, and when deployed in the execution device 110, the distributed storage system may include one or more memories, and optionally, when there are multiple memories, different memories are used to store different types of data, for example, the model parameters of the target model generated by the training device and the data of the target detection result obtained by the target model 101 may be stored in two different memories, respectively.

The calculation module 111 uses the object model 101 to process an input image to be processed (e.g., a video stream, a picture stream, or an image), for example, to find data of interest (e.g., a person or an object in the image/video) from the input image to be processed, to determine the position and size of the object in the image or video.

Finally, the I/O interface 112 returns the processing result, such as the position and size of the object in the image or video obtained as described above, to the client device 140, thereby providing it to the user.

It should be noted that the training device 120 may generate corresponding target models 101 for different targets or different tasks based on different training data, and the corresponding target models 101 may be used to achieve the targets or complete the tasks, so as to provide the user with the required results.

In the case shown in fig. 1, the user may manually give the input data, which may be operated through an interface provided by the I/O interface 112. Alternatively, the client device 140 may automatically send the input data to the I/O interface 112, and if the client device 140 is required to automatically send the input data to obtain authorization from the user, the user may set the corresponding permissions in the client device 140. The user can view the result output by the execution device 110 at the client device 140, and the specific presentation form can be display, sound, action, and the like. The client device 140 may also serve as a data collection terminal, collecting input data of the input I/O interface 112 and output results of the output I/O interface 112 as new sample data, and storing the new sample data in the database 130. Of course, the input data inputted to the I/O interface 112 and the output result outputted from the I/O interface 112 as shown in the figure may be directly stored in the database 130 as new sample data by the I/O interface 112 without being collected by the client device 140.

It should be noted that fig. 1 is only a schematic diagram of a system architecture provided by an embodiment of the present invention, and the position relationship between the devices, modules, etc. shown in the diagram does not constitute any limitation, for example, in fig. 1, the data storage system 150 is an external memory with respect to the execution device 110, and in other cases, the data storage system 150 may be disposed in the execution device 110.

In this embodiment of the present application, the training device 120, the executing device 110, and the client device 140 may be three different physical devices, or the training device 120 and the executing device 110 may be on the same physical device or a cluster, or the executing device 110 and the client device 140 may be on the same physical device or a cluster.

Fig. 2 is a schematic flow chart of a multi-tag class balancing method according to a first embodiment of the present invention. The method may be specifically executed by the training device 120 shown in fig. 1, where sample sets of multiple instances in the method may be training data maintained in the database 130 shown in fig. 1, optionally, steps S202 to S206 of the method may be executed in the training device 120, or may be executed in advance by other functional modules before the training device 120, that is, training data received or acquired from the database 130 is preprocessed, and a new sample set is obtained as an input to the training device 120 through the sampling process described in steps S202 to S206, and training of a model is executed by the training device.

Optionally, the method may be processed by a CPU, or may be processed by both the CPU and the GPU, or may use other processors suitable for processing without using the GPU, and the application is not limited thereto.

The method includes, but is not limited to, the steps of:

s201: a sample set is obtained that includes a plurality of instances, each instance including one or more category labels.

The sample set may be collected by the data collection device 160 shown in fig. 1, wherein each instance in the sample set collected by the data collection device 160 is a manually pre-labeled result or a result automatically labeled by an existing labeling system. The sample set may also be a result obtained from a 2D data set BDD or a 3D data set nuScenes, or the like, or a result obtained from a data set from another place such as a memory, and automatically labeled by an existing labeling system. The sample set may include all data sets in the 2D data set BDD or the 3D data set nuScenes, or may include only a part of the data sets.

The sample set is briefly described as an automatic driving data set in the present application. In this example, there are 10 categories in the sample set, bus, traffic light, sign, person, bicycle, car, truck, motorcycle, train, and rider. However, it is to be understood that for the set of automated driving data, the categories of the sample set are not limited to the above categories, but may be other categories. The sample set is not limited to an autopilot data set, but may be a data set from a dog in Stanford Dogs Dataset, or a data set from a facial image of Labelled Faces in the Wild, or the like. Obviously, the categories of the sample sets are different for different sample sets.

The example may be a picture sample in a 2D data set. The example may also be a point cloud sample in a 3D dataset. An example may be as shown in fig. 3. In fig. 3, the example includes 6 category labels, traffic sign, rider, bike, truck, and car, respectively. In this embodiment, each object in the example is indicated by a two-dimensional border. Each category label is used to represent an object. The class labels are labeled on the instances, and the labeling of these class labels may help determine the type of each object. For example, a category tag-car indicates that the object of the box selection is a car, a category tag-traffic sign indicates that the object of the box selection is a traffic light, a category tag-rider indicates that the object of the box selection is a rider, a category tag-bike indicates that the object of the box selection is a bicycle, and a category tag-truck indicates that the object of the box selection is a truck. The same category label belongs to a class. The 6 category labels belong to 5 categories, which are cars, traffic lights, riders, bicycles, and trucks, respectively.

S202: and classifying the instances in the sample set according to the class labels to form a plurality of classes, wherein each instance is classified into one or more classes, and each class corresponds to one class.

In this embodiment, if an instance includes a category label, the index of the instance is added under the category. If an instance includes multiple category labels, the index of the instance is added to the corresponding category. If an instance includes multiple identical category labels, the index of the instance is added to the category once. Taking the example in fig. 3 as an example, the example includes 2 traffic sign category tags, 1 car category tag, 1 rider category tag, 1 bike category tag, and 1 truck category tag, and the index numbers of the example are added under the category traffic light, the category car, the category rider, the category bicycle, and the category truck, respectively. The index numbers of the add instances under categories can be as shown in table 1 below:

TABLE 1

After the instances in the sample set are classified, a plurality of classes are formed, and the number of instances in different classes may be the same or different, as shown in fig. 4. In fig. 4, the number of examples in the automobile category is the largest and 27558, and the number of examples in the bicycle category is the smallest and 6263. In fig. 4, the head category accounts for most instances and the tail category accounts for few instances, for example, the automobile category and the human category account for most instances, and the train category, the rider category, the motorcycle category and the bicycle category account for few instances.

S203: a subclass of a plurality of the classes is determined.

The subclass is a plurality of classes of which the number is smaller than a first threshold value. Specifically, the determining a subclass of the plurality of classes includes: determining the sampling rate of each class according to the number of the current examples of each class and the preset target example number; determining the class with the sampling rate greater than zero as a subclass of the plurality of classes. The determining the sampling rate of each class according to the number of the current instances of each class and the preset target instance number comprises: according to the formula

Determining a sampling rate, wherein f _t Is a sampling rate of class t, N _aim For the preset target number of instances, N _i Is the number of current instances of class t. Said N is _aim The average value of the number of instances in each category may be, or slightly larger than, or slightly smaller than the average value of the number of instances in each category. Wherein if said N is _aim Too large, easily resulting in an excessive total number of instances in the sample set, if said N _aim Too small, it is easy to cause too many iterations and slow convergence. The class having the sampling rate greater than zero is a large one of the plurality of classes.

For example, taking fig. 4 as an example, the number of current examples of the bicycle class is 6263, and if the preset target number of examples is 14000, the formula is followed

If the sampling rate is determined to be about 1.24 and greater than zero, the bicycle class can be determined to be a subclass of the plurality of classes.

S204: and determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in the subclass and the number of all class labels in each instance, wherein the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is the sampling weight corresponding to the subclass in each instance.

In the present embodiment, each instance may include one or more category labels, which may be grouped into one or more classes. The different instances may or may not include the same duty cycle for the different classes. Taking the example of fig. 3 above as an example, the percentage of class traffic lights, class cars, class riders, class bicycles, and class trucks included in the above examples are 1/3, 1/6, 1/6, 1/6, and 1/6, respectively. Then the target class label fraction of different instances under one class may be the same or different, and therefore the new sample set formed by upsampling from different instances under the same class is also different. Therefore, prior to upsampling, a ratio of the duty ratio of the target class label of each instance in the different classes in the subclass over all its class labels to the sum of the duty ratios of the class labels of all instances in said class, i.e., a target sampling weight for each instance in each subclass, needs to be determined.

In this embodiment, the determining the target sampling weight of each instance in each subclass according to the number of target class labels in each instance in the subclass and the number of all class labels in each instance includes: determining the proportion of the target class labels of each example in each subclass according to the number of the target class labels in each example in each subclass and the number of all the class labels in each embodiment; and determining the target sampling weight of each example in each subclass according to the occupation ratio of the target class label of each example in each subclass.

In this embodiment, the determining the proportion of the target class label of each instance in each subclass according to the number of the target class labels in each instance in the subclass and the number of all class labels in each instance includes: according to the formula

The description will be given by taking a subclass m as an example, and as shown in FIG. 5, the subclass m is a bicycle and includesn instances, instance _1, …, instance _ i, …, and instance _ n, respectively. The category labels included in different instances may or may not be the same. For example, in fig. 5, the category label included in instance _1 is bike, truck, the category label included in instance _ i is bike, bike, rider, motor, and the category label included in instance _ n is bike, truck, car. Then the fraction of bike category labels of instance i in the bicycling class over all category labels of instance i, i.e. the fraction of said bike category labels of instance i in the bicycling class, is

It follows that if the number of target category labels of an instance in the subclass is greater relative to the number of all category labels in the instance, then the duty cycle of the target category labels of the instance of the class is higher; the lower the duty cycle of the target class label for an instance of the class if the number of target class labels of the instance in the subclass is lower relative to the number of all class labels in the instance.

In this embodiment, the determining the target sampling weight of each instance in each subclass according to the percentage of the target class label of each instance in each subclass includes: according to the formula

is the sum of the target class labels of all instances in the subclass t.

The subclass t is taken as an example for explanation, the subclass t comprises an example i and an example j, and the proportion of the bike class label of the example i is

The proportion of bike class labels for example j is

The sum of the ratios of the bike category labels for all instances of the subclass t is

From this, the target sampling weight of instance i in subclass t is given by

It follows that if the duty ratio of a target class label of an instance in each subclass is higher, the target sampling weight of the instance of the class is higher; the lower the duty cycle of a target class label for an instance in each subclass, the lower the target sampling weight for that instance of that class. That is, if the number of target class labels of an instance in each subclass is larger relative to the number of all class labels in the instance, the target sampling weight for the instance of the class may be higher; the lower the target sampling weight for an instance of each subclass may be if the number of target class labels for the instance is less relative to the number of all class labels in the instance.

S205: and carrying out weighted sampling on the examples of each subclass according to the target sampling weight of each example in the subclass.

In this embodiment, when a subclass is upsampled, if an instance in which the number of target class labels in the subclass is greater than the number of all class labels in the instance is upsampled, the newly added instance is mainly an instance including more target class labels, and imbalance of each class is reduced.

In this embodiment, the weighted sampling includes at least one of weighted random sampling WRS and weighted naive sampling.

In this embodiment, the performing weighted sampling on the instances of each subclass according to the target sampling weight of each instance in the subclass may include:

generating a random number of a preset interval aiming at each instance in the subclass;

determining the sampling score of each instance in the subclass by a weighted random sampling method WRS according to the random number and the target sampling weight;

and taking the target instance with the maximum sampling score as a new added instance.

In this embodiment, the preset interval is a (0, 1) interval. The preset interval can also be other intervals, such as (0,2) and the like. The random number generator can be used for generating random numbers in a plurality of (0, 1) intervals. Determining the sampling score of each instance in the subclass by a weighted random sampling method WRS according to the random number and the target sampling weight comprises: according to the formula

ti e tn determines the sampling rate, where S _ti Sample score, R, for instance i in class t _ti Random number, w, for instance i in class t _ti Is the target sampling weight of instance i in the subclass t, ti is the ith instance in the subclass t, and tn is the nth instance in the subclass t. For example, continuing with the above example of class t, the target sampling weight of instance i in subclass t is

The target sampling weight of instance j in subclass t is

If the random number of the example i in the subclass t is 0.8 and the random number of the example j in the subclass t is 0.6, the sampling scores of the example i and the example j in the subclass t can be determined to be 0.69 and 0.28 respectively, and the example i with the sampling score of 0.69 is used as a new example.

ordering all the instances in the subclass according to the sequence of the target sampling weight from small to large;

generating a random number of a preset interval;

and determining a new added example by a naive weighted sampling method according to the random number and the ordering of the target sampling weight.

In this embodiment, the preset interval is a (0, 1) interval. The preset interval can also be other intervals, such as (0,2) and the like. The random number generator can be used for generating a plurality of random numbers in (0, 1) intervals. The determining the new added example by a naive weighted sampling method according to the random number and the ordering of the target sampling weight comprises: accumulating the target sampling weights of the examples one by one from the target sampling weight of the first example according to the sequence of the target sampling weights until the accumulated target sampling weight reaches the random number when the target example is reached; and taking the target instance as a new instance. For example, continuing with the above example of class t, the target sampling weight of instance i in subclass t is

The target sampling weight of instance j in subclass t is

The target sampling weights are ordered from small to large as

If the random number is 0.5, the target sampling weight of the first instance is 0.4, which does not reach 0.5, the target sampling weight of the second instance is accumulated, the accumulated target sampling weight is 1, which reaches 0.5, the instance with the target sampling weight of 0.6 is determined as the target instance, and the target instance is taken as a newly-added instance.

In this embodiment, the sampling score of each instance of the subclass is determined by a weighted random sampling method WRS, and the sampling score of an instance with a higher target sampling weight is larger. By taking the target instance with the largest sampling score as the new added instance, the instance with the heavier target sampling weight is easier to sample, the new added instance is mainly the instance with more target class labels, and the imbalance of each class is reduced.

And determining the accumulated values one by one in the ordering from small to large according to the target sampling weight by a naive weighted sampling method, wherein the accumulated value one by one of the examples is larger when the target sampling weight is larger. By taking the target instance with the accumulated values reaching the random number one by one as the new added instance, the instance with the higher target sampling weight is easier to sample, the new added instance mainly comprises more target class labels, and the imbalance of each class is reduced.

S206: and constructing the subclass by taking the target instance determined according to the weighted sampling result as a new added instance.

In this embodiment, the add instance step is repeated multiple times to construct the subclass. Specifically, the step of adding the new instance, that is, step S205, is repeated a plurality of times, and the new instance is added to the original subclass to construct the subclass. For example, continuing with the description of the subclass t, which includes the instance i and the instance j, if the new instance includes the instance k and the instance l, the constructed subclass includes: example i, example j, example k, and example l.

In this embodiment, the method may further train the initial AI model according to the example to obtain a target model.

The method comprises the steps of obtaining a sample set comprising a plurality of examples, wherein each example comprises one or more category labels; classifying the instances in the sample set according to the class labels to form a plurality of classes, wherein each instance is classified into one or more classes, and each class corresponds to one class; determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in each subclass and the number of all class labels in each instance, wherein the subclass is a plurality of classes of which the number is smaller than a first threshold value, the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is a sampling weight corresponding to the subclass in each instance; according to the target sampling weight of each instance in the subclass, weighted sampling is carried out on the instance of each subclass, only the instance can be newly added, the original instance can be completely reserved, the instance is not lost, meanwhile, the more the number of the target class labels in the subclass is compared with the number of all class labels, the more the instance is easy to be up-sampled, the influence on other classes is small, and scene loss and indirect increase of the number of other classes can be avoided.

Fig. 6 is a schematic flow chart of a multi-tag class balancing method according to a second embodiment of the present invention.

The method may be specifically executed by the training device 120 shown in fig. 1, where sample sets of multiple instances in the method may be training data maintained in the database 130 shown in fig. 1, optionally, steps S602 to S612 of the method may be executed in the training device 120, or may be executed in advance by other functional modules before the training device 120, that is, training data received or acquired from the database 130 is preprocessed, and a new sample set is obtained as an input to the training device 120 through the sampling process described in steps S602 to S612, and training of a model is executed by the training device.

The method includes, but is not limited to, the steps of:

s601: a sample set is obtained that includes a plurality of instances, each instance including one or more category labels.

Step S601 in the second embodiment is similar to step S201 in the first embodiment, and please refer to fig. 2 for detailed description of step S201 in the first embodiment, which is not repeated herein.

S602: and classifying the instances in the sample set according to the class labels to form a plurality of classes, wherein each instance is classified into one or more classes, and each class corresponds to one class.

Step S602 in the second embodiment is similar to step S202 in the first embodiment, and please refer to fig. 2 for detailed description of step S202 in the first embodiment, which is not repeated herein.

S603: a subclass of a plurality of the classes is determined.

The process of determining the subclasses in the multiple classes in the second embodiment is similar to the process of determining the subclasses in the multiple classes in step S203 in the first embodiment, and specific reference is made to detailed description of step S203 in the first embodiment in fig. 2, which is not repeated herein.

In this embodiment, an iterative sampling manner is adopted, and after each sampling cycle, the number of instances of one or more classes is changed, so that in each sampling cycle, a target number of instances needs to be preset, and a plurality of subclasses in the classes are determined.

S604: and determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in the subclass and the number of all class labels in each instance, wherein the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is the sampling weight corresponding to the subclass in each instance.

Step S604 in the second embodiment is similar to step S204 in the first embodiment, and please refer to fig. 2 for detailed description of step S204 in the first embodiment, which is not repeated herein.

S605: and carrying out weighted sampling on the examples of each subclass according to the target sampling weight of each example in the subclass.

Step S605 in the second embodiment is similar to step S205 in the first embodiment, and please refer to fig. 2 for detailed description of step S205 in the first embodiment, which is not repeated herein.

S606: and constructing the subclass by taking the target instance determined according to the weighted sampling result as a new added instance.

Step S606 in the second embodiment is similar to step S206 in the first embodiment, and please refer to fig. 2 for detailed description of step S206 in the first embodiment, which is not repeated herein.

S607: the first number is determined.

The determining the first number comprises: determining the first number to be an average of the number of current instances of all classes. The first number may also be other numbers, such as a value slightly larger than the average of the number of instances in each category, or slightly smaller than the average of the number of instances in each category. The first number may be the same as the preset target number of instances or may be different from the preset target number of instances.

In this embodiment, an iterative sampling manner is adopted, and after each sampling cycle, the number of instances of one or more classes is changed, so that the first number needs to be determined during each sampling cycle.

S608: construct instances in the subclass to a first number.

In the present embodiment, constructing the instances in the subclass to the first number includes:

judging whether the number of the instances in the subclass reaches a first number;

if the number of the instances in the subclass does not reach the first number, continuing to perform weighted sampling on the instances of each subclass according to the target sampling weight of each instance in the subclass to construct the subclass until the number of the instances in the subclass reaches the first number, namely continuing to perform steps S605 to S606 until the number of the instances in the subclass reaches the first number.

Specifically, before generating a random number in a preset interval for each instance of the subclass, the method further includes: and determining the number of instances to be constructed according to the number of the current instances of the subclass and the first number aiming at the subclass. The number of instances to be constructed may be the difference between the first number and the number of current instances of the subclass. For example, for a subclass t, assuming that the number of instances is n, the number of instances that the subclass t needs to construct is delta avg-n. Wherein, delta is the number of the examples to be constructed, avg is the first number, and n is the number of the current examples of the subclass t. In this embodiment, it is determined whether the number of instances in the subclass reaches a first number, and if the number of instances in the subclass does not reach the first number, the weighted sampling is repeatedly performed on the instances in the subclass to obtain delta new added instances. Thus, the number of instances in each subclass can be made to reach the first number. Taking the above fig. 4 as an example, a schematic diagram after weighted sampling is performed on each subclass is shown in fig. 7. In fig. 7, the number of instances of all subclasses, such as signs, buses, trains, riders, motorcycles, and bicycles, is increased to a first number.

S609: a second quantity is determined.

The determining the second number comprises: determining the second quantity as r N _max Where r is [0.5,0.8 ]]，N _max Is the number of instances of the maximum class. The r is not limited to the above-mentioned values, but may be other values such as 0.48, 0.81, and the like. If the second number is too large, the number of repeated samples in a new sample set is easily large, and overfitting is easily caused, and if the second number is too small, the equalization effect of each category is not obvious easily.

In this embodiment, an iterative sampling manner is adopted, and after each sampling cycle, the number of instances of one or more classes is changed, and then the second number needs to be determined during each sampling cycle.

S610: it is determined whether the number of instances in each subclass reaches a second number.

S611: if the number of the instances in each subclass does not reach the second number, continuously classifying the instances in the sample set according to the class labels to form a plurality of classes, continuously determining the target sampling weight of each instance in each subclass, continuously performing weighted sampling on the instances of each subclass, and continuously constructing the subclass by taking the target instances determined according to the weighted sampling result as newly-added instances until the number of the instances in each subclass reaches the second number.

In the present embodiment, if the number of instances in the subclass does not reach the second number, steps S602-S610 are repeatedly performed, so that the number of instances in each subclass can reach the second number. Taking the above-mentioned fig. 4 as an example, a schematic diagram of constructing a subclass in each subclass iteration is shown in fig. 8. In fig. 8, after 3-round sampling, the number of instances of all subclasses, such as trucks, traffic lights, signs, buses, trains, riders, motorcycles, and bicycles, is increased to a second number.

S612: if the number of instances in each subclass reaches a second number, the instances in all classes are output.

In this embodiment, the method may further train the initial AI model according to the example to obtain the target model.

The method comprises the steps of obtaining a sample set comprising a plurality of examples, wherein each example comprises one or more category labels; classifying the instances in the sample set according to the class labels to form a plurality of classes, wherein each instance is classified into one or more classes, and each class corresponds to one class; determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in each subclass and the number of all class labels in each instance, wherein the subclass is a plurality of classes of which the number is smaller than a first threshold value, the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is a sampling weight corresponding to the subclass in each instance; according to the target sampling weight of each example in the subclass, carrying out weighted sampling on the example of each subclass; constructing the subclass by taking the target instance determined according to the weighted sampling result as a newly added instance; constructing instances in the subclass to a first number; determining whether the number of instances in each subclass reaches a second number; if the number of the instances in each subclass does not reach the second number, continuously classifying the instances in the sample set according to the class labels to form a plurality of classes, continuously determining the target sampling weight of each instance in each subclass, continuously performing weighted sampling on the instances of each subclass, and continuously constructing the subclass by using the target instances determined according to the weighted sampling result as new instances until the number of the instances in each subclass reaches the second number, only the instances can be newly added, the original instances can be completely reserved without losing the instances, meanwhile, the instances with the larger number of the target class labels in the subclass relative to the number of all the class labels can be easily up-sampled, the influence on other classes is small, and based on an iterative sampling strategy, the preset target instance number, the first number and the second number can be adjusted during each sampling round, the iteration can be quitted only when the number of the examples in each category reaches the second number, and the influence on other categories is further avoided, so that the scene loss and the indirect increase of the number of other categories can be avoided, the original data is reserved, and the problem of category imbalance is solved.

Referring to fig. 9, fig. 9 is a schematic structural diagram of a multi-tag class equalizing apparatus according to an embodiment of the present invention, where the multi-tag class equalizing apparatus 900 may include an obtaining unit 901 and a sampling unit 902.

The multi-label class equalizing apparatus 900 may be configured to perform the steps of the multi-label class equalizing method according to the embodiment of the present application. For example, the obtaining unit 901 may be configured to perform step S201 in the method shown in fig. 2, and the sampling unit 902 may be configured to perform step S202 to step S206 in the method shown in fig. 2. For another example, the obtaining unit 901 may be configured to perform step S601 in the method shown in fig. 6, and the sampling unit 902 may be configured to perform steps S602 to S612 in the method shown in fig. 6.

The multi-label class equalizing apparatus 900 may also include a training unit 903. The training unit 903 is configured to train the initial AI model according to the example to obtain a target model.

Referring to fig. 10, fig. 10 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention. The electronic apparatus 1000 shown in fig. 10 includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. The memory 1001, the processor 1002, and the communication interface 1003 are communicatively connected to each other via a bus 1004.

The Memory 1001 may be a Read Only Memory (ROM), a static Memory device, a dynamic Memory device, or a Random Access Memory (RAM). The memory 1001 may store a program, and when the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to perform the steps of the multi-tag class balancing method according to the embodiment of the present application.

The processor 1002 may be a general-purpose Central Processing Unit (CPU), a microprocessor, an Application Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU), or one or more Integrated circuits, and is configured to execute related programs to implement the functions required to be executed by the units in the multi-tag class equalization apparatus according to the embodiment of the present Application, or to execute the multi-tag class equalization method according to the embodiment of the present Application.

The processor 1002 may also be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the multi-tag class equalization method of the present application may be implemented by integrated logic circuits of hardware or instructions in the form of software in the processor 1002. The processor 1002 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in ram, flash, rom, prom, or eprom, registers, etc. as is well known in the art. The storage medium is located in the memory 1001, and the processor 1002 reads information in the memory 1001, and completes, in combination with hardware of the processor, a function that needs to be executed by a unit included in the multi-tag class equalization apparatus according to the embodiment of the present application, or executes the multi-tag class equalization method according to the embodiment of the present application.

The communication interface 1003 enables communication between the electronic device 1000 and other devices or communication networks using transceiver means, such as, but not limited to, transceivers. For example, data (such as a sample set including multiple instances in the first and second embodiments of the present application) may be obtained through the communication interface 1003.

Bus 1004 may include a pathway to transfer information between various components of electronic device 1000 (e.g., memory 1001, processor 1002, communication interface 1003).

Optionally, the electronic device 1000 may further include an output component, for example, a display, a sound device, and the like, where the output component is used to show the parameters to be used by the training model to the developer, so that the developer may know the parameters, modify the parameters, and input the modified parameters into the electronic device 1000 through an input component (e.g., a mouse, a keyboard, and the like). In addition, the electronic device 1000 may further display the trained target model to the developer through the output component.

It should be understood that the obtaining unit 901 in the multi-tag class equalizing apparatus 900 may correspond to the communication interface 1003 in the electronic device 1000, and the sampling unit 902 may correspond to the processor 1002.

Referring to fig. 11, fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, where the electronic device 1100 includes a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104. The memory 1101, the processor 1102 and the communication interface 1103 are communicatively connected to each other through a bus 1104.

The Memory 1101 may be a Read Only Memory (ROM), a static Memory device, a dynamic Memory device, or a Random Access Memory (RAM). The memory 1101 may store a program, and the processor 1102 and the communication interface 1103 are used to perform the steps of the object detection method when the program stored in the memory 1101 is executed by the processor 1102.

The processor 1102 may be a general-purpose Central Processing Unit (CPU), a microprocessor, an Application Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU), or one or more Integrated circuits, and is configured to execute related programs to perform the target detection method.

The processor 1102 may also be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the object detection method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 1102. The processor 1102 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 1101, and the processor 1102 reads the information in the memory 1101, and completes the target detection method by combining the hardware thereof.

The communication interface 1103 enables communication between the electronic apparatus 1100 and other devices or communication networks using transceiver means, such as, but not limited to, a transceiver. For example, the communication interface 1103 establishes a communication connection with the training device, and the parameters of the target model sent by the training device may be received through the communication interface 1103. The parameters of the object model may be stored in memory 1101 for invocation.

Bus 1104 may include a path that conveys information between various components (e.g., memory 1101, processor 1102, communication interface 1103) of electronic device 1100.

Optionally, the electronic device 1100 may further include an output component, such as a display, a sound, and the like, for displaying the target detection result obtained by the target model to a user.

The target detection method may be any target detection method in the prior art, and is not described herein again.

It should be noted that although the electronic device 1000 and the electronic apparatus 1100 shown in fig. 10 and 11 only show memories, processors, and communication interfaces, in a specific implementation process, those skilled in the art should understand that the electronic device 1000 and the electronic apparatus 1100 also include other components necessary for normal operation. Also, it will be understood by those skilled in the art that the electronic device 1000 and the electronic apparatus 1100 may also include hardware components that implement other additional functions, according to particular needs. Furthermore, those skilled in the art will appreciate that the device electronics 1000 and the electronic device 1100 may also include only those components necessary to implement the embodiments of the present application, and need not include all of the components shown in FIG. 10 or FIG. 11.

It is understood that the electronic device 1000 corresponds to the training device 120 in fig. 1, and the electronic apparatus 1100 corresponds to the executing device 110 in fig. 1. Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In addition to the above method and apparatus, the embodiment of the present invention further provides a computer-readable storage medium, which stores instructions that, when executed on a processor, implement the multi-label class balancing method shown in fig. 2 or fig. 6.

A computer program product comprising computer executable instructions, the computer executable instructions being stored in a computer readable storage medium; the computer-executable instructions may be read from the computer-readable storage medium by at least one processor of a device, and execution of the computer-executable instructions by the at least one processor causes the device to implement the multi-tag class-balancing method shown in fig. 2 or fig. 6.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A multi-label class balancing method, comprising:

obtaining a sample set comprising a plurality of instances, each instance comprising one or more category labels;

classifying the instances in the sample set according to the class labels to form a plurality of classes, wherein each instance is classified into one or more classes, and each class corresponds to one class;

determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in each subclass and the number of all class labels in each instance, wherein the subclass is a plurality of classes of which the number is smaller than a first threshold value, the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is a sampling weight corresponding to the subclass in each instance;

and carrying out weighted sampling on the examples of each subclass according to the target sampling weight of each example in the subclass.

2. The multi-label class balancing method of claim 1, further comprising:

and constructing the subclass by taking the target instance determined according to the weighted sampling result as a new added instance.

3. The multi-label class balancing method of claim 2, further comprising:

constructing instances in the subclass to a first number;

determining whether the number of instances in each subclass reaches a second number;

if the number of the instances in each subclass does not reach the second number, continuously classifying the instances in the sample set according to the class labels to form a plurality of classes, continuously determining the target sampling weight of each instance in each subclass, continuously performing weighted sampling on the instances of each subclass, and continuously constructing the subclass by taking the target instances determined according to the weighted sampling result as newly-added instances until the number of the instances in each subclass reaches the second number.

4. The multi-labeled class balancing method of claim 3, wherein prior to said constructing the instances in the subclass to the first number, the method further comprises:

determining the first number to be an average of the number of current instances of all classes.

5. The multi-labeled class balancing method of claim 3, wherein before the determining whether the number of instances in each subclass reaches the second number, the method further comprises:

determining the second quantity as r x N _max Where r is [0.5,0.8 ]]，N _max Is the number of instances of the maximum class.

6. The method for multi-label class balancing according to any of the claims 1-5, characterized in that:

the weighted sampling comprises at least one of weighted random sampling method WRS and naive weighted sampling method.

7. The method for multi-label class balancing according to any of the claims 1-6, wherein determining the target sampling weight for each instance in each subclass based on the number of target class labels in each instance in the subclass and the number of all class labels in each instance comprises:

determining the proportion of the target class labels of each example in each subclass according to the number of the target class labels in each example in the subclass and the number of all the class labels in each example;

and determining the target sampling weight of each example in each subclass according to the occupation ratio of the target class label of each example in each subclass.

8. The method of claim 7, wherein determining the fraction of the target class labels for each instance in each subclass based on the number of target class labels in each instance in the subclass and the number of all class labels in each instance comprises:

according to the formula

9. The multi-labeled class balancing method according to claim 7, wherein the determining the target sampling weight of each instance in each subclass according to the duty ratio of the target class label of each instance in each subclass comprises:

according to the formula

is the sum of the target class labels of all instances in the subclass t.

10. A multi-tag class equalization apparatus, comprising:

an obtaining unit for obtaining a sample set comprising a plurality of instances, each instance comprising one or more category labels;

a sampling unit, configured to classify the instances in the sample set according to the class labels to form a plurality of classes, where each instance is classified into one or more classes, and each class corresponds to a class;

the sampling unit is used for determining a target sampling weight of each instance in each subclass according to the number of target class labels in each instance in each subclass and the number of all class labels in each instance, wherein the subclass is a plurality of classes of which the number is smaller than a first threshold value, the target class labels are class labels corresponding to the subclass in each instance, and the target sampling weight is a sampling weight corresponding to the subclass in each instance;

the sampling unit is further configured to perform weighted sampling on the instances of each subclass according to the target sampling weight of each instance in the subclass.

11. The multi-label class equalizing apparatus of claim 10, wherein:

the sampling unit is further used for constructing the subclass by taking the target instance determined according to the weighted sampling result as a new sample.

12. The multi-label class equalizing apparatus of claim 11, wherein:

the sampling unit is further configured to construct instances in the subclass to a first number;

the sampling unit is further configured to determine whether the number of instances in each subclass reaches a second number;

the sampling unit is further configured to, if the number of instances in each subclass does not reach the second number, continue to classify the instances in the sample set according to the class labels to form a plurality of classes, continue to determine a target sampling weight for each instance in each subclass, and continue to perform weighted sampling on the instances of each subclass until the number of instances in each subclass reaches the second number.

13. The multi-labeled class equalizing apparatus of claim 12, wherein prior to said constructing instances in said subclass to a first number:

the sampling unit is further configured to determine that the first number is an average of the number of current instances of all classes.

14. The multi-label class balancing apparatus of claim 12, wherein prior to said determining whether the number of instances in each subclass reaches a second number:

the sampling unit is further configured to determine that the second quantity is r N _max Where r is [0.5,0.8 ]]，N _max Is the number of instances of the maximum class.

15. A multi-tag class equalizing apparatus according to any one of claims 10-14, wherein:

16. A multi-tag class equalizing apparatus according to any one of claims 10-15, wherein:

the sampling unit is used for determining the proportion of the target class label of each example in each subclass according to the number of the target class labels in each example in the subclass and the number of all the class labels in each example;

the sampling unit is further used for determining a target sampling weight of each example in each subclass according to the proportion of the target class label of each example in each subclass.

17. The multi-tag class equalizing apparatus of claim 16, wherein:

the sampling unit is also used for generating a formula

18. The multi-label class equalizing apparatus of claim 16, wherein:

the sampling unit is also used for calculating the formula

is the sum of the target class labels of all instances in the subclass t.

19. An electronic device comprising a processor and a memory, the memory storing program instructions which, when invoked by the processor, implement the multi-tag class balancing method of any one of claims 1 to 9.

20. A vehicle characterized in that it comprises an electronic device according to claim 19.

21. A computer-readable storage medium characterized in that it stores a program that causes a computer device to implement the multi-label class equalization method according to any one of claims 1 to 9.

22. A computer program product, comprising computer executable instructions stored in a computer readable storage medium; at least one processor of a device may read the computer-executable instructions from the computer-readable storage medium, execution of which by the at least one processor causes the device to perform the multi-tag class balancing method of any one of claims 1 to 9.