KR102547527B1 - Method and device for labeling objects - Google Patents

Abstract

An embodiment of the present invention discloses an object labeling method and apparatus. In one specific embodiment of the method, obtaining an image to be labeled including a target object; determining location information of a target object according to a pretrained first detection model and the image to be labeled; determining a size of a labeling box adapted to the target object according to the location information and a preset labeling box size set; and labeling the target object using a labeling box according to the determined size. The above embodiment eliminates the need to manually adjust the labeling box, thereby reducing the amount of manual work.

Description

Object labeling method and apparatus {METHOD AND DEVICE FOR LABELING OBJECTS}

Embodiments of the present invention relate to the field of computer technology, and specifically to an object labeling method and apparatus.

With the continuous progress of deep learning algorithms and the continuous improvement of the accuracy of pedestrian target detection algorithms, some mobile terminals may use the pedestrian target detection algorithm in deep learning to detect pedestrians. However, deep learning is characterized by having to train a model with a large amount of manually labeled data, which is a necessary condition as well as a feature, and the amount of data and the quality of the data determine the effectiveness of model detection. In the case of pedestrian detection, it is necessary to manually label the position of the person in the picture and repeat this process 100,000 or millions of times, which requires a lot of manpower and time.

Embodiments of the present invention provide an object labeling method and apparatus.

In a first aspect, an embodiment of the present invention provides an object labeling method, the method comprising: acquiring an image to be labeled including a target object; determining location information of a target object according to a pretrained first detection model and the image to be labeled; determining a size of a labeling box adapted to the target object according to the location information and a preset labeling box size set; and labeling the target object using a labeling box according to the determined size.

In some embodiments, the method further comprises: obtaining a non-labeled area of the image to be labeled; and generating a mosaic in the non-labeling area.

In some embodiments, the method further includes determining, according to the image after mosaic generation, a training sample; and acquiring a second detection model by using an image including a mosaic among the training samples as an input and using the labeling box as an expected output.

In some embodiments, the method includes obtaining labeling box information including a size of the labeling box; and obtaining a labeling box size set by performing clustering on the sizes of the labeling boxes.

In some embodiments, the acquiring of the image to be labeled and the non-labeling area may include determining, according to the labeling box, a labeling area; and determining the non-labeling area according to the labeling area.

In a second aspect, an embodiment of the present invention provides an object labeling device, the device comprising: an image acquiring unit, configured to acquire an image to be labeled including a target object; a position determination unit, configured to determine position information of a target object according to a pretrained first detection model and the image to be labeled; a size determining unit, configured to determine a size of a labeling box adapted to the target object according to the location information and a preset labeling box size set; and an object labeling unit configured to label the target object using a labeling box according to the determined size.

In some embodiments, the device obtains an unlabeled area of the image to be labeled; and a processing device configured to generate a mosaic in the unlabeled area.

In some embodiments, the device determines, according to the image after mosaic generation, a training sample; and a training unit, configured to take the image in the training sample as an input and the labeling box as an expected output, to obtain a second detection model.

In some embodiments, the device acquires labeling box information including a size of the labeling box; and a size set determining unit, configured to perform clustering on sizes of the labeling boxes to obtain a labeling box size set.

In some embodiments, the processing device further determines, according to the labeling box, a labeling area; and determine a non-labeling area according to the labeling area.

In a third aspect, an embodiment of the present invention provides an electronic device including one or a plurality of processors and a storage device in which one or a plurality of programs are stored, wherein the one or plurality of programs are stored in the one or plurality of processors When executed by said one or plurality of processors to implement the method described in any embodiment of the first aspect.

In a fourth aspect, an embodiment of the present invention provides a computer readable medium having a computer program stored thereon, which, when executed by a processor, implements a method described in any embodiment of the first aspect.

In a fifth aspect, an embodiment of the present invention provides a computer program stored in a computer readable storage medium, wherein the computer program, when executed by a processor, implements the method described in any embodiment of the first aspect.

According to the object labeling method and apparatus provided in the above embodiment of the present invention, first, an image to be labeled including a target object may be obtained. Next, location information of the target object is determined according to the pretrained first detection model and the image to be labeled. Next, the size of the labeling box for labeling the target object is determined according to the location information and a preset labeling box size set. Finally, according to the determined size, the target object is labeled using a labeling box. The method of this embodiment selects a labeling box adapted from a set of preset labeling box sizes, and uses the labeling box to label a target object, so that the labeling box does not need to be manually adjusted, reducing manual workload.

Other features, objects and advantages of the present invention will become more apparent upon reading and reference to the detailed description of the non-limiting embodiments shown in the accompanying drawings below.
1 is an exemplary system architecture in which one embodiment of the present invention may be applied.
2 is a flowchart of an embodiment of an object labeling method according to the present invention.
3 is a schematic diagram of one application scene of the object labeling method according to the present invention.
4 is a flowchart of another embodiment of an object labeling method according to the present invention.
5 is a structural schematic diagram of an embodiment of an object labeling device according to the present invention.
6 is a structural schematic diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present invention.

The present invention will be described in more detail with reference to the accompanying drawings and examples below. It will be understood that the specific embodiments described herein are only for interpreting the related invention, and the present invention is not limited thereto. In addition, it should be noted that only parts related to the invention are shown in the drawings for convenience of explanation.

It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other unless contradictory. The present invention will be described in detail with reference to the accompanying drawings below and reference to examples.

1 illustrates an exemplary system architecture 100 to which an embodiment of the object labeling method or object labeling apparatus of the present invention may be applied.

As shown in FIG. 1 , the system architecture 100 may include terminal devices 101 , 102 , and 103 , a network 104 and a server 105 . Network 104 provides a medium of communication link between terminal devices 101 , 102 , 103 and server 105 . Network 104 may include a variety of connection types, such as wired, wireless communication links, or fiber optic cables.

Users can receive or transmit messages and the like by interacting with the server 105 via the network 104 using the terminal devices 101 , 102 , and 103 . The terminal devices 101, 102, and 103 may be connected to cameras to collect images. Various communication client applications can be installed in the terminal devices 101, 102, and 103, such as image browsing type applications, adaptive browsing type applications, web browser applications, shopping type applications, search type applications, instant message tools, Mail clients, social media software, etc.

The terminal devices 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they may be various electronic devices having a display screen, including, but not limited to, smart phones, smart rearview mirrors, smart odometers, tablet PCs, portable laptop computers, and desktop computers. it is not going to be When the terminal devices 101, 102, and 103 are software, they may be installed in the above-listed electronic devices. It may be implemented as a plurality of software or software modules (for example, providing distributed services), or may be implemented as one software or software module, and is not specifically limited herein.

The server 105 may be a server that provides various services, and may be, for example, a background server that supports images collected by the terminal devices 101, 102, and 103. The background server may perform processing such as analysis on acquired data such as an image to be labeled, and may feed back a processing result (eg, a labeling result of an image) to the terminal devices 101 , 102 , and 103 .

It should be noted that a server can be hardware or software. If the server is hardware, it may be implemented as a distributed server cluster composed of a plurality of servers or may be implemented as a single server. If the server is software, it may be implemented as a plurality of software or software modules (for example, providing a distributed service), or may be implemented as one software or software module, and is not specifically limited herein.

It should be explained that the object labeling method provided by the embodiments of the present invention is generally performed by the server 105. Correspondingly, the object labeling device is generally installed in the server 105.

It should be understood that the number of terminal devices, networks and servers in FIG. 1 is only exemplary, and any number of terminal devices, networks and servers may be provided according to actual needs.

With continued reference to FIG. 2 , it illustrates a flow 200 of one embodiment of an object labeling method according to an embodiment of the present invention. The object labeling method of this embodiment includes the following steps.

In step 201, an image to be labeled is obtained.

In this embodiment, a subject performing the object labeling method (for example, the server 105 shown in FIG. 1 ) may obtain an image to be labeled through a wired connection method or a wireless connection method. The image to be labeled may be an image collected by an image collection device installed in a vehicle. The image to be labeled includes a target object, and the target object may include a pedestrian or a vehicle.

In step 202, location information of a target object is determined according to a pretrained first detection model and an image to be labeled.

In this embodiment, after the performer acquires the image to be labeled, location information of the target object may be determined using a first detection model trained in advance. The first detection model may identify a target object in an image to be labeled, which may be a variety of deep learning algorithms, for example, a deep learning neural network, a convolutional neural network, or a combination of various neural networks. The first detection model may include a plurality of convolutional layers, and hardware requirements are high when the first detection model is executed. The first detection model can be obtained by training on published pedestrian data.

An execution entity may obtain location information of a target object by inputting the image to be labeled into the first detection model. The location information may include information such as the center and edge of the target object.

In step 203, the size of a labeling box for labeling the target object is determined according to the location information and a preset labeling box size set.

After the location information of the target object is determined, the performing entity may determine the size of the labeling box adapted to the target object by associating a preset labeling box size set. The labeling box size set includes a plurality of labeling box sizes, and the plurality of labeling box sizes are sizes of labeling boxes of target objects among existing labeling images. In this way, since the size of the labeling box obtained through adaptation is relatively close to the size of the labeling box set by the user, manual adjustment is not required.

In some selectable implementation forms of this embodiment, the set of labeling box sizes includes the following steps not shown in FIG. 2 , that is, obtaining labeling box information including labeling box sizes; and performing clustering on the size of the labeling box to obtain a labeling box size set.

In this implementation form, the performing entity may obtain labeling box information. The labeling box information may be manually labeled. The performing entity first obtains previously published labeling data, and may determine each labeling box information from the labeling data. The information of the labeling box includes information such as the size and location of the labeling box. The performer may acquire a plurality of labeling box sizes by performing clustering on the size of each labeling box. Specifically, the performer may perform clustering such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN) for the size of each labeling box using an existing cluster algorithm.

In step 204, the target object is labeled using a labeling box according to the determined size.

After the size adapted to the target object is determined, the performing entity may label the target object using a labeling box. It can be understood that the size of the labeling box is the size of the labeling box obtained through adaptation.

With continuing reference to FIG. 3 , this is a schematic diagram of an application scene of the object labeling method according to the present embodiment. In the application scene of FIG. 3 , the odometer installed in the vehicle 301 collects a plurality of images in the driving environment, uploads the images to the server 302, and the server 302 has a first detection model trained in advance. is set The server 302 inputs a plurality of images to the first detection model and determines the location information of the pedestrian included in each image. Next, the size of the labeling box adapted to the pedestrian is determined. Finally, label the pedestrians using the labeling box. The server 302 may output the image after labeling to the terminal device 303 . The terminal device 303 may be a terminal used by a labeling personnel, and the labeling personnel may examine the image after labeling to determine whether the pedestrian's labeling box needs to be adjusted.

According to the object labeling method provided in the above embodiment of the present invention, first, an image to be labeled including a target object may be obtained. Next, location information of the target object is determined according to the pretrained first detection model and the image to be labeled. Next, the size of the labeling box for labeling the target object is determined according to the location information and a preset labeling box size set. Finally, according to the determined size, the target object is labeled using a labeling box. The method of this embodiment selects a labeling box adapted from a set of preset labeling box sizes, and uses the labeling box to label a target object, so that the labeling box does not need to be manually adjusted, reducing manual workload.

With continued reference to FIG. 4 , it illustrates a flow 400 of another embodiment of a method for object labeling in accordance with the present invention. As shown in FIG. 4 , the object labeling method of this embodiment may include the following steps.

In step 401, an image to be labeled is obtained.

In step 402, location information of a target object is determined according to a pretrained first detection model and an image to be labeled.

In step 403, a size of a labeling box adapted to the target object is determined according to the location information and a preset labeling box size set.

In step 404, the target object is labeled using a labeling box according to the determined size.

In step 405, an unlabeled area of an image to be labeled is obtained.

In this embodiment, the performing entity may acquire a non-labeling area of an image to be labeled. The unlabeled area may include a blurry and indistinct target object or may not include a target object. Correspondingly, an area excluding a non-labeling area among images to be labeled may be referred to as a labeling area. The labeling area includes a plurality of target objects to be labeled, and a performing entity may obtain a labeling area of an image to be labeled from a labeling person. A labeling person may obtain an image after labeling, select a labeling area from the image, and transmit the image after selecting the labeling area to a performing entity. A portion other than the labeling area is the non-labeling area.

In some selectable implementation forms of this embodiment, the performing entity may perform the following steps not shown in FIG. 4 , namely, determining a labeling area according to a labeling box; and determining a non-labeling area according to the labeling area. The non-labeling area may be determined.

In this implementation mode, the performing entity may determine an enclosing frame surrounding each labeling box, set an area within the enclosing frame as a labeling area, and set an area other than the enclosing frame as a non-labeling area.

In step 406, a mosaic is created in the non-labeled area.

In this embodiment, after the non-labeling area is determined, the performer may create a mosaic in the non-labeling area. Specifically, the performer can shuffle the pixel positions of the non-labeled area on a block-by-block basis, so that the non-labeled area does not particularly protrude compared to surrounding pixels, resulting in a clearer image and better training effect.

In step 407, a training sample is determined according to the image after mosaic generation.

The performer may use the mosaic-generated image as a training sample. After the number of training samples reaches a preset number, the model may be trained using a plurality of training samples.

In step 408, a second detection model is obtained by taking an image including a mosaic among training samples as an input and a labeling box as an expected output.

In this embodiment, the performer may obtain a second detection model by inputting an image including a mosaic but not a labeling box among training samples as a model and inputting the labeling box as an expected output of the model. Preferably, the number of convolutional layers included in the second detection model is relatively small, and the amount of calculation is relatively small. In this way, the second detection model after training can be installed in a mobile terminal with a low hardware level, for example, in a smart rearview mirror. It is installed to realize the rapid detection of pedestrians.

The object labeling method provided in the usual embodiments of the present invention can obtain a labeling image of good quality and generate it by training a second detection model, which is advantageous for implementing a deep learning algorithm.

Further, referring to FIG. 5 , which is an implementation of the method shown in each of the above drawings, the present invention provides an embodiment of an object labeling device, and the device embodiment corresponds to the method embodiment shown in FIG. 2 . , The device can be specifically applied to various electronic devices.

As shown in FIG. 5 , an object labeling apparatus 500 of this embodiment includes an image acquisition unit 501, a position determination unit 502, a size determination unit 503, and an object labeling unit 504.

The image acquisition unit 501 is configured to acquire an image to be labeled containing a target object.

The position determination unit 502 is configured to determine position information of a target object according to a first detection model trained in advance and an image to be labeled.

The size determination unit 503 is configured to determine the size of the labeling box adapted to the target object according to the location information and the preset labeling box size set.

The object labeling unit 504 is configured to label the target object using the labeling box according to the determined size.

In some optional implementation forms of this embodiment, the device 500 may further include a processing device, not shown in FIG. 5 , configured to obtain a non-labeled area of an image to be labeled, and generate a mosaic in the non-labeled area. can

In some optional implementation forms of this embodiment, the apparatus 500 may further include a training unit, not shown in FIG. 5 , configured to obtain a non-labeled area of an image to be labeled, and generate a mosaic in the non-labeled area. can

In some optional implementation forms of this embodiment, the apparatus 500 acquires labeling box information including a size of the labeling box; It may further include a size set determining unit not shown in FIG. 5, configured to perform clustering on sizes of labeling boxes to obtain a labeling box size set.

In some optional implementation forms of this embodiment, the processing device also determines, according to the labeling box, a labeling area; and determine a non-labeling area according to the labeling area.

It should be understood that the units 501 to 504 described in the object labeling apparatus 500 respectively correspond to respective steps in the method described with reference to FIG. 2 . From this, the operations and features described for the object labeling method are similarly applied to the device 500 and the units included therein, and thus are not further described herein.

Referring to FIG. 6 below, it is a structural schematic diagram that can implement an electronic device 600 (for example, a server or a terminal device of FIG. 1) according to an embodiment of the present invention. The electronic device shown in FIG. 6 is only an example, and is not intended to limit any function or scope of use of the embodiment of the present invention.

As shown in FIG. 6, the electronic device 600 varies according to a program stored in a read-only memory (ROM) 602 or a program loaded into a random access memory (RAM) 603 from a storage device 608 and It may include a processing unit 601 (eg, central processing unit, graphics processing unit, etc.) capable of performing appropriate operations and processing. The RAM 603 also stores various programs and data necessary for operating the electronic device 600 . The processing unit 601 , ROM 602 and RAM 603 are connected to each other via a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604.

Input devices 606 generally include, for example, touch screens, touch pads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, and the like; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage device 608 including, for example, magnetic tape, hard drives, etc.; and communication device 609 can be coupled to I/O interface 605 . The communication device 609 may allow the electronic device 600 to exchange data by communicating with another device wirelessly or wired. Although FIG. 6 shows an electronic device 600 with various devices, it should be understood that not all illustrated devices need be implemented or equipped. More or fewer devices may alternatively be implemented or provided. Each block shown in FIG. 6 may represent one device, or may represent a plurality of devices according to demand.

In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart may be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product including a computer program embodied in a computer readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication device 609 or installed from storage device 608 or ROM 602 . When the computer program is executed by the processing device 601, the functions defined in the method of the embodiment of the present invention are performed. It should be noted that a computer readable medium according to an embodiment of the present invention may be a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or element, or any combination thereof. More specific examples of computer-readable storage media include electrical connection by one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination thereof.

The computer readable medium may be included in the electronic device or may exist separately without being assembled in the electronic device. obtaining, by the electronic device, an image to be labeled including a target object when one or more programs are loaded on the computer readable medium and the one or more programs are executed by the electronic device; determining location information of a target object according to a pretrained first detection model and an image to be labeled; determining a size of a labeling box adapted to a target object according to location information and a preset labeling box size set; and labeling the target object using the labeling box according to the determined size.

Computer program codes for performing operations of embodiments of the present invention may be written in one or more programming languages, or a combination thereof. The programming languages include object oriented programming languages including Java, Smalltalk, C++ and conventional procedural programming languages including the "C" language or similar programming languages. The program code may run entirely on the user's computer, partly on the user's computer, as a standalone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. can be executed In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a LAN or WAN, or to an external computer (for example, through the Internet using an Internet Service Provider).

The flow diagrams and block diagrams in the drawings illustrate the implementable architectures, functions and operations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block of a flowchart or block diagram may represent a module, program segment, or portion of code that includes one or more executable instructions for implementing a specified logical function. It should be noted that in some alternative implementations, the functions indicated in the blocks may also be implemented in a different order than shown in the figures. For example, two blocks presented sequentially may actually be executed in parallel, and sometimes in reverse order, depending on the relevant function, which is determined by the relevant function. In addition, each block in the block diagram and/or flowchart and the combination of blocks in the block diagram and/or flowchart may be implemented in a dedicated hardware-based system that performs a designated function or operation, or may be implemented by combining dedicated hardware and computer instructions. It should be noted that there are

The units described in the embodiments of the present invention may be implemented by software or hardware. The described unit may also be installed in a processor, for example a processor comprising an image acquisition unit, a position determination unit, a size determination unit and an object labeling unit. Here, the names of these units are not limited to the units themselves in some cases, and for example, the image acquisition unit may be described as "a unit that acquires an image to be labeled".

The above description is merely a description of the preferred embodiment of the present invention and the principles of applied technology. Those skilled in the art to which the present invention pertains, the scope of the present invention mentioned in the embodiments of the present invention is not limited to the technical solutions according to the specific combination of the above technical features, and at the same time, the above without departing from the spirit of the present invention Other technical solutions formed by arbitrarily combining technical features or equivalent features, for example, technical solutions formed by exchanging the features and technical features having similar functions disclosed in the present invention (but not limited thereto) with each other. It should be understood that it includes means.

Claims (13)

As an object labeling method,
obtaining an image to be labeled including a target object;
determining location information of a target object according to a pretrained first detection model and the image to be labeled;
determining a size of a labeling box adapted to the target object according to the location information and a preset labeling box size set; and
Labeling the target object using a labeling box according to the determined size
including,
Acquiring labeling box information including a size of the labeling box; and
Obtaining a set of labeling box sizes by performing clustering on the sizes of the labeling boxes
Further comprising, object labeling method. The method of claim 1, wherein the method,
acquiring a non-labeling area of the image to be labeled; and
Generating a mosaic in the non-labeling area
Further comprising, object labeling method. The method of claim 2, wherein the method,
determining a training sample according to the image after mosaic generation; and
Acquiring a second detection model using the image in the training sample as an input and the labeling box as an expected output
Further comprising, object labeling method. delete According to claim 2,
The step of acquiring the image to be labeled and the unlabeled area,
determining a labeling area according to the labeling box; and
Determining the non-labeling area according to the labeling area
Including, object labeling method. As an object labeling device,
an image acquisition unit configured to acquire an image to be labeled containing a target object;
a position determination unit, configured to determine position information of a target object according to a pretrained first detection model and the image to be labeled;
a size determining unit, configured to determine a size of a labeling box adapted to the target object according to the location information and a preset labeling box size set; and
An object labeling unit configured to label the target object using a labeling box according to the determined size.
including,
The apparatus further comprises a size set determining unit, wherein the size set determining unit comprises:
obtaining labeling box information including the size of the labeling box;
and perform clustering on the sizes of the labeling boxes to obtain a set of labeling box sizes. According to claim 6,
The device further comprises a processing device, wherein the processing device comprises:
obtain a non-labeled area of the image to be labeled;
Object labeling apparatus configured to generate a mosaic in the non-labeling area. According to claim 7,
The apparatus further comprises a training unit, wherein the training unit comprises:
Determine a training sample according to the image after mosaic generation;
and obtain a second detection model by taking the image in the training sample as an input and the labeling box as an expected output. delete The method of claim 7, wherein the processing device further,
determine a labeling area according to the labeling box;
An object labeling apparatus, configured to determine a non-labeling area according to the labeling area. As an electronic device,
one or more processors; and
Including a storage device in which one or a plurality of programs are stored,
When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3 and 5. . A computer readable medium in which a computer program is stored,
A computer readable medium which, when the program is executed by a processor, implements the method according to any one of claims 1 to 3 and 5. A computer program stored on a computer readable storage medium,
A computer program which, when executed by a processor, causes the computer program to implement the method according to any one of claims 1 to 3 and 5.

Images