JP7249372B2 - Methods and apparatus, electronic devices, computer readable storage media and computer programs for labeling objects - Google Patents

Description

TECHNICAL FIELD Embodiments of the present application relate to the field of computer technology, and in particular to methods and apparatus, electronic devices, computer-readable storage media and computer programs for labeling objects.

With the continuous progress of deep learning algorithms, the accuracy of pedestrian target detection algorithms continues to improve, and some mobile terminals use pedestrian target detection algorithms in deep learning to detect pedestrians. is now possible. However, it is a characteristic as well as a prerequisite of deep learning that a large amount of manually labeled data is required to train the model, and the amount and quality of the data determine the effectiveness of model detection. For pedestrian detection, the position of each person in the photo needs to be manually labeled, and the process needs to be repeated 100,000 or even 1,000,000 times, which requires a lot of manpower and time input. become.

Embodiments of the present application provide methods and apparatus for labeling objects.

In a first aspect, embodiments of the present application provide a method for labeling an object, the method comprising the steps of obtaining a labeling object image comprising a target object, a pre-trained first detection model and determining location information of a target object based on a labeling target image; determining a size of a labeling frame adapted to the target object based on the location information and a predetermined labeling frame size set; and determining the determined size. labeling the target object with a labeling frame based on .

In some embodiments, the method further includes obtaining non-labeled regions of the image to be labeled and generating a mosaic in the non-labeled regions.

In some embodiments, the method further comprises determining training samples based on the images after the mosaic has been generated; as the expected output and obtaining a second detection model.

In some embodiments, the method further includes obtaining labeling frame information including labeling frame sizes, and clustering the labeling frame sizes to obtain a labeling frame size set.

In some embodiments, obtaining the labeling region and the non-labeling region of the image to be labeled includes: determining the labeling region based on the labeling frame; and determining the non-labeling region based on the labeling region. and determining.

In a second aspect, embodiments of the present application provide an apparatus for labeling an object, an image acquisition unit configured to acquire a labeling object image comprising a target object and a pre-trained first detection a position determination unit configured to determine position information of a target object by means of a model and an image of said labeling object; and a size of a labeling frame adapted to said target object based on said position information and a predetermined set of labeling frame sizes. and an object labeling unit arranged to label the target object with a labeling frame based on the determined size.

In some embodiments, the apparatus further comprises a processing device configured to obtain non-labeled regions of the image to be labeled and generate a mosaic in the non-labeled regions.

In some embodiments, the apparatus determines training samples based on the images after the mosaic is generated, the images in the training samples as input, the labeling frames as expected output, and a second detection. Further includes a training unit configured to obtain the model.

In some embodiments, the apparatus further comprises a size set determining unit configured to obtain labeling frame information including sizes of labeling frames, cluster the sizes of the labeling frames, and obtain a labeling frame size set. include.

In some embodiments, the size set determining unit is further configured to determine a labeling region based on the labeling frame and a non-labeling region based on the labeling region.

In a third aspect, embodiments of the present application provide an electronic device, including one or more processors and a storage device storing one or more programs, wherein the one or more programs are Execution on said one or more processors causes said one or more processors to implement the method according to any embodiment of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer-readable medium, wherein a computer program is stored on the computer-readable medium, the method according to any embodiment of the first aspect when the program is executed by a processor. Realize

In a fifth aspect, embodiments of the present application provide a computer program which, when executed by a processor, implements the method according to any embodiment of the first aspect.

A method and apparatus for labeling an object according to the above embodiments of the present application may first acquire a labeling object image containing the target object. Then, position information of the target object is determined based on the pre-trained first detection model and the labeling object image. Then, determining the size of the labeling frame for labeling the target object based on the location information and the predetermined labeling frame size set. Finally, the target object is labeled using the labeling frame based on the determined size. The method of this embodiment selects a suitable labeling frame from a predetermined labeling frame size set, and labels the target object with the labeling frame, thereby eliminating the need for manual adjustment of the labeling frame and manual work. load is reduced.

Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of the non-limiting embodiments made with reference to the following drawings.
1 is an exemplary system architecture diagram in which one embodiment of the present application may be applied; FIG. 1 is a flow chart of one embodiment of a method for labeling an object according to the present application; 1 is a schematic diagram of one application scene of the method for labeling an object according to the present application; FIG. FIG. 4 is a flowchart of another embodiment of a method for labeling objects according to the present application; FIG. 1 is a structural schematic diagram of one embodiment of an apparatus for labeling an object according to the present application; FIG. 1 is a configuration schematic diagram of a computer system applied to an electronic device for realizing an embodiment of the present application; FIG.

Hereinafter, the present application will be described in more detail in combination with drawings and examples. It should be understood that the specific examples described herein are for the purpose of interpreting the related invention only, and are not intended to limit the same. For convenience of explanation, only the parts related to the invention are shown in the drawings.

It should be noted that, as long as there is no contradiction, the embodiments in the present application and the features in the embodiments can be combined with each other. Hereinafter, the present application will be described in detail by combining embodiments with reference to the drawings.

FIG. 1 shows an exemplary system architecture 100 to which embodiments of methods for labeling objects or apparatus for labeling objects according to the present application can be applied.

As shown in FIG. 1, system architecture 100 may include terminals 101 , 102 , 103 , network 104 and server 105 . Network 104 is an intermediary for providing communication links between terminal devices 101 , 102 , 103 and server 105 . Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

Users can send and receive messages and the like by interacting with the server 105 via the network 104 using the terminal devices 101 , 102 , 103 . Terminals 101, 102, 103 can be connected to cameras to capture images. Various communication client applications such as image browsing applications, adaptive browsing applications, web page browser applications, shopping applications, search applications, instant messaging tools, e-mail clients, social platforms, etc. Software etc. can be installed.

The terminal devices 101, 102 and 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices with display screens, including smart phones, smart rear view mirrors, smart driving recorders, tablets, laptops, desktop computers, etc. but not limited to these. If the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It may be implemented as multiple pieces of software or software modules (eg, for providing distributed services) or as a single piece of software or software module. It is not specifically limited here.

Server 105 may be a server that provides various services, such as a back-end server that provides support for images collected at terminals 101 , 102 , 103 . The back-end server can perform processing such as analysis on the acquired data such as the image to be labeled, and feed back the processing result (for example, the image labeling result) to the terminal devices 101 , 102 , and 103 .

Note that the server may be hardware or software. If the server is hardware, it may be implemented as a distributed server cluster of multiple servers, or it may be implemented as a single server. If the server is software, it may be implemented as multiple software or software modules (eg, for providing distributed services) or as a single software or software module. It is not specifically limited here.

Also, the method for labeling objects according to embodiments of the present application is typically performed by the server 105 and correspondingly the apparatus for labeling objects is typically provided on the server 105 .

It should be understood that the numbers of terminal equipment, networks and servers in FIG. 1 are merely exemplary. It may have any number of terminal devices, networks and servers according to actual needs.

With continued reference to FIG. 2, FIG. 2 illustrates a process 200 of one embodiment of a method for labeling objects according to the present application. The method for labeling an object of this example includes the following steps.
At step 201, an image to be labeled is acquired.

In this embodiment, the entity executing the method for labeling the object (for example, the server 105 shown in FIG. 1) can acquire the labeling object image through a wired connection method or a wireless connection method. The image to be labeled may be an image acquired by an image acquisition device attached to the vehicle. The image to be labeled includes a target object, and the target object may include a pedestrian, a vehicle, or the like.

In step 202, the location information of the target object is determined based on the pre-trained first detection model and the labeling object image.

In this embodiment, after obtaining the image to be labeled, the agent can use the pre-trained first detection model to determine the location information of the target object. The first detection model can recognize the target object in the labeling target image, and may be various deep learning algorithms such as a deep neural network and a convolutional neural network. A combination or the like may be used. The first detection model may include multiple convolutional layers and is hardware demanding to run the first detection model. A first detection model can be obtained by training with public pedestrian data.

The execution subject can input the labeling target image into the first detection model to obtain the position information of the target target. The location information may include information such as the center of the target object, the frame, and the like.

In step 203, determine the size of the labeling frame for labeling the target object based on the location information and the predetermined labeling frame size set.

After determining the location information of the target object, the performing entity can combine a predetermined set of labeling frame sizes to determine the size of the labeling frame adapted to the target object. The labeling frame size set includes a plurality of labeling frame sizes, wherein the plurality of labeling frame sizes is the size of the labeling frame of the target object in the conventional labeling image. In this way, the size of the labeling frame obtained by adaptation is relatively close to the size of the labeling frame set by the user, so there is no need to manually adjust it again.

In some alternative implementations of this embodiment, the above labeling frame size set can be implemented by the following steps not shown in FIG. Obtaining labeling frame information including the size of the labeling frames, clustering the sizes of the labeling frames, and obtaining a labeling frame size set.

In this mode of implementation, the execution subject can obtain the labeling frame information. The labeling frame information may be manually labeled. The execution subject can first obtain the published labeling data and determine each labeling frame information from the labeling data. The labeling frame information includes information such as the size and position of the labeling frame. An actor can cluster the size of each labeling frame to obtain multiple labeling frame sizes. Specifically, the actor can cluster the size of each labeling frame using a conventional clustering algorithm, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise). mentioned.

At step 204, the labeling frame is used to label the target object according to the determined size.

After determining the size of the adaptive target object, the agent can label the target object using the labeling frame. Note that the size of the labeling frame is the size of the labeling frame obtained by adaptation.

Continuing to refer to FIG. 3, FIG. 3 is a schematic diagram of one application scene of the method for labeling an object according to this embodiment. In the application scene of FIG. 3, a driving recorder attached to a vehicle 301 has collected multiple images of the driving environment. and uploading the image to a server 302, where the server 302 is provided with a pre-trained first detection model. Server 302 inputs a plurality of images into the first detection model and determines location information of pedestrians contained in each image. Then, the size of the labeling frame that accommodates the pedestrian is determined. Finally, the pedestrian is labeled with the labeling frame. The server 302 may further output the labeled image to the terminal device 303 . The terminal device 303 can be a terminal used by the labeler, and the labeler can further check the image after labeling to confirm whether the pedestrian's labeling frame needs to be adjusted. can be done.

A method for labeling an object according to the above embodiments of the present application may first acquire a labeling object image that includes a target object. Location information of the target object can then be determined based on the pre-trained first detection model and the image to be labeled. Then, the size of the labeling frame for labeling the target object can be determined based on the location information and the predetermined labeling frame size set. Finally, the target object can be labeled using the labeling frame based on the determined size. The method of the present embodiment can select an adaptive labeling frame from a predetermined labeling frame size set and label the target object using the labeling frame, thereby eliminating the need to manually adjust the labeling frame. , the burden of manual work is reduced.

With continued reference to FIG. 4, FIG. 4 illustrates a process 400 of another embodiment of a method for labeling objects according to the present application. As shown in FIG. 4, the method for labeling objects of this embodiment may include the following steps.

At step 401, an image to be labeled is acquired.

At step 402, the location information of the target object is determined based on the pre-trained first detection model and the labeling object image.

In step 403, determine the size of the labeling frame adapted to the target object based on the location information and the predetermined labeling frame size set.

At step 404, the target object is labeled with a labeling frame based on the determined size.

At step 405, the non-labeled region of the image to be labeled is obtained.

In this embodiment, the execution subject can also obtain the non-labeling region of the image to be labeled. The non-labeling region may contain a blurry obscure target object or may contain no target object. Correspondingly, the labeling region contains a plurality of target objects to be labeled, except that the region outside the non-labeling region in the labeling target image may be called the labeling region, and the performing subject is the labeling target image from the labeler. labeling area can be obtained. After the labeling person has obtained the labeled image, a labeling region can be selected from the image. And the image after selecting the labeling area is sent to the execution subject. A portion other than the labeling region is a non-labeling region.

In some alternative implementations of this embodiment, the performing entity can determine the non-labeling regions in the following steps not shown in FIG. A labeling region is determined based on the labeling frame, and a non-labeling region is determined based on the labeling region.

In this implementation, the actor can determine the enclosing frame that can enclose each labeling frame. Also, the area in the enclosing frame can be the labeling area, and the area outside the enclosing frame can be the non-labeling area.

At step 406, a mosaic is generated in the non-labeled regions.

In this embodiment, after determining the non-labeling regions, the performing entity can generate a mosaic in the non-labeling regions. Specifically, the agent can shuffle the positions of the pixels in the non-labeling regions by blocks, such that the non-labeling regions are not particularly noticeable compared to the surrounding pixels. This way, the image will be cleaner and the training effect will be better.

At step 407, training samples are determined based on the images after the mosaic has been generated.

The actor can use the mosaicked images as training samples. After the number of training samples reaches a predetermined number, multiple training samples can be used to train the model.

At step 408, taking the images with mosaics in the training samples as input and the labeling frames as the expected output, a second detection model is obtained.

In this embodiment, the actor can take the images that contain the mosaic but not the labeling frames in the training samples as the input of the model and the labeling frames as the expected output of the model to obtain a second detection model. Preferably, the number of convolutional layers included in the second detection model is small, and the computational complexity is small, so that the second detection model after training is placed at the moving edge with poor hardware conditions. It may also be provided, for example, in smart rearview mirrors to achieve rapid detection of pedestrians.

The method for labeling an object according to the above embodiments of the present application can obtain labeling images with good quality and train to generate the second detection model, which is advantageous for implementing deep learning algorithms.

Further referring to FIG. 5, as an implementation for the method shown in the above figures, the present application provides one embodiment of an apparatus for labeling an object, which apparatus embodiment corresponds to the method embodiment shown in FIG. However, the device can be specifically applied to various electronic devices.

As shown in FIG. 5, the apparatus 500 for labeling an object of this embodiment includes an image acquisition unit 501 , a positioning unit 502 , a size determination unit 503 and an object labeling unit 504 .

The image acquisition unit 501 is configured to acquire a labeling object image containing the target object.

The position determination unit 502 is configured to determine position information of the target object based on the pre-trained first detection model and the labeling object image.

The size determining unit 503 is configured to determine the size of the labeling frame adapted to the target object based on the position information and the predefined labeling frame size set.

The object labeling unit 504 is configured to label the target object with a labeling frame based on the determined size.

In some alternative implementations of this embodiment, the device 500 is configured to obtain non-labeling regions of the image to be labeled and generate a mosaic in the non-labeling regions, not shown in FIG. A processor may also be included.

In some alternative implementations of this embodiment, the device 500 is configured to obtain non-labeling regions of the image to be labeled and generate a mosaic in the non-labeling regions, not shown in FIG. A training unit may also be included.

In some alternative implementations of this embodiment, the apparatus 500 obtains labeling frame information including the size of the labeling frame, performs clustering on the size of the labeling frame, and obtains a set of labeling frame sizes. may further include a size set determination unit, not shown in FIG.

In some alternative implementations of this embodiment, the size set determination unit is further configured to determine the labeling region based on the labeling frame and the non-labeling region based on the labeling region.

It should be understood that the units 501 to 504 described in the apparatus 500 for labeling objects correspond to the steps in the method illustrated in FIG. Accordingly, the operations and features described for the method for labeling objects above are similarly applicable to apparatus 500 and its included units, and will not be described again here.

Referring now to FIG. 6, FIG. 6 shows a schematic configuration diagram applied to an electronic device (eg, server or terminal device in FIG. 1) 600 for realizing an embodiment of the present disclosure. The electronic device shown in FIG. 6 is merely an example and does not pose a limitation on the functionality and scope of use of the embodiments of the present disclosure.

As shown in FIG. 6, electronic device 600 performs various suitable operations and processes by means of programs stored in read only memory (ROM) 602 or programs loaded into random access memory (RAM) 603 from storage device 608 . An executing processing unit (eg, central processor, graphics processor, etc.) 601 may be included. Various programs and data necessary for operating the electronic device 600 are also stored in the RAM 603 . Processing unit 601 , ROM 602 and RAM 603 are connected together by bus 604 . Input/output (I/O) interface 605 is also connected to bus 604 .

Typically, input devices 606 including, for example, touch screens, touch pads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc., and output devices 607, including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc., such as tapes, hard disks. Storage devices 608 , including, for example, and communication devices 609 may be connected to I/O interfaces 605 . Communication device 609 may allow electronic device 600 to exchange data with other devices through wireless or wired communication. Although FIG. 6 shows electronic device 600 with various devices, it should be understood that it does not require that all of the devices shown be implemented or include it. Alternately implement or include more or fewer devices. Each box shown in FIG. 6 may represent one device, or multiple devices as desired.

In particular, according to embodiments of the present disclosure, the processes described in the flowcharts above can be implemented in computer software programs. For example, an embodiment of the present disclosure includes a computer program product comprising a computer program borne on a computer readable medium, the computer program including program code for performing the methods illustrated in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network by communication device 609 , installed from storage device 608 , or installed from ROM 602 . The computer program, when executed by the processing unit 601, performs the functions described above limited to the methods of the embodiments of the present disclosure. By way of clarification, the computer-readable medium described in the embodiments of this disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the foregoing. More specific examples of computer readable storage media are electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable May include, but is not limited to, read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage components, magnetic storage devices, or any suitable combination of the foregoing not. In embodiments of the present disclosure, a computer-readable storage medium may be a tangible medium containing or storing any program that is used in or in combination with an instruction execution system, apparatus or device. Also good. In an embodiment of the present disclosure, the computer readable signal medium may be a data signal carrying computer readable program code embodied in baseband or transmitted as part of a carrier wave. Such transmitted data signals may take many forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that stores a program for use with or in combination with an instruction execution system, apparatus or device. Can be sent, transmitted or transported. Program code contained on a computer readable medium may be transported on any suitable medium including, but not limited to, wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

The computer-readable medium may be included in the electronic device, or may exist separately and not be implemented in the electronic device. One or more programs are carried on the computer-readable medium, and when the one or more programs are executed on the electronic device, cause the electronic device to acquire a labeling object image including a target object; determining the location information of the target object based on the trained first detection model and the labeling object image; determining the size of the labeling frame adapted to the target object according to the location information and a predetermined labeling frame size set; Label the target object using the labeling frame according to the determined size.

Computer program code for carrying out the operations of the embodiments of the present disclosure may be written in one or more program design languages, or combinations thereof, wherein the program design language is a target-facing program design language, such as Java, Smalltalk, C++. and further includes conventional procedural programming languages, such as the "C" language or similar program design languages. Program code may run entirely on a user computer, partially on a user computer, as a separate software package, partially on a user computer, partially on a remote computer, or entirely remote. It may run on a computer or server. In the context of remote computers, the remote computer can be connected to the user computer by any type of network, including a local area network (LAN) or wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider). connected by the Internet).

The flowcharts and block diagrams in the figures illustrate possible system architectures, functionality, and operation of various embodiments of systems, methods and computer program products of this disclosure. In this regard, each box in a flowchart or block diagram can represent a module, program segment, or portion of code, wherein the module, program segment, or portion of code is one or more predetermined Contains executable instructions to implement logic functions. It should be noted that in some switchable implementations it may occur that the functions labeled in the boxes are also different from the order labeled in the figures. For example, two boxes shown as connecting can actually be executed essentially in parallel, and they may be executed in reverse order depending on the functionality involved. It should also be noted that each box in the block diagrams and/or flowcharts, and combinations of boxes in the block diagrams and/or flowcharts, are implemented in dedicated hardware-based systems that perform the specified functions or operations. or be implemented in a combination of dedicated hardware and computer instructions.

Units relating to the embodiments of the present disclosure may be implemented in a software manner or may be implemented in a hardware manner. The units described may be located in a processor, for example the processor may be described as comprising an image acquisition unit, a position determination unit, a size determination unit and an object labeling unit. The names of these units do not in some circumstances constitute a limitation to the units themselves, for example the image acquisition unit may be further described as "acquiring a unit of an image to be labeled".

The above description is merely that of the preferred embodiment of the present disclosure and the technical principles used. Those skilled in the art will understand that the scope of the invention regarding the embodiments of the present disclosure is not limited by technical means configured by a specific combination of the above technical features, and that the above technical features or It should be understood that other technical measures formed by performing any combination of equivalent features should also be included. For example, the above features are technical means formed by replacing each other with technical features having similar functions disclosed in the embodiments of the present disclosure (but not limited to them).

Claims (9)

A method for labeling an object, comprising:
obtaining a labeling target image including a target target;
determining location information of a target object based on a pre-trained first detection model and the labeling object image;
determining a labeling frame size adapted to the target object based on the location information and a predetermined labeling frame size set;
labeling the target object with a labeling frame based on the determined size;
obtaining a non-labeled region of the image to be labeled;
generating a mosaic in the non-labeling regions;
determining a training sample based on the images after the mosaic is generated, taking the images in the training samples as inputs and the labeling frames as expected outputs to obtain a second detection model;
A method for labeling a subject containing The method for labeling the object comprises :
obtaining labeling frame information including the size of the labeling frame;
clustering the labeling frame sizes to obtain a labeling frame size set;
2. The method of claim 1, further comprising: The step of obtaining a non-labeling region of the labeling target image includes:
determining a labeling region based on the labeling frame;
and determining the non-labeled regions based on the labeled regions. An apparatus for labeling an object, comprising:
an image acquisition unit configured to acquire a labeling object image including the target object;
a position determination unit configured to determine position information of a target object based on a pre-trained first detection model and the labeling target image;
a sizing unit configured to determine a labeling frame size adapted to the target object based on the location information and a predetermined labeling frame size set;
an object labeling unit configured to label the target object with a labeling frame based on the determined size;
a processing device configured to obtain non-labeled regions of the image to be labeled and generate a mosaic in the non-labeled regions;
a training unit configured to determine a training sample based on the images after the mosaic has been generated, taking the images in the training samples as inputs and the labeling frames as expected outputs to obtain a second detection model;
Apparatus for labeling an object containing The device for labeling the object comprises :
Get the labeling frame information, including the size of the labeling frame,
5. The apparatus of claim 4 , further comprising a size set determining unit configured to cluster sizes of the labeling frames to obtain a labeling frame size set. The size set determination unit further determines a labeling region based on the labeling frame;
5. Apparatus according to claim 4 , configured to determine said non-labeling region based on said labeling region. an electronic device,
one or more processors;
a storage device in which one or more programs are stored;
An electronic device that causes the one or more processors to implement the method according to any one of claims 1 to 3 when the one or more programs are executed by the one or more processors. A computer readable medium storing a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 3 . A computer program for implementing the method of any one of claims 1 to 3 when said program is executed by a processor.

Images