WO2022134390A1

WO2022134390A1 - Labeling method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022134390A1
Application number: PCT/CN2021/087285
Authority: WO
Inventors: 罗泽丰; 钟华平; 何聪辉
Original assignee: 深圳市商汤科技有限公司
Priority date: 2020-12-22
Filing date: 2021-04-14
Publication date: 2022-06-30
Also published as: CN112508020A; KR20220093091A; JP2023510443A

Abstract

Provided are a labeling method and apparatus, electronic device, and storage medium, said method comprising: receiving a first operation to label a target image (S11), said first label operation being used for drawing a first drawn graphic indicating a first image region in said target image; if a second drawn graphic in said target image overlaps at least partially with said first drawn graphic, then adjusting the first drawn graphic to obtain a labeling result corresponding to the first label operation (S12).

Description

Labeling method and device, electronic device and storage medium

technical field

The present disclosure relates to the field of computer technologies, and in particular, to a labeling method and apparatus, an electronic device, and a storage medium.

Background technique

In the field of artificial intelligence, neural networks can be trained to automatically identify and understand the information contained in images. In the process of training neural networks, labeled images are often used for training.

In the process of annotating an image, the annotated area is often determined by drawing graphics on the image.

SUMMARY OF THE INVENTION

The present disclosure proposes an annotation scheme.

According to an aspect of the present disclosure, a labeling method is provided, comprising:

receiving a first annotation operation on the target image, wherein the first annotation operation is used to draw a first drawing graphic indicating a first image area in the target image;

In the case that the second drawing graphic in the target image and the first drawing graphic at least partially overlap, the first drawing graphic is adjusted to obtain a labeling result corresponding to the first labeling operation.

In a possible implementation manner, before adjusting the first drawing graph, the method further includes:

Detecting whether there is a drawing figure that overlaps with the first drawing figure in at least one previously drawn figure in the target image.

A first user instruction is received, where the first user instruction is used to instruct to adjust the first drawing graphic.

In a possible implementation manner, the adjusting the first drawing graphic to obtain the labeling result corresponding to the first labeling operation includes:

A drawing graph obtained by removing the portion of the first drawing graph overlapping the second drawing graph from the first drawing graph is taken as a labeling result corresponding to the first labeling operation.

In the case where the first drawing graph and the second drawing graph are border graphs, determine, according to the positions of the first drawing graph and the second drawing graph, that the second drawing graph is located in the first drawing graph. a first line segment element in the drawing figure and a second line segment element in the first drawing figure outside the second drawing figure;

The drawing figure formed by the first line segment element and the second line segment element is used as the labeling result corresponding to the first labeling operation.

In a possible implementation manner, after the receiving the first labeling operation on the target image, the method further includes:

Displaying the target image and a plurality of drawing graphics including the first drawing graphics of the target image on the display interface;

receiving a second user instruction, where the second user instruction is used to select the first drawing graphic from the plurality of drawing graphics;

The selected first drawing graphic is determined according to the position indicated in the display interface by the second user instruction and the positions of the plurality of drawing graphics in the display interface.

In a possible implementation manner, the target image is displayed on a web interface.

In a possible implementation manner, the first labeling operation is used to perform semantic segmentation labeling on the target image.

According to an aspect of the present disclosure, there is provided a labeling device, comprising:

a receiving module, configured to receive a first annotation operation on the target image, wherein the first annotation operation is used to draw a first drawing graphic indicating a first image area in the target image;

An adjustment module, configured to adjust the first drawing graph in the case where the second drawing graph in the target image and the first drawing graph at least partially overlap to obtain the corresponding value of the first labeling operation Label the results.

In a possible implementation, the apparatus further includes:

A detection module, configured to detect whether there is a drawing figure that overlaps with the first drawing figure in at least one previously drawn figure in the target image.

In a possible implementation, the apparatus further includes:

A first user instruction receiving module, configured to receive a first user instruction, where the first user instruction is used to instruct to adjust the first drawing graphic.

In a possible implementation manner, the adjustment module is configured to remove the drawn graphic obtained by removing the portion of the first drawn graphic that overlaps with the second drawn graphic from the first drawn graphic, as The labeling result corresponding to the first labeling operation.

In a possible implementation manner, the adjustment module is configured to, in the case that the first drawing graph and the second drawing graph are border graphs, according to the first drawing graph and the second drawing graph the position of the graphic, determining the first line segment element in the second drawing graphic that is located in the first drawing graphic and the second line segment element in the first drawing graphic that is located outside the second drawing graphic; The drawing figure formed by the first line segment element and the second line segment element is used as the labeling result corresponding to the first labeling operation.

In a possible implementation manner, the apparatus further includes:

a display module, configured to display the target image and a plurality of drawing graphics including the first drawing graphics of the target image on a display interface;

a second user instruction receiving module, configured to receive a second user instruction, where the second user instruction is used to select the first drawing graphic from the plurality of drawing graphics;

A determining module, configured to determine the selected first drawing graphics according to the position indicated in the display interface by the second user instruction and the positions of the plurality of drawing graphics in the display interface.

According to an aspect of the present disclosure, there is provided an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the above method when executed by a processor.

According to an aspect of the present disclosure, the present disclosure provides a computer program comprising computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

In the embodiment of the present disclosure, in the process of labeling the target image, for the first drawing graphics drawn by the user's first labeling operation on the target image, the second drawing graphics and the first drawing graphics in the target image can exist. In the case of at least partial overlap, the first drawing graphic is automatically adjusted to obtain the labeling result corresponding to the first labeling operation. Compared with manually adjusting the first drawing graph manually, the efficiency of labeling the target image is improved, and the labeling time is saved. Especially in the case where the outer contour of the second drawing graph is relatively complex, the user does not need to perform additional manual adjustment for the same contour portion, which reduces the resource consumption caused by the user repeating the partial labeling process.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the present disclosure.

FIG. 1 shows a flowchart of a labeling method according to an embodiment of the present disclosure.

FIG. 2 shows a schematic diagram of drawing a graph according to an embodiment of the present disclosure.

FIG. 3 shows a schematic diagram of an annotation result according to an embodiment of the present disclosure.

FIG. 4 shows a schematic diagram of drawing a graph according to an embodiment of the present disclosure.

FIG. 5 shows a schematic diagram of an annotation result according to an embodiment of the present disclosure.

FIG. 6 shows a flowchart of an annotation method according to an embodiment of the present disclosure.

FIG. 7 shows a block diagram of an annotation apparatus according to an embodiment of the present disclosure.

FIG. 8 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

FIG. 9 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements with the same or similar function. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are set forth in the following detailed description. It will be understood by those skilled in the art that the present disclosure may be practiced without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.

FIG. 1 shows a flowchart of a labeling method according to an embodiment of the present disclosure. As shown in FIG. 1 , the labeling method includes:

In step S11, a first labeling operation on the target image is received, wherein the first labeling operation is used to draw a first drawing graphic indicating a first image area in the target image.

The target image may be an image to be marked, and the image may be an image in any format, which is not limited in this embodiment of the present disclosure. Optionally, the target image can be a sample image used for machine learning, and the sample image can be used for supervised learning in machine learning after labeling.

A user such as an annotator can annotate the first image area in the target image through the first annotation operation. In some embodiments, the user can determine the first image area to be marked by drawing a graph. Optionally, the graph drawn by the user can be a polygon or a circle, and the drawn graph can be regular or irregular. A regular graph, which is not limited in this embodiment of the present disclosure. The user may add meanings at the semantic level to the image information in the determined first image area by adding annotation information to the first image area.

For example, the content in the target image is a person standing on the lawn. During the process of labeling the lawn, the user can draw a drawing that matches the shape of the image area where the lawn is located, and then add annotation information to the drawn drawing. "Lawn" to give the selected image area "Lawn" semantics.

In step S12, in the case that the second drawing graphic in the target image and the first drawing graphic at least partially overlap, adjust the first drawing graphic to obtain the corresponding image of the first labeling operation. Label the results.

The second drawing graph can also be a graph drawn by the user, or can also be a graph obtained in other ways, for example, can be a graph drawn after performing image recognition on the target image through a neural network.

In a possible implementation manner, the second drawing graph may be a drawing graph specified by a user, that is, when the first drawing graph and the second drawing graph specified by the user at least partially overlap, the first drawing graph is automatically Adjustment. The user can specify the second drawing graph by selecting the drawing graph in the target image, or the user can also specify the second drawing graph by selecting the information such as the logo of the drawing graph. manner, which is not limited in the embodiments of the present disclosure.

In a possible implementation manner, the second drawing graphics may also be one or more drawing graphics among the drawing graphics included in the target image. For example, the second drawing graphics may be the same as the first drawing graphics included in the target image. There are at least partially overlapping drawing graphics in the drawing graphics. In this way, when there is a drawing graphics which at least partially overlap with the first drawing graphics in the target image, the first drawing graphics is automatically adjusted without user specification, which improves the user experience. Ease of use.

The first drawing graphic and the second drawing graphic at least partially overlap, and it may be that the second drawing graphic and the image area indicated by the first drawing graphic exist at least partially the same area. For example, please refer to Figure 2, the screen content of the target image is a person standing in the lawn, the area selected by the polygon (the first drawing figure) below is the lawn area, and the area selected by the leftmost polygon (the second drawing figure) is For portraits, the legs of the portraits cover part of the lawn, and the portraits other than the legs do not cover the lawn. The first drawing includes the image area where the lawn is located, and also includes the image area where the legs that block part of the lawn are located. The second drawing includes the image area where the portrait is located. Obviously, the first drawing and the second drawing are in the image area. There is some overlap.

In the case that the first drawing graph and the second drawing graph partially overlap, the first drawing graph can be adjusted automatically, and the specific adjustment methods can be various, for example, the overlapping part can be removed from the first drawing graph , to obtain the first drawing figure with the overlapping part removed, or the overlapping part in the first drawing figure can be reduced. For example, please refer to FIG. 3 , after removing the overlapping part in the first drawing graph, the adjusted first drawing graph obtained is shown in FIG. 3 , the first drawing graph no longer includes the occlusion in the second drawing graph Lawn Legs.

After the adjustment of the first drawing graphic is completed, the image area marked by the adjusted first drawing graphic may be used as the image area marked by the first marking operation, and the marking result corresponding to the first marking operation is obtained. The annotated target image can be used for supervised learning in machine learning. In the embodiment of the present disclosure, by removing or reducing the overlapping part in the first drawing graph, the interference information in the first drawing graph is removed or reduced, which can Improve the accuracy of neural networks derived from machine learning.

In the embodiment of the present disclosure, in the process of labeling the target image, for the first drawing graphics drawn by the user's first labeling operation on the target image, the second drawing graphics and the first drawing graphics in the target image can exist. In the case of at least partial overlap, the first drawing graphic is automatically adjusted to obtain the labeling result corresponding to the first labeling operation. Compared with manually adjusting the first drawing graph manually, the efficiency of labeling the target image is improved, and the labeling time is saved.

There may also be a variety of video processing methods provided by the present disclosure. In a possible implementation manner, before adjusting the first drawing graphics, the method further includes: detecting that at least one of the target images is in Whether there is a drawing graphic that overlaps with the first drawing graphic in the first drawing graphic.

The previously drawn graphic may be a graphic drawn before the first drawn graphic, and the previously drawn graphic may be a graphic drawn by a user, or may be a graphic obtained by other methods, for example, a neural network may be used to perform a neural network on the target image. Graphics drawn after image recognition. Here, taking the previously drawn graphic as an example of a graphic drawn by the user, after the user has finished drawing the previously drawn graphic and then the first drawn graphic, it can be detected whether the previously drawn graphic overlaps with the first drawn graphic.

In some embodiments, the detection step in this implementation manner may be performed after it is determined that the drawing of the first drawing graph is completed, that is, the execution of the detecting step may be triggered after the drawing of the first drawing graph is completed, without the need for After receiving the user's instruction, no other operation of the user is required, so that the user has fewer operations and higher convenience.

In this implementation manner, there may be various specific detection processes. Optionally, at least one previously drawn graphic in the target image may be detected based on at least one previously drawn graphic and the position coordinates of the first drawn graphic in the target image. Whether there is a drawing graph that overlaps with the first drawing graph in the graph.

The graphics drawn in the embodiments of the present disclosure may be border graphics or filled graphics. Wherein, the border graphic may be a graphic including only the lines of the border, and the filled graphic may be a graphic including the border and the content filled in the border.

In the case where both the previously drawn graph and the first drawn graph are border graphs, it can be determined whether there is at least one of the previously drawn graphs in the frame of the first drawn graph according to the coordinates of the previously drawn graph and the coordinates of the first drawn graph. Part of the frame, when it is determined that at least part of the frame of the previously drawn figure exists in the frame of the first drawn figure, it is determined that there is a drawing figure that overlaps with the first drawn figure in the previously drawn figure. In addition, it is also possible to determine whether there is at least a part of the point set that constitutes the point set of the previously drawn figure in the frame of the first drawing figure, and at least part of the point set that constitutes the previously drawn figure exists in the frame of the first drawing figure. In the case of the set, it is determined that there is a drawing pattern that overlaps with the first drawing pattern in the preceding drawing pattern.

In the case where both the previously drawn graph and the first drawn graph are filled graphs, it can be determined whether there are pixels with the same coordinates in the previously drawn graph and the first drawn graph. When there are pixels with the same coordinates, it is determined that there is a drawing pattern that overlaps with the first drawing pattern in the previously drawn pattern.

In a possible implementation manner, before adjusting the first drawing graph, the method further includes: receiving a first user instruction, where the first user instruction is used to instruct the first drawing graph to be adjusted Adjustment.

The user instructs the terminal device or the server to adjust the first drawing graphic by issuing the first user instruction. Wherein, optionally, the user may issue the first user instruction by operating a specific control in an application program of the terminal device. For example, after drawing the first drawing graphic, the user can perform a specified operation on a specific control of the application operation interface to issue the first user instruction, the control can be displayed on the interface in the form of a touch button, and the specified operation can be It is a click operation or other operation. After the specified operation is detected, the application will be triggered to automatically adjust the first drawn graphic. In some embodiments, the user may also send the first user instruction through a network connection (eg, connecting to a server through a specific URL link), or send the first user instruction through other means, which is not limited in this embodiment of the present disclosure.

In some embodiments, after receiving the first user instruction, it may be detected whether there is a drawing figure that overlaps with the first drawing figure in at least one previously drawn figure in the target image, and when it is detected that there is a drawing figure that overlaps with the first drawing figure in the target image In the case that there is a partially overlapped drawing pattern in the first drawing pattern, the first drawing pattern is automatically adjusted.

Alternatively, in some embodiments, it is also possible to first detect whether there is a drawing pattern that overlaps with the first drawing pattern in at least one previously drawn pattern in the target image. In the case of partially overlapping drawing graphics, a prompt is sent to the user in the user interface of the terminal device, prompting the user that there is a second drawing graphics overlapping with the first drawing graphics, and the user can choose whether to automatically adjust the first drawing graphics. , the user can issue a first user instruction to instruct to automatically adjust the first drawing graphic.

In the case where it is detected that there is a drawing graph partially overlapping with the first drawing graph in the target image, the first drawing graph and the second drawing graph having the overlapping portion can be displayed in a display style different from other drawing graphs, for the convenience of users. For example, the first drawing graph and the second drawing graph with overlapping parts may be displayed with a red border, and the other drawn graphs may be displayed with a gray border.

In a possible implementation manner, the adjusting the first drawing graph to obtain the labeling result corresponding to the first labeling operation includes: comparing the first drawing graph with the second drawing graph The overlapping part is a drawing graph obtained by removing the first drawing graph as a labeling result corresponding to the first labeling operation.

In this embodiment of the present disclosure, the semantics of the image area indicated by the overlapping portion of the first drawing graph and the second drawing graph may be the same as the semantics of the image area indicated by the second drawing graph, but overlapped with the first image area. The semantics of image regions other than parts are not the same. In this case, the drawing graph obtained by removing the portion of the first drawing graph overlapping with the second drawing graph from the first drawing graph may be used as the labeling result corresponding to the first labeling operation. The annotation result obtained in this way removes the image area in the first image area with the same semantics as the second image area, and the obtained first image area indicated by the first drawing graph has more accurate semantics, which can improve the neural network obtained by machine learning. Accuracy. At the same time, the user does not need to manually adjust the first image area, the user operates less, and the convenience is high, the efficiency of labeling the target image is improved, and the labeling time is saved.

In a possible implementation manner, the adjusting the first drawing graph to obtain the labeling result corresponding to the first labeling operation includes: the first drawing graph and the second drawing graph being: In the case of a border graphic, according to the positions of the first drawn graphic and the second drawn graphic, determine the first line segment element located in the first drawn graphic and the first Drawing a second line segment element outside the second drawing figure in the drawing; using the drawing figure formed by the first line segment element and the second line segment element as the labeling result corresponding to the first labeling operation.

The position of drawing the figure may be represented by the coordinates of the line segment elements that constitute the drawing figure, or it may also be represented by the coordinates of the point that constitutes the drawing figure.

In some embodiments, the drawing graph can be a polygon, and the polygon can be regarded as a graph composed of line segment elements, and the position of the line segment element can be represented by the coordinates of the line segment endpoints. For example, the line segment element can be represented as (x ₁ , y ₁ , x ₂ , y ₂ ), (x ₁ , y ₁ ) and (x ₂ , y ₂ ) represent the coordinates of the two vertices of the line element, respectively. Then the coordinates of the multiple line segment elements constituting the drawing graph can represent the position of drawing the graph.

In some embodiments, the frame of the drawing graph is composed of a series of point sets. For example, the drawing graph can be regarded as a set composed of pixel points, and the coordinates of the pixel points constituting the drawing graph can represent the position of drawing the graph. .

In the process of drawing graphics, the user can detect the coordinates of the drawing tool used for drawing graphics in the target image in the process of drawing, and then obtain the coordinates of pixel points and/or the coordinates of endpoints of line segment elements constituting the drawn graphics. The drawing tools here can be tools such as brushes and paint brushes, whose coordinates in the target image can be changed by moving the mouse, or can also be changed by touching the display screen/touchpad or the like.

According to the coordinates of the pixel points of the second drawing graph and the first drawing graph, the first line segment element in the second drawing graph and the first line segment element located outside the second drawing graph in the first drawing graph can be determined. Two line segment elements. Alternatively, it is also possible to determine, according to the second drawing graph and the coordinates of the end points of the line segment elements of the first drawing graph, the first line segment element located in the first drawing graph in the second drawing graph and the first line segment element in the first drawing graph and the 2. Draw the second line segment element outside the graphics.

Specifically, the coordinate range of the first pixel that constitutes the first drawing graph can be recorded as (x _min ～x _max , y _min ～y _max ), if the x value of the coordinates of a certain pixel of the second drawing graph and the If the y value falls within the coordinate range, it can be determined that the pixel point is located in the first drawing figure, otherwise, it can be determined that the pixel point is outside the first drawing figure. In this way, the first line segment element in the second drawing graph which is located in the first drawing graph and the second line segment element in the first drawing graph which is outside the second drawing graph can be determined.

Or, if both the x-value and y-value of the coordinates of the end point of the line segment element of the second drawing graph fall within the above coordinate range, it can be determined that the end point is located in the first drawing graph; otherwise, it can be determined that the end point is located in the first drawing graph. In addition to the graphics, in addition, the intersection of the line segment elements of the first graphics and the second graphics can also be determined. The endpoints of the line segment elements of the second drawing graph can determine the first line segment element in the second drawing graph that is located in the first drawing graph and the second line segment that is outside the second drawing graph in the first drawing graph. element.

After the first line segment element and the second line segment element are determined, the drawing figure formed by the first line segment element and the second line segment element can be used as the labeling result corresponding to the first labeling operation.

For example, in the first drawing graph P1 and the second drawing graph P2 shown in FIG. 4 , the set of line segment elements of P1 can be expressed as {a ₁ a ₂ , a ₂ b ₃ , b ₃ a ₄ , a ₄ a ₃ , a ₃ b ₂ , b ₂ a ₁ }, the set of line segment elements of P2 can be expressed as {b ₁ b ₂ , b ₂ b ₃ , b ₃ b ₄ , b ₄ b ₅ , b ₅ b ₆ , b ₆ b ₇ ,b ₇ b ₁ }, where a ₁ -a ₄ and b ₁ -b ₇ are the endpoints of the line segment. Among them, the line segment elements {b ₃ a ₄ , a ₄ a ₃ , a ₃ b ₂ } of P1 are located in P2, {b ₁ b ₂ , b ₃ b ₄ , b ₄ b ₅ , b ₅ b ₆ , b ₆ b ₇ ,b ₇ b ₁ } is located outside P2. As shown in Figure 5, the segment elements {b ₃ a ₄ , a ₄ a ₃ , a ₃ b ₂ } and {b ₁ b ₂ , b ₃ b ₄ , b ₄ b ₅ , b ₅ b ₆ , b ₆ can be combined The drawing graph formed by b ₇ , b ₇ b ₁ } is used as the labeling result corresponding to the first labeling operation, and the set of line segment elements of the second drawing graph can remain unchanged.

In this embodiment of the present disclosure, in the case where the first drawing graph and the second drawing graph are border graphs, it is determined that the second drawing graph is located in the first drawing graph according to the positions of the first drawing graph and the second drawing graph. The first line segment element in the graph and the second line segment element in the first drawing graph outside the second drawing graph, thus, the drawing graph composed of the first line segment element and the second line segment element can be used as the first label The labeling result corresponding to the operation can quickly obtain the labeling result of the first labeling operation, which improves the efficiency of the labeling operation and provides a better user experience.

In a possible implementation manner, after the receiving the first annotation operation on the target image, the method further includes: displaying the target image and the first drawing graphics of the target image on a display interface multiple drawing graphics including the drawing; receiving a second user instruction, the second user instruction is used to select the first drawing graphic from the multiple drawing graphics; according to the second user instruction on the display interface The position indicated in the display interface and the positions of the plurality of drawing graphics in the display interface determine the selected first drawing graphics.

The user selects the first drawing figure from the plurality of drawing figures by issuing a second user instruction. Wherein, optionally, the user may select the first drawing graphic by performing a selection operation on the drawing graphic in the target image. Specifically, when the user triggers the second instruction, the user can determine the position indicated by the mouse or the touch point in the display interface when the second instruction is triggered, and the positions of multiple drawing graphics in the display interface can also be drawn after the graphics are drawn. Determine, when it is determined that the position indicated by the second user instruction is located inside the first drawing graphic, the selected first drawing graphic can be determined. Thus, the first drawing graphic selected by the second user instruction can be determined.

Alternatively, the user may also select the first drawing graphic by selecting information such as the identifier of the drawing graphic, which will not be described in detail in this disclosure.

In the embodiment of the present disclosure, the user implements the designation operation on the first drawing graph by issuing a second user instruction, and the user can independently specify the first drawing graph to be adjusted, which is convenient for the user to adjust the drawing graph that needs to be adjusted. It can improve the accuracy of the neural network obtained by machine learning, and can meet the needs of diverse users, and the user experience is better.

In the field of artificial intelligence, semantic segmentation is an important research content in the field of computer vision. Image semantic segmentation can be applied to many application scenarios, such as target recognition, target detection and other fields. Through semantic segmentation, an image can be divided into regions with different semantic information. For example, after semantic segmentation of an image, semantic labels can be added to objects in the image (such as sky, lawn, people, trees, small animals, etc.) .

In the process of annotating an image, graphics are drawn on the image to indicate the annotated area, and polygons are usually used to frame areas with the same semantics. In the image to be semantically segmented, multiple things often occlude each other. For example, the content of the target image is a person standing in the lawn, and the person's legs in the image block part of the lawn. The edge of the occluded part is the boundary of semantic segmentation, and the semantics on both sides of the boundary are different. In the process of labeling semantic units with different semantics in the image, if the regions on both sides of the boundary line are labelled separately, the boundary line can be labelled twice. Since the shape of the boundary line is often irregular, the Labeling twice will result in lower efficiency of image labeling.

In the application scenario of semantic segmentation, the present disclosure provides a possible implementation manner, in which the first labeling operation is used to perform semantic segmentation labeling on the target image. The semantics of the image area indicated by the overlapping portion of the first drawing graph and the second drawing graph may be the same as the semantics of the image area indicated by the second drawing graph, and the semantics of the image area other than the overlapping portion of the first image area. Are not the same. For example, referring to the above example, the first image area is the area where the lawn is located, but includes the legs that block the lawn, and the second image area is the area where the person in the image is located. In this case, removing the overlapped part from the first drawing graph can obtain more accurate semantics of the first image region indicated by the first drawing graph, which can improve the accuracy of the neural network obtained by machine learning. At the same time, after the user draws the second drawing graph, if the user has marked the boundary line between the two semantic units by drawing the second drawing graph, in the process of drawing the first drawing graph, the user does not need to draw the two semantic units again. The boundary line between the two semantic units does not require the user to manually adjust the first drawn graphic, the user operation is less, the convenience is high, the efficiency of labeling the target image is improved, and the labeling time is saved.

In a possible implementation manner, the target image is displayed on a web interface, and the web interface may be, for example, a web interface encoded in HTML5 language. Then, in the process of drawing graphics, the graphics can be drawn through the canvas (Canvas) in HTML5, so as to realize the annotation of the target image.

Taking the target image displayed in the web interface as an example, the first labeling operation is used to perform semantic segmentation and labeling on the target image as a specific application scenario of the present disclosure, and the labeling method provided by the present disclosure will be exemplified. For the content described in detail, reference may be made to the foregoing related descriptions, and similarly, the content in this section can also be used to exemplify the foregoing content.

Referring to FIG. 6, in a possible application scenario provided by the present disclosure, the labeling method provided by the present disclosure includes:

Step 201, displaying the target image through a web page.

Step 202 , drawing graphics on the target image through a canvas (Canvas) in HTML5, to obtain a plurality of drawing graphics.

Drawing graphics can be polygons, and the process of drawing graphics is as follows:

First, the creation of polygon points is realized through the mouse button release event (mouseup) in JavaScript language. Specifically, after the user clicks the left button (presses the left button and then releases it), the user responds operation, the canvas can create the starting point of a new graph, or add new polygon vertices to the newly created graph. After the user clicks the right button (presses the right button and then releases it), in response to the user operation that the right button is released, the canvas can The start and end points of the graph are connected to achieve the closure of the created polygon.

Through the e event attribute in the callback event of the mouse move event (mousemove) in the JavaScript language, the position of the mouse in the browser is captured to determine the position of the user's operation on the target image, so as to determine the drawn graphics drawn by the user in the target image. on the location.

The drawing graph includes a plurality of drawing graphs, and the drawing graph is a graph drawn by the user for the region in the target image to be marked.

Step 203: Receive a selection operation instruction, and determine the first drawing graphic according to the selection operation instruction.

The selection operation instruction is used to select a graphic to be adjusted and processed from a plurality of drawing graphics, and the selection operation instruction is the second user instruction described above, and the graphic to be adjusted is referred to as the first drawing graphic here.

According to the position indicated by the selection operation instruction on the webpage and the positions of multiple drawing graphics in the webpage, when it is determined that the position indicated by the selection operation instruction on the webpage is inside the first drawing graphic, the selection can be determined. The first drawn graph.

Step 204, receiving an adjustment operation instruction.

The adjustment operation instruction is used to instruct to perform adjustment processing on the first drawing graphic in the web page, and the division operation instruction is the first user instruction described above.

Step 205 , in response to the adjustment operation instruction, detect whether there is a second drawing graphic that overlaps with the first drawing graphic among the plurality of drawing graphics in the target image.

The first drawing graphics and the second drawing graphics have overlapping parts. For example, the content of the image is a person standing on the lawn, and the legs of the person in the image cover part of the lawn. The first drawing graphics drawn by the user can be the area where the lawn is located. , Since the edge of the lawn is usually relatively straight, rather than an irregular shape, in order to facilitate the user to quickly draw the first drawing, the first drawing includes the area of the person's legs in the lawn, and the second drawing can be the area where people are located.

Step 206: Remove the overlapping portion of the first drawing graph and the second drawing graph from the first drawing graph to obtain an adjustment result.

In the case where the first drawing figure is a border figure, according to the positions of the first drawing figure and the second drawing figure, it can be determined that the first line segment element in the first drawing figure and the first drawing figure in the second drawing figure are located in the first drawing figure. The line segment element located outside the second drawing graph; then the graph formed by the first line segment element and the line segment element outside the second drawing graph in the first drawing graph is used as the adjustment result of the first drawing graph, and the adjusted The first drawn image is used to mark the semantic information of the first image area in the target image.

In addition, according to the positions of the first drawing graph and the second drawing graph, determine the second line segment element located in the second drawing graph in the first drawing graph, and then remove the second line segment element, so that there is no other line segment inside the second drawing graph element.

Compared with the first drawing graph, the area in the second drawing graph is removed from the adjustment result of the first drawing graph, which is used to mark the semantic units in the target image. In the case where the boundary line between two semantic units is drawn, in the process of drawing another semantic unit, there is no need for the user to draw the boundary line between the two semantic units, which improves the efficiency of labeling and saves time for labeling.

In a possible implementation manner, the labeling method may be performed by an electronic device such as a terminal device or a server, and the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone , Personal Digital Assistant (Personal Digital Assistant, PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc., the method can be implemented by the processor calling the computer-readable instructions stored in the memory. Alternatively, the method may be performed by a server.

It can be understood that the above-mentioned method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides labeling devices, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any labeling method provided by the present disclosure. Repeat.

FIG. 7 shows a block diagram of an annotation apparatus according to an embodiment of the present disclosure. As shown in FIG. 7 , the apparatus includes:

A receiving module 301, configured to receive a first annotation operation on a target image, wherein the first annotation operation is used to draw a first drawing graphic indicating a first image area in the target image;

The adjustment module 302 is configured to adjust the first drawn graphic under the condition that the second drawn graphic in the target image and the first drawn graphic at least partially overlap, so as to obtain the corresponding first annotation operation labeling results.

In a possible implementation, the apparatus further includes:

In a possible implementation manner, the adjustment module 302 is configured to remove the drawn graphic obtained by removing the portion of the first drawn graphic that overlaps with the second drawn graphic from the first drawn graphic, as the labeling result corresponding to the first labeling operation.

In a possible implementation manner, the adjustment module 302 is configured to, in the case that the first drawing graph and the second drawing graph are border graphs, according to the first drawing graph and the second drawing graph The position of the drawing figure, determining the first line segment element located in the first drawing figure in the second drawing figure and the second line segment element outside the second drawing figure in the first drawing figure; The drawing figure formed by the first line segment element and the second line segment element is used as the labeling result corresponding to the first labeling operation.

In a possible implementation, the apparatus further includes:

In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments, and the specific implementation and technical effects may refer to the above method embodiments. It is concise and will not be repeated here.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

Embodiments of the present disclosure also provide a computer program product, including computer-readable codes. When the computer-readable codes are run on a device, a processor in the device executes instructions for implementing the labeling method provided in any of the above embodiments. .

An embodiment of the present disclosure further provides another computer program product for storing computer-readable instructions, which, when executed, cause the computer to perform the operations of the labeling method provided in any of the foregoing embodiments.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 8 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. terminal.

8, an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814 , and the communication component 816 .

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when electronic device 800 is in operating modes, such as calling mode, recording mode, and voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 . For example, the sensor assembly 814 can detect the on/off state of the electronic device 800, the relative positioning of the components, such as the display and the keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or one of the electronic device 800 Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as wireless network (WiFi), second generation mobile communication technology (2G) or third generation mobile communication technology (3G), or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A programmed gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation is used to perform the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium, such as a memory 804 comprising computer program instructions executable by the processor 820 of the electronic device 800 to perform the above method is also provided.

FIG. 9 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 9, electronic device 1900 includes processing component 1922, which further includes one or more processors, and a memory resource represented by memory 1932 for storing instructions executable by processing component 1922, such as applications. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) interface 1958 . The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server ^™ ), a graphical user interface based operating system (Mac OS X ^™ ) introduced by Apple, a multi-user multi-process computer operating system (Unix ^™ ), Free and Open Source Unix-like Operating System (Linux ^™ ), Open Source Unix-like Operating System (FreeBSD ^™ ) or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as memory 1932 comprising computer program instructions executable by processing component 1922 of electronic device 1900 to perform the above-described method.

The present disclosure may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

A labeling method, comprising:

receiving a first annotation operation on the target image, wherein the first annotation operation is used to draw a first drawing graphic indicating a first image area in the target image;

In the case that the second drawing graphic in the target image and the first drawing graphic at least partially overlap, the first drawing graphic is adjusted to obtain a labeling result corresponding to the first labeling operation.
The method according to claim 1, wherein before adjusting the first drawing graph, the method further comprises:

Detecting whether there is a drawing figure that overlaps with the first drawing figure in at least one previously drawn figure in the target image.
The method according to any one of claims 1 or 2, wherein before adjusting the first drawing graph, the method further comprises:

A first user instruction is received, where the first user instruction is used to instruct to adjust the first drawing graphic.
The method according to any one of claims 1-3, wherein the adjusting the first drawing graphic to obtain a labeling result corresponding to the first labeling operation comprises:

A drawing graph obtained by removing the portion of the first drawing graph overlapping the second drawing graph from the first drawing graph is taken as a labeling result corresponding to the first labeling operation.
The method according to any one of claims 1-4, wherein the adjusting the first drawing graph to obtain a labeling result corresponding to the first labeling operation comprises:

In the case where the first drawing graph and the second drawing graph are border graphs, determine, according to the positions of the first drawing graph and the second drawing graph, that the second drawing graph is located in the first drawing graph. a first line segment element in the drawing figure and a second line segment element in the first drawing figure outside the second drawing figure;

The drawing figure formed by the first line segment element and the second line segment element is used as the labeling result corresponding to the first labeling operation.
The method according to any one of claims 1-5, wherein after the receiving the first labeling operation on the target image, the method further comprises:

Displaying the target image and a plurality of drawing graphics of the target image including the first drawing graphics on a display interface;

receiving a second user instruction, where the second user instruction is used to select the first drawing graphic from the plurality of drawing graphics;

The selected first drawing graphic is determined according to the position indicated in the display interface by the second user instruction and the positions of the plurality of drawing graphics in the display interface.
The method according to any one of claims 1-6, wherein the target image is displayed on a web interface.
The method according to any one of claims 1-7, wherein the first labeling operation is used to perform semantic segmentation labeling on the target image.
A labeling device, characterized in that, comprising:

A receiving module, for receiving the first labeling operation to the target image, wherein the first labeling operation is used to draw the first drawing figure indicating the first image area in the target image;

An adjustment module, configured to adjust the first drawing graph in the case where the second drawing graph in the target image and the first drawing graph at least partially overlap to obtain the corresponding value of the first labeling operation Label the results.
An electronic device, comprising:

processor;

memory for storing processor-executable instructions;

wherein the processor is configured to invoke the memory-stored instructions to perform the method of any one of claims 1-8.
A computer-readable storage medium on which computer program instructions are stored, characterized in that, when the computer program instructions are executed by a processor, the method described in any one of claims 1 to 8 is implemented.
A computer program, comprising computer readable code, when the computer readable code is executed in an electronic device, a processor in the electronic device executes the code for implementing any one of claims 1-8. Methods.