JP2023510443A - Labeling method and device, electronic device and storage medium - Google Patents

Abstract

TECHNICAL FIELD The present disclosure relates to labeling methods and devices, electronic devices, and storage media. The method includes receiving a first labeling operation on a target image, the first labeling operation for drawing a first drawing figure indicating a first image region in the target image; adjusting the first drawing figure to obtain a labeling result corresponding to the first labeling operation when the drawing figure and the first drawing figure at least partially overlap.
[Selection drawing] Fig. 1

Description

TECHNICAL FIELD The present disclosure relates to the field of computer technology, and more particularly to labeling methods and apparatus, electronic devices, and storage media.

In the field of artificial intelligence, neural networks can be trained to automatically recognize and understand information contained in images. In the process of training neural networks, they are often trained using pre-labeled images.

In the process of labeling an image, it is often the case that a labeling region is determined by drawing a figure on the image.

The present disclosure provides technical means of labeling.

According to one aspect of the disclosure:
receiving a first labeling operation for a target image, the first labeling operation for drawing a first drawing figure indicating a first image region in the target image;
obtaining a labeling result corresponding to the first labeling operation by adjusting the first drawing figure when the second drawing figure and the first drawing figure in the target image at least partially overlap;
To provide a labeling method comprising:

In one possible implementation, the method further comprises, before adjusting the first drawing figure,
It includes detecting whether or not there is a drawing figure that overlaps with the first drawing figure among at least one previous drawing figure in the target image.

In one possible implementation, the method further comprises, before adjusting the first drawing figure,
Receiving a first user command for instructing adjustment of the first drawing figure.

In one possible implementation, adjusting the first drawing figure to obtain a labeling result corresponding to the first labeling operation includes:
A drawing figure obtained by removing a portion overlapping with the second drawing figure from the first drawing figure is used as a labeling result corresponding to the first labeling operation.

In one possible implementation, adjusting the first drawing figure to obtain a labeling result corresponding to the first labeling operation includes:
When the first drawing figure and the second drawing figure are frame figures, the position of the second drawing figure within the first drawing figure based on the positions of the first drawing figure and the second drawing figure Determining a first line segment element located outside the second drawing figure of the first drawing figure and a second line segment element located outside the second drawing figure;
setting a drawing figure composed of the first line segment element and the second line segment element as a labeling result corresponding to the first labeling operation.

In one possible implementation, the method, after accepting a first labeling operation for the target image, further comprises:
displaying on a display screen the target image and a plurality of drawing figures including the first drawing figure in the target image;
receiving a second user command for selecting the first drawing figure from the plurality of drawing figures;
determining the first drawing figure to be selected based on the position indicated by the second user command on the display screen and the positions of the plurality of drawing figures on the display screen.

In one possible implementation, the target image is displayed on a web page screen.

In one possible implementation, said first labeling operation is for performing semantic segmentation labeling on said target image.

According to one aspect of the disclosure:
a reception module that receives a first labeling operation for a target image, the first labeling operation for drawing a first drawing figure indicating a first image area in the target image;
an adjustment module that adjusts the first drawing figure to obtain a labeling result corresponding to the first labeling operation when the second drawing figure and the first drawing figure in the target image at least partially overlap. A labeling device comprising:

In one possible implementation, the apparatus further includes a detection module for detecting whether or not there is a drawn figure that overlaps with the first drawn figure among at least one previous drawn figure in the target image. Prepare.

In one possible implementation, the device further comprises a first user command receiving module for receiving a first user command to instruct to adjust the first drawing figure.

In one possible implementation mode, the adjustment module takes the drawing figure obtained by removing the portion overlapping the second drawing figure from the first drawing figure as the labeling result corresponding to the first labeling operation.

In one possible implementation mode, the adjustment module adjusts the first drawing figure and the second drawing figure based on the positions of the first drawing figure and the second drawing figure when the first drawing figure and the second drawing figure are frame figures. Determining a first line segment element positioned within the first drawing figure of 2 drawing figures and a second line segment element positioned outside the second drawing figure of the first drawing figure, and determining the first line segment element and the second line segment element is defined as a labeling result corresponding to the first labeling operation.

In one possible implementation, the device comprises:
a display module for displaying the target image and a plurality of drawing figures including the first drawing figure in the target image on a display screen;
a second user command receiving module for receiving a second user command for selecting the first drawing figure from the plurality of drawing figures;
a determination module that determines the first drawing figure to be selected based on the position indicated by the second user command on the display screen and the positions of the plurality of drawing figures on the display screen.

In one possible implementation, the target image is displayed on a web page screen.

In one possible implementation, said first labeling operation is for performing semantic segmentation labeling on said target image.

According to one aspect of the present disclosure, it includes a processor and a memory for storing instructions executable by the processor, wherein the processor invokes the instructions stored in the memory to perform the method. To provide an electronic device configured to:

According to one aspect of the present disclosure, a computer readable storage medium having computer program instructions stored thereon also provides a computer readable storage medium that, when executed by a processor, implements the above method. .

According to one aspect of the present disclosure, there is provided a computer program product comprising computer readable code for causing a processor in an electronic device to perform the method described above when the computer readable code is executed in the electronic device.

In the embodiment of the present disclosure, in the process of labeling the target image, for the first drawing figure drawn by the user's first labeling operation on the target image, the second drawing figure and the first drawing figure in the target image are at least partially If overlapping, the first drawing figure can be automatically adjusted to obtain the labeling result corresponding to the first labeling operation. Compared to manual adjustment of the first drawing figure, the efficiency of labeling the target image is improved, and the time required for labeling is saved. In particular, when the outer contour of the second drawing figure is relatively complicated, the user does not have to manually adjust the same contour portion separately, and resource consumption due to repeated partial labeling by the user is reduced.

It should be noted that the foregoing general description and the following detailed description are exemplary and interpretive only and are not limiting of the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of illustrative embodiments with reference to the drawings.

Here, the drawings incorporated as part of the present specification are suitable for the embodiments of the present disclosure and are used together with the description to describe the technical solutions of the present disclosure.
4 shows a flow chart of a labeling method according to an embodiment of the present disclosure; FIG. 2 shows a schematic diagram of a drawing figure according to an embodiment of the present disclosure; FIG. FIG. 4 shows a schematic diagram of a labeling result according to an example of the present disclosure; FIG. 2 shows a schematic diagram of a drawing figure according to an embodiment of the present disclosure; FIG. FIG. 4 shows a schematic diagram of a labeling result according to an example of the present disclosure; 4 shows a flow chart of a labeling method according to an embodiment of the present disclosure; 1 shows a block diagram of a labeling device according to an embodiment of the present disclosure; FIG. 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure; FIG. 1 shows a block diagram of an electronic device according to an embodiment of the present disclosure; FIG.

Various illustrative embodiments, features, and aspects of the disclosure are described in detail below with reference to the drawings. The same reference numerals in the drawings indicate elements of the same or similar function. Although the drawings show various aspects of the embodiments, the drawings need not be drawn to scale unless otherwise indicated.

As used herein, the term "exemplary" means "serving as an example, example, or illustration." Any embodiment described herein as "exemplary" is not to be construed as preferred or superior to other embodiments.

As used herein, the term "and/or" simply describes a related relationship with a related subject and indicates that there can be three relationships, e.g., A and/or B means that A only We may indicate three cases where there is, both A and B are present, and only B is present. Also, the term "at least one" herein refers to any one of a plurality or any combination of at least two of a plurality, e.g., at least one of A, B, C Containing may indicate containing any one or more elements selected from the set consisting of A, B and C selected.

Also, various specific details are set forth in the specific embodiments below in order to more effectively describe the present disclosure. It should be understood by one of ordinary skill in the art that the present disclosure may equally be practiced without some of the specific details. In some embodiments, detailed descriptions of methods, means, elements and circuits that are familiar to those of ordinary skill in the art are not provided in order to emphasize the spirit of the present disclosure.

FIG. 1 shows a flow chart of a labeling method according to an embodiment of the present disclosure. As shown in FIG. 1, the labeling method includes steps S11 and S12.

In step S11, a first labeling operation for drawing a first drawing figure indicating a first image area in the target image is received.

The target image may be the image to be labeled, and the image may be of any type, and the embodiments of this disclosure are not limited in this regard. The target images may be sample images for machine learning, and the sample images are available for supervised machine learning after labeling.

A user, such as a labeler, can label a first image region of the target image by a first labeling operation. In some embodiments, a user can draw a shape to determine the first image region to be labeled. The shapes drawn by the user may be polygons, they may be circles, and the shapes drawn may be regular or irregular, although the embodiments of the present disclosure are not limited to this. . The user can give semantic meaning to the image information in the determined first image region by adding labeling information to the first image region.

For example, if the content of the target image is a person standing on the lawn, in the process of labeling the lawn, the user draws a drawing figure that matches the shape of the image area where the lawn is located, and then draws the drawn figure. Labeling information "grass" can be added to the drawn graphics to give "grass" semantics to selected image regions.

In step S12, when the second drawing figure and the first drawing figure in the target image at least partially overlap, the first drawing figure is adjusted to obtain a labeling result corresponding to the first labeling operation.

The second drawing figure may be a figure drawn by the user, or may be a figure obtained by another method. For example, it may be a figure drawn by performing image recognition on the target image using a neural network.

In one possible embodiment, the second drawing figure may be a drawing figure specified by the user. That is, when the first drawing figure and the second drawing figure specified by the user at least partially overlap, the first drawing figure is automatically adjusted. The user may specify the second drawing figure by performing a selection operation on the drawing figure in the target image, or may specify the second drawing figure by selecting information such as the mark of the drawing figure. good. The embodiment of the present disclosure does not limit the specific implementation of designating the second drawing figure.

In one possible embodiment, the second drawing figure may be one or more of the drawing figures included in the target image. For example, the second drawing figure may be a drawing figure that at least partially overlaps with the first drawing figure among the drawing figures included in the target image. In this manner, when there is a drawing figure that overlaps at least a part of the first drawing figure in the target image, the first drawing figure is automatically adjusted. Convenience is improved.

When the first drawing figure and the second drawing figure at least partially overlap, at least a part of the image area indicated by the second drawing figure and the first drawing figure may be the same. . For example, referring to FIG. 2, the screen content of the target image is a person standing on the lawn, the area indicated by the lower polygon (first drawing figure) is the lawn area, and the leftmost The area indicated by the polygon (second drawing figure) is the human figure. . The first drawing figure includes the image area where the lawn is located, and also includes the image area where the legs shielding a part of the lawn are located, and the second drawing figure is where the human figure is located. Contains the image area. It is clear that at least a portion of the first drawing figure and the second drawing figure overlap.

When the first drawing figure and the second drawing figure at least partially overlap, the first drawing figure may be automatically adjusted. There may be a plurality of specific adjustment methods. For example, the overlapping part may be removed from the first drawing figure to obtain the first drawing figure from which the overlapping part has been removed, or the overlapping part may be reduced in the first drawing figure. For example, referring to FIG. 3, the adjusted first drawing figure obtained by removing the overlapped portion in the first drawing figure has the legs shielding the lawn in the second drawing figure as shown in FIG. No longer included.

After finishing the adjustment of the first drawing figure, the image area labeled with the adjusted first drawing figure is used as the image area labeled by the first labeling operation, and the labeling result corresponding to the first labeling operation is obtained. good. Target images after labeling can be used for supervised machine learning. In the embodiment of the present disclosure, the disturbance information in the first drawing figure is removed or reduced by removing or reducing the overlapping portion in the first drawing figure, so the accuracy of the neural network obtained by machine learning is improved. be able to.

In the embodiment of the present disclosure, in the process of labeling the target image, for the first drawing figure drawn by the user's first labeling operation on the target image, the second drawing figure and the first drawing figure in the target image are at least partially If overlapping, the first drawing figure can be automatically adjusted to obtain the labeling result corresponding to the first labeling operation. Compared to manual adjustment of the first drawing figure, the efficiency of labeling the target image is improved, and the time required for labeling is saved.

There may be multiple marking methods provided in this disclosure. In one possible embodiment, before adjusting the first drawing figure, the method further includes, among at least one previous drawing figure in the target image, a drawing figure having a portion that overlaps with the first drawing figure. includes detecting whether there is a

The previously drawn figure may be a figure drawn earlier than the first drawn figure. The figure to be drawn may be a figure drawn by the user, or may be a figure obtained by another method, for example, a figure drawn by performing image recognition on the target image using a neural network. . Here, taking as an example the case where the previously drawn figure is a figure drawn by the user, if the user draws the previously drawn figure and then draws the first drawn figure, the previously drawn figure overlaps the first drawn figure. It is possible to detect whether there is a part or not.

In some embodiments, the detection step in this implementation may be performed when it is confirmed that the drawing of the first drawing figure is finished. In other words, the detection step may be triggered without receiving a user command or performing any other operation by the user after the drawing of the first drawing figure is completed. In this way, the number of user operations is small, and the convenience is high.

In this implementation mode, there may be a plurality of specific detection processes. For example, based on the positional coordinates of at least one previous drawing figure and the first drawing figure in the target image, among at least one previous drawing figure in the target image, there is a drawing figure that overlaps with the first drawing figure. It may be detected whether or not

A graphic drawn in the embodiment of the present disclosure may be a frame graphic or a filling graphic. Here, the frame graphic may be a graphic including only the lines of the frame, and the filling graphic may be a graphic including the frame and the contents filled in the frame.

When both the preceding drawing figure and the first drawing figure are frame figures, at least part of the preceding drawing figure is placed within the frame of the first drawing figure based on the coordinates of the first drawing figure and the coordinates of the first drawing figure. It is determined whether or not a frame exists, and if it is determined that at least a part of the frame of the previous drawing figure exists within the frame of the first drawing figure, the part of the previous drawing figure that overlaps with the first drawing figure is determined. It may be determined that certain drawing figures exist. Also, it is determined whether or not at least a part of the dot set forming the preceding figure is within the frame of the first drawing figure, and the dot set constituting the preceding drawing figure is inside the frame of the first drawing figure. When it is determined that there is at least part of the drawing figure, it may be determined that there is a drawing figure that overlaps the first drawing figure among the previous drawing figures.

When both the preceding drawing figure and the first drawing figure are filling figures, it is determined whether or not there is a pixel point having the same coordinates in the first drawing figure and the first drawing figure. If it is determined that there is a pixel point having the same coordinates in the first drawing figure, it may be determined that there is a drawing figure having a portion that overlaps with the first drawing figure in the previous drawing figure.

In one possible implementation, the method, before adjusting the first drawing figure, further includes receiving a first user command to adjust the first drawing figure.

The user issues a first user command to instruct the terminal device or server to adjust the first drawing figure. For example, the user may issue the first user command by manipulating a specific control in the application program of the terminal device. For example, after the drawing of the first drawing figure is completed, the user may issue the first user command by performing a specified operation on a specific control on the application program operation screen. The control may be displayed on the screen in the form of a touch button, and the specifying operation may be a click operation or other operation. When the specifying operation is detected, the application program will be triggered to automatically adjust the first drawing figure. In some embodiments, a user may issue a first user command over a network connection (e.g., by being connected to a server via a particular URL link) or otherwise issue a first user command. well, and the disclosure is not limited in this regard.

In some embodiments, after receiving the first user command, detecting whether or not there is a drawing figure that overlaps with the first drawing figure among at least one previous drawing figure in the target image, and The first drawing figure may be automatically adjusted when it is detected that there is a drawing figure that overlaps the first drawing figure in the target image.

Alternatively, in some embodiments, it is first detected whether or not there is a drawing figure that overlaps the first drawing figure among at least one previous drawing figure in the target image, and the first drawing figure in the target image is detected first. When it is detected that there is a drawing figure that overlaps with the drawing figure, a presentation is issued to the user on the user interface of the terminal device to present to the user that there is a second drawing figure that overlaps with the first drawing figure. Then, the user may select whether or not to automatically adjust the first drawing figure. At this time, the user can issue a first user command to instruct the automatic adjustment of the first drawing figure.

When it is detected that a drawing figure having a portion overlapping with the first drawing figure exists in the target image, the first drawing figure and the second drawing figure having the overlapping part are separated from the other drawing figures for convenience of user's observation. It may be displayed in different display forms. For example, the first drawing figure and the second drawing figure having overlapping portions may be displayed with a red frame, and the other drawing figures may be displayed with a gray frame.

In one possible embodiment, adjusting the first drawing figure to obtain a labeling result corresponding to the first labeling operation may be performed by removing the portion of the first drawing figure that overlaps the second drawing figure. and setting the removed drawing figure as a labeling result corresponding to the first labeling operation.

In an embodiment of the present disclosure, the semantics of the image area indicated by the overlapping portion of the first drawing figure and the second drawing figure are the same as the semantics of the image area indicated by the second drawing figure, but the first image area may be different from the semantic of the image area other than the overlapping portion. In this case, the drawing figure obtained by removing the portion of the first drawing figure that overlaps the second drawing figure from the first drawing figure may be used as the labeling result corresponding to the first labeling operation. As a result of the labeling obtained in this way, since the image area having the same semantics as the second image area is removed in the first image area, the semantic of the first image area indicated by the obtained first drawing figure is more It becomes accurate and can improve the accuracy of the neural network obtained by machine learning. In addition, the user does not need to manually adjust the first drawing figure, the number of user operations is small, the convenience is high, the efficiency of labeling the target image is improved, and the time required for labeling is saved.

In one possible embodiment, adjusting the first drawing figure to obtain a labeling result corresponding to the first labeling operation is performed by adjusting the first drawing figure and the second drawing figure as frame figures. In this case, based on the positions of the first drawing figure and the second drawing figure, the first line segment element positioned within the first drawing figure of the second drawing figure and the second drawing figure of the first drawing figure Determining a second line segment element positioned outside the drawing figure, and setting a drawing figure composed of the first line segment element and the second line segment element as a labeling result corresponding to the first labeling operation. and including.

The position of the drawing figure may be represented by the coordinates of the line segment elements that make up the drawing figure, or by the coordinates of the points that make up the drawing figure.

In some examples, the drawing figure may be a polygon. A polygon may be regarded as a figure composed of line segment elements. The position of the line segment element may be represented by the coordinates of the end points of the line segment. For example, a line segment element may be represented by (x ₁ , y ₁ , x ₂ , y ₂ ), where (x ₁ , y ₁ ) and (x ₂ , y ₂ ) are the two Represents the coordinates of a vertex. Then, the coordinates of the plurality of line segment elements forming the drawing figure can represent the position of the drawing figure.

In some embodiments, the drawing figure frame consists of a series of dot sets. For example, a drawing figure may be regarded as a set of pixel points, and the coordinates of the pixel points forming the drawing figure can represent the position of the drawing figure.

In the process of drawing a figure by the user, if the coordinates in the target image of the drawing tool for drawing the figure are detected during drawing, the coordinates of the pixel points and/or the coordinates of the endpoints of the line segment elements that make up the drawing figure can be obtained. be done. The drawing tool here may be a tool such as a drawing pen, a drawing brush, or the like. The coordinates of the target image of the drawing tool may be changed by moving the mouse, or may be changed by touching the display screen/touch panel or the like.

Based on the coordinates of the pixel points of the second drawing figure and the first drawing figure, the first line segment element of the second drawing figure positioned within the first drawing figure and the first drawing figure positioned outside the second drawing figure A second segment element may be determined. Further, based on the coordinates of the end points of the line segment elements of the second drawing figure and the first drawing figure, the second drawing of the first line segment element and the first drawing figure located within the first drawing figure of the second drawing figure A second segment element located outside the figure may be determined.

Specifically, the coordinate range of the first pixel points forming the first drawing figure may be described as (x _min to x _max , y _min to y _max ). If the x-value and y-value of the coordinates of a pixel point on the second drawing figure are within this coordinate range, it can be determined that this pixel point is within the first drawing figure; It can be determined that it is outside the drawing figure. As a result, the first line segment elements positioned within the first drawing figure of the second drawing figure and the second line segment elements positioned outside the second drawing figure of the first drawing figure can be determined.

Alternatively, if both the x-value and y-value of the coordinates of the end point of the line segment element of the second drawing figure are within the above coordinate range, it can be determined that this end point is within the first drawing figure; It can be determined that this end point is outside the first drawing figure. Also, the intersection of the line segment elements of the first drawing figure and the second drawing figure is determined, and the intersection and the end points of the line segment elements of the second drawing figure located within the first drawing figure and outside the first drawing figure are determined. Based on the end point of the line segment element of the second drawing figure located at , the first line segment element located within the first drawing figure of the second drawing figure and the first drawing figure located outside the second drawing figure of the first drawing figure A 2-segment element can be determined.

After determining the first line segment element and the second line segment element, the drawing figure composed of the first line segment element and the second line segment element can be the labeling result corresponding to the first labeling operation.

For example _{, for a} first _drawing figure _P1 and a second drawing figure P2 _{as shown} in FIG _. a ₃ b ₂ ,b ₂ a ₁ }, and the set of segment elements of P2 is {b ₁ b ₂ ,b ₂ b ₃ ,b ₃ b ₄ ,b ₄ b ₅ ,b ₅ b ₆ ,b ₆ b ₇ , b ₇ b ₁ }, where a ₁ -a ₄ and b ₁ -b ₇ are both endpoints of line segments. where the segment elements {b ₃ a ₄ ,a ₄ a ₃ ,a ₃ b ₂ } of P1 are in P2 and {b ₁ b ₂ ,b ₃ b ₄ ,b ₄ b ₅ ,b ₅ b ₆ , b ₆ b ₇ , b ₇ b ₁ } are outside P2. As _shown in FIG _. 5, _{line segment elements {b3a4,a4a3,a3b2 } and { b1b2 , b3b4 , b4b5 , b5b6 , b6} b ₇ , b ₇ b ₁ } may be set as the labeling result corresponding to the first labeling operation, and the set of line segment elements of the second drawing figure may be retained as it is.

In the embodiment of the present disclosure, when the first drawing figure and the second drawing figure are frame figures, a A drawing figure composed of the first line segment element and the second line segment element is determined by determining the positioned first line segment element and the second line segment element positioned outside the second drawing figure of the first drawing figure. The labeling result corresponding to the first labeling operation can be obtained, the labeling result of the first labeling operation can be obtained quickly, the efficiency of the labeling operation is improved, and the user experience is good.

In one possible embodiment, the method further includes displaying the target image and a plurality of drawing figures including the first drawing figure in the target image on a display screen after receiving a first labeling operation for the target image. receiving a second user command for selecting the first drawing figure from the plurality of drawing figures; and selecting the position indicated by the second user command on the display screen and the plurality of drawing figures on the display screen. determining the first drawing figure to be selected based on the position of the drawing figure.

The user issues a second user command to select the first drawing figure from the plurality of drawing figures. For example, the user may select the first drawing figure by performing a selection operation on the drawing figure in the target image. Specifically, when the user triggers the second user command, the user can determine the position indicated by the mouse or touch point on the display screen at the time of triggering the second user command. The positions of the plurality of graphics to be drawn on the display screen may be determined after the graphics have been drawn. The first drawing figure to be selected can be determined when the position indicated by the second user command is inside the first drawing figure. Thereby, the first drawing figure selected by the second user command can be determined.

Alternatively, the user may select the first drawing figure by selecting information such as the mark of the drawing figure. A detailed description is omitted in this disclosure.

In the embodiment of the present disclosure, the specifying operation for the first drawing figure is realized by the user issuing the second user command. In order to facilitate adjustment of the drawing figure to be adjusted by the user, the user may himself/herself designate the first drawing figure to be adjusted. This improves the accuracy of the neural network obtained by machine learning, satisfies various user needs, and provides a good user experience.

In the field of artificial intelligence, semantic segmentation is one important research topic in the field of computer vision. Image semantic segmentation is applicable to many application scenes, such as object recognition, object detection, and so on. Semantic segmentation allows dividing an image into regions with different semantic information. For example, by performing semantic segmentation on a single image, objects in the image can be assigned semantic labels (eg, sky, lawn, people, trees, small animals, etc.).

During the process of labeling an image, a graphic is drawn on the image to indicate the labeled region. Usually, polygons select regions that have the same semantics. In an image targeted for semantic segmentation, multiple objects may occlude each other. For example, the content of the target image may be a person standing on a lawn, and the person's legs in the image may occlude part of the lawn. The edge of the occluded part is the semantic segmentation boundary, and the semantics are different on both sides of the boundary. In the process of labeling semantic units having different semantics in an image, when labeling regions on both sides of a boundary line, the boundary line may be labeled twice. Since the shape of the boundary line is usually irregular, the efficiency of image labeling is low when labeling twice.

In the application scene of semantic segmentation, this disclosure provides one possible implementation. In this implementation, the first labeling operation is for performing semantic segmentation labeling on the target image. The semantics of the image area indicated by the overlapping portion of the first drawing figure and the second drawing figure are the same as the semantics of the image area indicated by the second drawing figure, but the image other than the overlapping part of the first image area It may be different from the domain semantics. For example, referring to the example above, the first image area is the area where the grass is located, but includes the legs shielding the grass, and the second image area is the area where the person in the image is located. be. In this case, the overlapping part is removed from the first drawing figure, and the semantic of the first image region indicated by the obtained first drawing figure is more accurate, and the accuracy of the neural network obtained by machine learning is improved. can be made Further, when the user labels the boundary line between two semantic units by drawing the second drawing figure after the user has finished drawing the second drawing figure, in the process of drawing the first drawing figure, the user There is no need to redraw the boundary line between the two semantic units, and the user does not need to manually adjust the first drawing figure. , saving time in labeling.

In one possible implementation, the target image is displayed on a web page screen. This web page screen may be, for example, a web page screen encoded in HTML5 language. Then, in the process of drawing a figure, the labeling of the target image may be realized by drawing the figure on a canvas in HTML5.

In the following, the case where a target image is displayed on a web page screen is taken as an example, and the case where the first labeling operation is for performing semantic segmentation and labeling on the target image is taken as a specific application scene of the present disclosure. , exemplarily describe the labeling method according to the present disclosure. For matters not described in detail in this section, reference is made to the above related descriptions. The content in this section can also be used as an exemplary explanation for the above content.

Referring to FIG. 6, in one possible application scene according to the present disclosure, the labeling method according to the present disclosure includes steps 201-206.

In step 201, a target image is displayed on a web page screen.

At step 202, graphics are drawn on the target image using a canvas in HTML5 to obtain a plurality of drawn graphics.

A drawing figure may be a polygon. The specific process of figure drawing is as follows.
First, creation of polygonal points is realized by a mouseup event (mouseup) in the JavaScript language. Specifically, after the user clicks the left mouse button (presses and releases the left button), the canvas creates a starting point of a new shape, New polygon vertices can be given to a newly created figure. After the user clicks the right mouse button (presses and releases the right mouse button), the canvas connects the start and end points of the shape to create the polygon closure can be achieved.

The e event attribute of the callback event of the mouse move event (mousemove) in the JavaScript language captures the position of the mouse in the browser, determines the position of the user operation in the target image, and thereby determines the drawing figure drawn by the user. Determine the position in the target image.

There are a plurality of drawn figures, and the drawn figure is a figure drawn by the user for the area in the target image to be labeled.

At step 203, a selection operation command is received, and a first drawing figure is determined based on the selection operation command.

A selection operation command is used to select a figure to be adjusted from a plurality of drawing figures. The selection operation command is the aforementioned second user command. Here, the figure to be adjusted is referred to as a first drawing figure.

Based on the position indicated by the selection operation command on the web page screen and the positions of the plurality of drawing figures on the web page screen, it is determined that the position indicated by the selection operation command on the web page screen is inside the first drawing figure. case, the first drawing figure to be selected can be determined.

At step 204, an adjustment operation command is accepted.

The adjustment operation command is for instructing adjustment processing of the first drawing figure on the web page screen, and the adjustment operation command is the aforementioned first user command.

In step 205, it is detected whether or not there is a second drawing figure that overlaps with the first drawing figure among the plurality of drawing figures in the target image according to the adjustment operation command.

There is an overlapping portion between the first drawing figure and the second drawing figure. For example, the content of the target image is a person standing on a lawn, and the person's legs in the image occlude a portion of the lawn. The first drawing figure drawn by the user may be the area where the lawn is located. Since the edge of the lawn is generally straight rather than irregular, the first drawing figure includes the area where the legs of the person in the lawn are located so that the user can quickly draw the first drawing figure. Alternatively, the second drawing figure may be the area where the person is positioned.

In step 206, the portion of the first drawing figure that overlaps with the second drawing figure is removed from the first drawing figure to obtain the adjustment result.

When the first drawing figure is a frame figure, the first line segment element and the first drawing figure positioned within the first drawing figure of the second drawing figure based on the positions of the first drawing figure and the second drawing figure It is possible to determine line segment elements located outside the second drawing figure of . Then, a figure composed of the first line segment element and the line segment element positioned outside the second drawing figure of the first drawing figure is used as the adjustment result for the first drawing figure. The adjusted first drawing figure is used for labeling the semantic information of the first image area of the target image.

Also, based on the positions of the first drawing figure and the second drawing figure, a second line segment element positioned within the second drawing figure of the first drawing figure is determined, and the second line segment element is removed. In this way, there are no other line segment elements inside the second drawing figure.

The adjustment result for the first drawing figure is obtained by removing the region in the second drawing figure from the first drawing figure, and is used for labeling the semantic units in the target image, and the user performs the second drawing figure. When the boundary line between two semantic units is labeled by drawing a figure, the user needs to redraw the boundary line between the two semantic units in the process of drawing another semantic unit within it. It improves labeling efficiency and saves labeling time.

In one possible implementation, the above labeling method can be applied to User Equipment (UE), Mobile Equipment, User Terminal, Terminal, Cellular Phone, Cordless Phone, Personal Digital Assistant (PDA), Handheld It may be executed by an electronic device such as a terminal device such as an appliance, a computing device, an in-vehicle device, a wearable device, or a server, and may be realized by calling computer readable instructions stored in memory by a processor, or , may be executed by the server.

It should be understood that the above method embodiments referred to in this disclosure can be combined with each other to form embodiments without violating principles and logic. Due to space limitations, detailed descriptions are omitted in this disclosure. Those skilled in the art should understand that the execution order of each step in the above method according to the specific embodiments is specifically determined by its function and internal logic.

Note that the present disclosure further provides a labeling device, an electronic device, a computer-readable storage medium, and a program. Any one of these can be used to implement any one of the labeling methods according to the present disclosure. For the corresponding technical solutions and descriptions, please refer to the corresponding descriptions of the methods, and detailed descriptions are omitted.

FIG. 7 shows a block diagram of a labeling device according to an embodiment of the disclosure. As shown in FIG. 7, the device comprises:
a receiving module 301 that receives a first labeling operation for a target image, the first labeling operation for drawing a first drawing figure indicating a first image area in the target image;
an adjustment module 302 for adjusting the first drawing figure to obtain a labeling result corresponding to the first labeling operation when the second drawing figure and the first drawing figure in the target image at least partially overlap; Prepare.

In one possible implementation, the apparatus further includes a detection module for detecting whether or not there is a drawn figure that overlaps with the first drawn figure among at least one previous drawn figure in the target image. Prepare.

In one possible implementation, the device further comprises a first user command receiving module for receiving a first user command to instruct to adjust the first drawing figure.

In one possible implementation, the adjustment module 302 removes the portion of the first drawing figure that overlaps with the second drawing figure as the labeling result corresponding to the first labeling operation.

In one possible implementation, the adjustment module 302, when the first drawing figure and the second drawing figure are frame figures, based on the positions of the first drawing figure and the second drawing figure, Determining a first line segment element positioned within said first drawing figure of a second drawing figure and a second line segment element positioned outside said second drawing figure of said first drawing figure; and said first line segment element. The labeling result corresponding to the first labeling operation is a drawing figure composed of the segment element and the second line segment element.

In one possible implementation, the device comprises:
a display module for displaying the target image and a plurality of drawing figures including the first drawing figure in the target image on a display screen;
a second user command receiving module for receiving a second user command for selecting the first drawing figure from the plurality of drawing figures;
a determination module that determines the first drawing figure to be selected based on the position indicated by the second user command on the display screen and the positions of the plurality of drawing figures on the display screen.

In one possible implementation, the target image is displayed on a web page screen.

In one possible implementation, said first labeling operation is for performing semantic segmentation labeling on said target image.

In some embodiments, the functions or modules provided by the apparatus according to the embodiments of the present disclosure can be used to perform the methods described in the method embodiments above, and the specific implementations and technical effects thereof. can be referred to the description of the method embodiment above, and for the sake of simplification, the detailed description is omitted here.

An embodiment of the present disclosure further provides a computer readable storage medium having computer program instructions stored thereon, said computer program instructions, when executed by a processor, effecting the above method. . The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present disclosure includes a processor and a memory storing instructions executable by the processor, the processor being configured to invoke the instructions stored in the memory to perform the method. Further provided is an electronic device.

Embodiments of the present disclosure include computer readable code, and when the computer readable code is executed in a device, a processor in the device executes instructions for implementing a labeling method according to any of the above embodiments. Further provided is a computer program product for execution.

An embodiment of the present disclosure is a computer program product having computer readable instructions stored therein which, when executed, cause a computer to perform the operations of a labeling method according to any of the embodiments above. is further provided.

An electronic device may be provided as a terminal, server, or other form of device.

FIG. 8 is a block diagram illustrating electronic device 800 according to an embodiment of the present disclosure. For example, electronic device 800 may be a terminal such as a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, or the like.

Referring to FIG. 8, electronic device 800 includes processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component. 816 may be included.

The processing component 802 typically controls the overall operation of the electronic device 800, such as operations related to display, telephone calls, data communications, camera operations and recording operations. Processing component 802 may include one or more processors 820 that execute instructions to perform all or part of the steps of the methods described above. Processing component 802 may also include one or more modules for interaction with other components. For example, processing component 802 may include multimedia modules for interaction with multimedia component 808 .

Memory 804 is configured to store various types of data to support operations in electronic device 800 . These data include, by way of example, instructions for any application programs or methods that operate on electronic device 800, contact data, phone book data, messages, pictures, videos, and the like. Memory 804 may be, for example, static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM ), magnetic memory, flash memory, magnetic disk or optical disk, or any combination thereof.

Power supply component 806 provides power to each component of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components related to power generation, management, and distribution for electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and a user. In some examples, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, it may be implemented as a touch screen that receives input signals from the user. A touch panel includes one or more touch sensors to detect touches, slides, and gestures on the touch panel. The touch sensor may detect not only the boundaries of touch or slide movement, but also the duration and pressure associated with the touch or slide operation. In some examples, multimedia component 808 includes a front-facing camera and/or a rear-facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operational mode, such as a photographing mode or imaging mode. Each front and rear camera may have a fixed optical lens system or a focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC) that receives external audio signals when the electronic device 800 is in operational modes, such as call mode, recording mode and voice recognition mode. configured as The received audio signals may also be stored in memory 804 or transmitted via communication component 816 . In some examples, audio component 810 further includes a speaker for outputting audio signals.

I/O interface 812 provides an interface between processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, and the like. These buttons may include, but are not limited to, home button, volume button, start button and lock button.

Sensor component 814 includes one or more sensors for assessing the condition of various aspects of electronic device 800 . For example, the sensor component 814 can detect the on/off state of the electronic device 800, the relative positioning of components such as the display and keypad of the electronic device 800, and the sensor component 814 can further detect the electronic device 800 or the electronic device 800. Changes in the position of a certain component, presence or absence of contact between the user and the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and temperature changes of the electronic device 800 can be detected. Sensor component 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor component 814 may also include optical sensors for use in imaging applications, such as complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensors. In some examples, the sensor component 814 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to provide wired or wireless communication between electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile technology (2G) or a third generation mobile technology (3G), or a combination thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate near field communication. For example, the NFC module can be implemented by Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 includes one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processors (DSPDs), programmable logic devices (PLDs), field programmable gate arrays ( FPGA), controller, microcontroller, microprocessor or other electronic component, and can be used to perform the above methods.

The exemplary embodiment further provides a non-volatile computer-readable storage medium, e.g., memory 804, containing computer program instructions, which, when executed by processor 820 of electronic device 800, perform the methods described above. can be made

FIG. 9 is a block diagram illustrating an electronic device 1900 according to an embodiment of the disclosure. For example, electronic device 1900 may be provided as a server. Referring to FIG. 9, electronic device 1900 includes a processing component 1922, which includes one or more processors, and memory resources, typically memory 1932, for storing instructions, such as application programs, executable by processing component 1922. include. An application program stored in memory 1932 may include one or more modules each corresponding to a set of instructions. The processing component 1922 is also configured to perform the method by executing instructions.

Electronic device 1900 further includes a power component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input/output (I/O) interface 1950 configured to connect electronic device 1900 to a network. O) may include an interface 1958; Electronic device 1900 may be configured with an operating system stored in memory 1932, such as the Windows Server operating system (Windows Server ^™ ), a graphical user interface-based operating system (Mac OS X ^™ ) published by Apple Inc., a multi-user, multi-process computer. It can operate based on an operating system (Unix ^™ ), a Unix-like operating system with free and open source code (Linux ^™ ), a Unix-like operating system with open source code (FreeBSD ^™ ), or the like.

The exemplary embodiment further provides a non-volatile computer-readable storage medium, such as memory 1932, containing computer program instructions, which are executed by processing component 1922 of electronic device 1900 to perform the method. can be made

The present disclosure may be systems, methods and/or computer program products. The computer program product may include a computer readable storage medium having computer readable program instructions for causing a processor to implement aspects of the present disclosure.

A computer-readable storage medium may be a tangible device capable of storing and storing instructions for use with an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, e.g. Including mechanical encoding devices such as endoprotrusions, and any suitable combination of the above. Computer-readable storage media, as used herein, refers to instantaneous signals themselves, such as radio waves or other freely propagating electromagnetic waves, or electromagnetic waves propagated through waveguides or other transmission media (e.g., passing through fiber optic cables). pulsed light), or as an electrical signal transmitted via wires.

The computer readable program instructions described herein may be downloaded to each computing/processing device from a computer readable storage medium or may be downloaded externally via networks such as the Internet, local area networks, wide area networks and/or wireless networks. It may be downloaded to a computer or external storage device. A network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface within each computing/processing device receives computer-readable program instructions from the network, transfers the computer-readable program instructions for storage on a computer-readable storage medium within each computing/processing device.

Computer program instructions for performing operations of the present disclosure may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented instructions such as Smalltalk, C++, etc. The source or target code may be written in any combination of one or more programming languages, including programming languages and common procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on the user's computer. It may be executed entirely on a remote computer, or may be executed entirely on a remote computer or server. When involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or (e.g. Internet service It may be connected to an external computer (via the Internet using a provider). In some embodiments, state information in computer readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA), such that the electronic circuit Aspects of the present disclosure may be implemented by executing computer readable program instructions.

Aspects of the present disclosure have been described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure; It is to be understood that any combination of blocks in the flowchart and/or block diagrams can be implemented by computer readable program instructions.

These computer readable program instructions are provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus, and when these instructions are executed by the processor of the computer or other programmable data processing apparatus, the flowcharts and/or A device may be manufactured to implement the functions/acts specified in one or more blocks of the block diagrams. These computer readable program instructions may be stored on a computer readable storage medium and cause computers, programmable data processing devices and/or other devices to operate in a specific manner. Accordingly, the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture having instructions for implementing aspects of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

Computer readable program instructions are loaded into a computer, other programmable data processing device, or other machine and executed by the computer by causing the computer, other programmable data processing device, or other machine to perform a series of operational steps. A process may be generated. As such, instructions executed on a computer, other programmable data processing device, or other machine implement the functions/acts specified in one or more blocks of the flowchart illustrations and/or block diagrams.

The flowcharts and block diagrams in the drawings illustrate possible system architectures, functionality, and operation of systems, methods and computer program products according to embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram can represent a portion of a module, program segment, or instruction, which is a single unit for implementing a specified logical function. Contains one or more executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may be executed substantially in parallel, or may be executed in reverse order depending on the functionality involved. It should be noted that each block in the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by a dedicated system based on hardware that performs the specified functions or operations, or may be implemented by a dedicated system. It should also be noted that the implementation may be a combination of hardware and computer instructions.

The computer program product can be tangibly implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a Software Development Kit (SDK).

While embodiments of the present disclosure have been described above, the above description is illustrative only and is not intended to be exhaustive or limited to the embodiments shown. Various modifications and alterations will be apparent to those skilled in the art without departing from the scope and spirit of each described embodiment. The terms chosen herein are used to suitably interpret the principle, practical application, or improvement over existing technology of each embodiment, or to allow others skilled in the art to understand each embodiment presented herein. It is.

Claims (12)

receiving a first labeling operation for a target image, the first labeling operation for drawing a first drawing figure indicating a first image region in the target image;
obtaining a labeling result corresponding to the first labeling operation by adjusting the first drawing figure when the second drawing figure and the first drawing figure in the target image at least partially overlap;
A labeling method comprising: Further, before adjusting the first drawing figure,
2. The labeling according to claim 1, further comprising detecting whether or not there is a drawing figure having a portion overlapping with the first drawing figure among at least one previous drawing figure in the target image. Method. Further, before adjusting the first drawing figure,
3. A labeling method according to claim 1, comprising receiving a first user command to adjust said first drawing figure. Obtaining a labeling result corresponding to the first labeling operation by adjusting the first drawing figure described above includes:
4. The method according to any one of claims 1 to 3, further comprising setting a drawing figure obtained by removing a portion of said first drawing figure overlapping with said second drawing figure as a labeling result corresponding to said first labeling operation. The labeling method according to item 1. Obtaining a labeling result corresponding to the first labeling operation by adjusting the first drawing figure described above includes:
When the first drawing figure and the second drawing figure are frame figures, the position of the second drawing figure within the first drawing figure based on the positions of the first drawing figure and the second drawing figure Determining a first line segment element located outside the second drawing figure of the first drawing figure and a second line segment element located outside the second drawing figure;
5. The method according to any one of claims 1 to 4, further comprising: setting a drawing figure composed of said first line segment element and said second line segment element as a labeling result corresponding to said first labeling operation. The labeling method according to item 1. After receiving the first labeling operation for the target image, further
displaying on a display screen the target image and a plurality of drawing figures including the first drawing figure in the target image;
receiving a second user command for selecting the first drawing figure from the plurality of drawing figures;
determining the first drawing figure to be selected based on the position indicated by the second user command on the display screen and the positions of the plurality of drawing figures on the display screen. The labeling method according to any one of claims 1 to 5. The labeling method according to any one of claims 1 to 6, wherein the target image is displayed on a web page screen. The labeling method according to any one of claims 1 to 7, wherein the first labeling operation is for performing semantic segmentation labeling on the target image. a receiving module for receiving a first labeling operation on a target image, the first labeling operation for drawing a first drawing figure indicating a first image area in the target image;
an adjustment module for adjusting the first drawing figure to obtain a labeling result corresponding to the first labeling operation when the second drawing figure and the first drawing figure in the target image at least partially overlap;
A labeling device comprising: a processor;
a memory for storing instructions executable by the processor;
An electronic device, wherein the processor is configured to call instructions stored in the memory to perform the labeling method according to any one of claims 1 to 8. A computer readable storage medium having computer program instructions stored thereon, said computer program instructions being adapted to implement a labeling method according to any one of claims 1 to 8 when executed by a processor. A computer-readable storage medium characterized by: Instructions comprising computer readable code for implementing a labeling method according to any one of claims 1 to 8 by a processor in said electronic device when said computer readable code is executed in said electronic device. A computer program characterized by executing

Images