CN110910401A - Semi-automatic image segmentation data annotation method, electronic device and storage medium - Google Patents
Semi-automatic image segmentation data annotation method, electronic device and storage medium Download PDFInfo
- Publication number
- CN110910401A CN110910401A CN201911052198.7A CN201911052198A CN110910401A CN 110910401 A CN110910401 A CN 110910401A CN 201911052198 A CN201911052198 A CN 201911052198A CN 110910401 A CN110910401 A CN 110910401A
- Authority
- CN
- China
- Prior art keywords
- image
- coordinate
- target area
- edge
- outermost
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000003709 image segmentation Methods 0.000 title claims abstract description 41
- 230000002093 peripheral effect Effects 0.000 claims abstract description 37
- 230000011218 segmentation Effects 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 2
- 241000699666 Mus <mouse, genus> Species 0.000 description 20
- 238000002372 labelling Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 239000003550 marker Substances 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 241000699670 Mus sp. Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/181—Segmentation; Edge detection involving edge growing; involving edge linking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
Abstract
The invention discloses a semi-automatic image segmentation data annotation method, an electronic device and a storage medium, wherein based on a plurality of first pixel coordinates located in a real target area and a plurality of second pixel coordinates located in a real background area selected by a user on an image to be annotated, whether each pixel in the image to be annotated belongs to the real target area or the real background area is judged based on an energy function, the outermost peripheral coordinates of a prediction target area are output, and then category information corresponding to the prediction target area is input by the user, so that an image annotation task can be completed. Therefore, the semi-automatic image segmentation data annotation method provided by the embodiment greatly simplifies the number of times of mouse clicking during image annotation, reduces the cost of manual annotation, and accelerates the efficiency of manual annotation.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a semi-automatic image segmentation data annotation method, an electronic device and a storage medium.
Background
The image segmentation algorithm based on deep learning is widely applied to a series of applications needing to finely identify the category and the position of an object, such as a garbage classification system, an automatic driving system, a processing defect detection system and the like, and then the image segmentation algorithm based on deep learning needs a large amount of manually labeled data for training. The current main image segmentation data method is to perform mouse click marking point by observing with naked eyes and manually judging according to the edge of a target object.
Therefore, in the prior art, in order to meet the requirement of training of an image segmentation network based on deep learning on image annotation data, a technical crowdsourcing platform has been developed, and some companies recruit ten thousand data annotators, but because such a data annotation method depends on visual observation and a data annotation method of manual judgment, when one image is annotated, a mouse needs to be manually clicked dozens of times or even hundreds of times, so that the efficiency is low.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. Therefore, the invention provides a semi-automatic image segmentation data annotation method, an electronic device and a storage medium, which can reduce the times of manually clicking a mouse during image annotation and improve the annotation efficiency.
In a first aspect, an embodiment of the present invention provides a semi-automatic image segmentation data annotation method, including the following steps:
step S1, displaying an image to be annotated, wherein the image to be annotated comprises a real target area and a real background area;
step S2, acquiring a plurality of first pixel coordinates located in the real target area, and generating a target area seed point set by the first pixel coordinates;
step S3, acquiring a plurality of second pixel coordinates located in the real background area, and generating a background area seed point set by the plurality of second pixel coordinates;
step S4, establishing a target seed gray value histogram based on the target region seed point set;
step S5, establishing a background seed gray value histogram based on the background region seed point set;
step S6, establishing an undirected graph for representing the image to be labeled, constructing an energy function, and carrying out image segmentation on the image to be labeled based on a minimum segmentation algorithm to obtain a divided binary picture of the image to be labeled, wherein the binary picture comprises a prediction target area and a prediction background area;
step S7, acquiring the outermost peripheral coordinate points of the prediction target area on the binary image based on an edge tracking algorithm, and generating an edge coordinate set from a plurality of outermost peripheral coordinate points;
step S8, generating the contour of the prediction target area based on the edge coordinate set, and highlighting the contour on the image to be annotated;
step S9, judging whether a complete selection instruction is received, if so, acquiring the category information of the target area, and saving the edge coordinate set and the category information as a json file; if not, the process returns to step S1.
The semi-automatic image segmentation data annotation method provided by the embodiment of the invention at least has the following beneficial effects: the method is based on a plurality of first pixel coordinates which are selected by a user on an image to be labeled and located in the real target area and a plurality of second pixel coordinates which are selected by the user and located in the real background area, judges whether each pixel in the image to be labeled belongs to the real target area or the real background area based on an energy function, outputs the outermost peripheral coordinates of the prediction target area, and then the user inputs the category information corresponding to the prediction target area, so that the image labeling task can be completed. Therefore, the semi-automatic image segmentation data annotation method provided by the embodiment greatly simplifies the number of times of mouse clicking during image annotation, reduces the cost of manual annotation, and accelerates the efficiency of manual annotation.
In another specific embodiment of the present invention, the step of "generating a plurality of outermost peripheral coordinate points into an edge coordinate set" in S7 further includes the steps of:
step S7.1: establishing a set A of all the outermost coordinate points, establishing a set A', and establishing any one of the outermost coordinate points p in the set A0Adding into the set A', except p0The outermost coordinate points other than the outermost coordinate points establish a set A2Setting the first mark coordinate point p as p0；
Step S7.2: judgment set A2Whether the number of the middle elements is zero or not, if not, executing a step S7.2a, and if so, executing a step S7.2b;
step S7.2a: compute set A2The distance d between all the outermost coordinate points in (a) and the first mark coordinate point p, and the first mark coordinate point p is set as a set A2Corresponding to the minimum value of the distance d, set a2The outermost peripheral coordinate point of (d) corresponding to the minimum value of the distance d is added to and from the set a2Deleting, returning to step S7.2;
step S7.2b: sorting the outermost coordinate points in the order in which they were added to the set a';
step S7.3: establishing an edge coordinate set, and adding p0Adding the mark coordinate point into the edge coordinate set, deleting the mark coordinate point from the set A ', and setting a second mark coordinate point p' as p0；
Step S7.4: judging whether the number of elements in the set A' is one, if not, executing a step S7.4a, and if so, executing a step S7.4b;
step S7.4a: judging whether three points of the second mark coordinate point p 'and two outermost peripheral coordinate points which are sequenced at the first two positions in the set A' are collinear, if so, executing a step S7.4a1, and if not, executing a step S7.4a 2;
step S7.4a1: deleting the outermost peripheral coordinate point which is ranked at the top one in the set A 'from the set A', and returning to the step S7.4;
step S7.4a2: setting the second mark coordinate point p 'as the outermost peripheral coordinate point ranked at the first position in the set a', adding the outermost peripheral coordinate point ranked at the first position in the set a 'to the edge coordinate set and deleting the outermost peripheral coordinate point from the set a', and returning to step S7.4;
step S7.4b: and adding the outermost peripheral coordinate point in the set A' into the edge coordinate set, and outputting the edge coordinate set.
In another specific embodiment of the present invention, the step S8 further includes:
step S8.4: and carrying out shadow processing on the prediction target area on the image to be marked.
In a second aspect, an embodiment of the present invention provides an electronic device, including: memory, a processor and a computer program stored on the memory and executable on the processor, the processor when executing the program implementing a semi-automated image segmentation data annotation method according to any one of the first aspect of the invention.
The electronic device according to the embodiment of the present invention performs the semi-automatic image segmentation data annotation method according to any one of the first aspect of the present invention, so that all the advantages of the first aspect of the present invention are obtained.
In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, storing computer-executable instructions, where: the computer-executable instructions are for performing a method of semi-automated image segmentation data annotation according to any one of the first aspect of the present invention.
All the advantages of the first aspect of the present invention are achieved because the computer-readable storage medium of the embodiment of the present invention stores thereon computer-executable instructions for executing the semi-automatic image segmentation data annotation method according to any one of the first aspect of the present invention.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow chart of a method according to a second embodiment of the present invention;
FIG. 2 is a diagram illustrating an effect of a semi-automatic image segmentation data annotation method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a graph partitioning algorithm according to a second embodiment of the present invention;
FIG. 4 is a schematic structural diagram of an electronic device according to a first embodiment of the invention;
reference numerals:
Detailed Description
The embodiments of the present invention will be described in detail below, and the embodiments described below with reference to the drawings are exemplary only for the purpose of explanation, and should not be construed as limiting the invention.
In the description of the present invention, the meaning of a plurality of means is one or more, the meaning of a plurality of means is two or more, and larger, smaller, larger, etc. are understood as excluding the number, and larger, smaller, inner, etc. are understood as including the number. If the first and second are described for the purpose of distinguishing technical features, they are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
In the existing data annotation method, an open-source image segmentation data annotation tool (such as Labelme, Labelimg, and the like) is used to perform dotting plotting on an edge of a target image (the edge of a target object in the image is drawn by clicking a mouse, and if the dotting plotting is performed on the target object with a complex shape, the mouse may need to be clicked hundreds of times), then the target image is separated from a background, that is, the image is segmented to obtain the target image, and then the obtained target image is subjected to data annotation. If the target image shape is complex, it may be necessary to plot hundreds of points, i.e., the number of mouse clicks may be as high as hundreds of times, which may cause fatigue to the eyes of the data annotator. If the shape of the target image is complex, the hands of the data annotator are also tired and sore due to long-time mouse clicking. The data annotation method has low efficiency, and when a plurality of target images need to be annotated or the shapes of the target images are complex, the image segmentation time is long, the data annotation speed is seriously influenced, and the data annotation efficiency is low.
Based on the above, the invention provides a semi-automatic image segmentation data annotation method, an electronic device and a storage medium, wherein an image to be annotated is displayed on an electronic screen, a plurality of first pixel coordinates in a real target region and a plurality of second pixel coordinates in a real background region in the image to be annotated are obtained by a user twice on the image to be annotated respectively through a mouse, the image to be annotated is segmented through constructing an energy function and based on a minimal segmentation algorithm, an outermost peripheral coordinate point of a prediction target region corresponding to the real target region is obtained, finally, a difference between the prediction target region and the real target region is judged by a annotator, whether a 'complete selection instruction' is input or not is judged, so that a computer finally stores an edge coordinate set and the category information as a json file, and the final annotation is completed. Therefore, the invention can reduce the number of clicking mice when the annotator annotates data, and allows the annotator to select whether to accept the result of the automatic segmentation by whether to input the 'complete selection instruction', thereby improving the annotation efficiency and ensuring the annotation precision.
While the following text sets forth a number of different embodiments or examples for implementing different aspects of the invention, it should be understood that the following description is intended to be illustrative only and is not intended to be limiting.
Referring to fig. 4, an electronic device 100 according to a first embodiment of the invention includes a memory 102 and a processor 101, and fig. 4 illustrates the processor 101 and the memory 102 as an example.
The processor and memory may be connected by a bus or other means, such as by a bus in FIG. 4.
The memory 102, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer-executable programs. Further, the memory 102 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 102 may optionally include memory 102 located remotely from the processor, which may be connected to the electronic device 100 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Those skilled in the art will appreciate that the device architecture shown in fig. 4 does not constitute a limitation of electronic device 100 and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components.
Referring to fig. 1 and fig. 4, an electronic device according to a first embodiment of the present invention is shown, in which a processor 101 in the electronic device 100 may be configured to invoke a semi-automatic image segmentation data annotation method stored in a memory 102, and perform the following steps:
step S1, displaying an image to be annotated, wherein the image to be annotated comprises a real target area and a real background area;
step S2, acquiring a plurality of first pixel coordinates located in the real target area, and generating a target area seed point set by the first pixel coordinates;
step S3, acquiring a plurality of second pixel coordinates located in the real background area, and generating a background area seed point set by the plurality of second pixel coordinates;
step S4, establishing a target seed gray value histogram based on the target region seed point set;
step S5, establishing a background seed gray value histogram based on the background region seed point set;
step S6, establishing an undirected graph for representing the image to be labeled, constructing an energy function, and carrying out image segmentation on the image to be labeled based on a minimum segmentation algorithm to obtain a divided binary picture of the image to be labeled, wherein the binary picture comprises a prediction target area and a prediction background area;
step S7, acquiring the outermost peripheral coordinate points of the prediction target area on the binary image based on an edge tracking algorithm, and generating an edge coordinate set from a plurality of outermost peripheral coordinate points;
step S8, generating the contour of the prediction target area based on the edge coordinate set, and highlighting the contour on the image to be annotated;
step S9, judging whether a complete selection instruction is received, if so, acquiring the category information of the target area, and saving the edge coordinate set and the category information as a json file; if not, the process returns to step S1.
Based on the hardware structure of the electronic device, embodiments of a semi-automatic image segmentation data annotation method according to the present invention are provided.
Referring to fig. 1, a semi-automatic image segmentation data annotation method according to a second embodiment of the present invention includes the following steps:
step S1, displaying an image to be annotated, wherein the image to be annotated comprises a real target area and a real background area;
step S2, acquiring a plurality of first pixel coordinates located in the real target area, and generating a target area seed point set by the first pixel coordinates;
step S3, acquiring a plurality of second pixel coordinates located in the real background area, and generating a background area seed point set by the plurality of second pixel coordinates;
step S4, establishing a target seed gray value histogram based on the target region seed point set;
step S5, establishing a background seed gray value histogram based on the background region seed point set;
step S6, establishing an undirected graph for representing the image to be labeled, constructing an energy function, and carrying out image segmentation on the image to be labeled based on a minimum segmentation algorithm to obtain a divided binary picture of the image to be labeled, wherein the binary picture comprises a prediction target area and a prediction background area;
step S7, acquiring the outermost peripheral coordinate points of the prediction target area on the binary image based on an edge tracking algorithm, and generating an edge coordinate set from a plurality of outermost peripheral coordinate points;
step S8, generating the contour of the prediction target area based on the edge coordinate set, and highlighting the contour on the image to be annotated;
step S9, judging whether a complete selection instruction is received, if so, acquiring the category information of the target area, and saving the edge coordinate set and the category information as a json file; if not, the process returns to step S1.
In this embodiment, the image to be annotated may be displayed to the annotator through a PC computer screen or a tablet computer screen, and the image to be annotated includes a real target area and a real background area. For example, a lawn and a football are in the image to be labeled, and the marker judges that the football belongs to the real target area and the lawn belongs to the real background area. The method for obtaining the first pixel coordinate includes that a annotator continuously presses a left mouse button on a PC and drags the mouse button in a real target area to obtain pixel points of the real target area dragged by the mouse to serve as the first pixel coordinate, and the first pixel points can be displayed in the real target area for feeding back to the annotator. After the first pixel coordinate is obtained, a second pixel coordinate is obtained in the same manner in the real background area.
In this embodiment, an undirected graph G ═ V, E > is used to represent an image to be segmented (i.e., an image to be annotated), and V and E are sets of vertices (vertex) and edges (edge), respectively. The drawings herein are slightly different from the ordinary drawings. The ordinary graph is composed of vertexes and edges, if the edges are directional, the graph is called a directed graph, otherwise, the graph is an undirected graph, and the edges are weighted, and different edges can have different weights and respectively represent different physical meanings. The graph in this embodiment is obtained by adding 2 vertices on the basis of a general graph, where the 2 vertices are respectively denoted by symbols "S" and "T" and are collectively referred to as terminal vertices. All other vertices must be connected to these 2 vertices to form part of the set of edges. Therefore, the graph used to represent the image to be labeled in this embodiment has two kinds of vertices and two kinds of edges.
The first vertices and edges are: the first common vertex corresponds to each pixel in the image. The junction of every two neighborhood vertices (corresponding to every two neighborhood pixels in the image) is an edge. Such edges are also called n-links.
The second type of vertices and edges are: in addition to the image pixels, there are two other terminal vertices, called S and T, respectively. There is a connection between each common vertex and the 2 terminal vertices, forming a second edge. This edge is also called t-li nks.
Fig. 3 shows the corresponding s-t map of the image, each pixel corresponding to a respective vertex in the map, and in addition to both the s and t vertices. The graph in fig. 3 has two types of edges, one representing the edge n-links connecting every two common vertices of the neighborhood, and the other representing the edge t-links connecting each common vertex with s and t. In foreground and background segmentation, s generally represents a foreground object (i.e., a real object region) and t generally represents a background (i.e., a real background region).
Each edge of the graph in fig. 3 has a non-negative weight we, which can also be understood as a cost. A cut is a subset C of the edge set E in the graph, and the cost of the cut (denoted as | C |) is the sum of the weights of all the edges of the subset C of edges.
This embodiment requires finding a set of edges that includes the 2 edges above, the breaking of all the edges in the set resulting in a separation of the residual "S" and "T" graphs, so this set of edges is called a "cut". if a cut has the smallest sum of all the weights of its edges, this is called a minimum cut, i.e. the result of the graph cut.
The weight of the edge in the graph determines the final segmentation result, and the weight of the edge can be determined by an energy function. The image segmentation can be regarded as a pixel labeling problem, the label of the target (s-node) is set to 1, and the label of the background (t-node) is set to 0, and the process can be obtained by minimizing the energy function through the graph segmentation. It is clear that cut occurring at the boundary of the object and background is what we want (which is equivalent to cutting apart where the background and object are connected in the image, which is equivalent to segmenting it). At the same time, the energy should be minimal at this time as well. Let L ═ L be the label of each pixel of the whole image1,l2,,,lpIn which liIs 0 (indicating that the pixel point is located in the real background region) or 1 (indicating that the pixel point is located in the real target region). That assumes a segmentation of the image as L, the energy of the image can be expressed as:
E(L)＝aR(L)+B(L)
wherein, R (L) is a region term, B (L) is a boundary term, and a is an important factor between the region term and the boundary term, and determines the influence of the region term and the boundary term on energy. If a is 0, then only boundary factors are considered, not area factors. And E (L) represents weight, namely loss function, also called energy function, and the aim of graph cutting is to optimize the energy function to minimize the value.
As for the area item,
wherein R isp(lp) Assignment label l for a representation pixel ppPenalty of Rp(lp) The weighting of the energy terms can be obtained by comparing the gray level of the pixel p with a gray level histogram of the given object and foreground, in other words that the pixel p belongs to the label lpI want pixel p to be assigned to the label l whose probability is the greatestpAt this time, we want the energy to be minimum, so generally take the negative logarithm of the probability, so the weight of t-link is as follows:
Rp(1)＝-ln Pr(Lp|'obj')；Rp(0)＝-ln Pr(Lp|'bkg')
as can be seen from the above two equations, the probability Pr (Ip | 'obj') when the gray value of the pixel p belongs to the object is greater than the background Pr (L)pI 'bkg'), then Rp(1) Is less than Rp(0) That is, when the pixel p is more likely to belong to the target, classifying p as the target makes the energy r (l) small. Then, if all pixels are correctly classified as either object or background, then the energy is minimal at this time.
As for the boundary item, it is,
wherein p and q are neighborhood pixels, the boundary item mainly reflects the boundary attribute of the segmentation L, B<p,q>Can be resolved as a penalty for discontinuity between pixels p and q, in general if p and q are more similar (e.g. their gray scale), then B is<p,q>The larger, if they are very different, then B<p,q>It is close to 0. In other words, if the difference between two neighboring pixels is small, it is highly likely that the two neighboring pixels belong to the same object or the same background, and if the difference is large, it indicates that the two pixels are likely to be located at the edge of the object and the background, the probability of being divided is high, so when the difference between the two neighboring pixels is large, B is larger<p,q>The smaller, the smaller the energy.
In this embodiment, an image is divided into two disjoint parts, namely, a target part and a background part, and the two disjoint parts are realized by using a graph partitioning technology. First, the graph is composed of vertices and edges, the edges having weights. Then we need to construct a graph with two classes of vertices, two classes of edges and two classes of weights. The common vertex is composed of each pixel of the image, and then there is an edge between every two neighboring pixels, and its weight is determined by the above-mentioned "boundary term". There are also two terminal vertices s (object) and t (background), each common vertex and s have a connection, i.e. an edge whose weight is given by the "regional energy term" Rp(1) The weight of each common vertex and the edge connected with t is determined by a' regionEnergy term "Rp(0) To decide. Thus, the weights of all edges can be determined, i.e. the graph is determined. At this time, the minimum cut can be found through a min cut algorithm, where the min cut is a set of weights and minimum edges, and the breaking of the edges just can separate the target and the background, i.e., the min cut corresponds to the minimization of energy. And min cut and max flow (minimal cut) of the graph are equivalent, so min cut of s-t graph can be found by max flow algorithm.
In the embodiment, a graph structure is used for representing the image to be annotated, an energy function is constructed, and the segmentation of the image to be annotated is realized based on a minimum segmentation or maximum flow algorithm. After the image is divided, the image is divided into two parts, namely a prediction target area and a prediction background area, all pixels of the prediction target area are set to be black, all pixels of the prediction background area are set to be white, and therefore all pixels of the image to be marked are divided into two numerical values, namely black and white, and the image is a binary image. And acquiring the outermost peripheral coordinate point of the prediction target area on the binary image by using an edge tracking algorithm on the binary image according to the data.
Since the graph segmentation algorithm is used in this embodiment to automatically acquire the outermost coordinate point of the prediction target region, in order to facilitate the annotator to determine the accuracy of the outermost coordinate point of the prediction target region obtained by the graph segmentation of this time, the contour of the prediction target region is generated based on the edge coordinate set, and the contour is highlighted on the image to be annotated, which facilitates the comparison by the annotator.
When the annotator considers that the predicted target area obtained this time is ideal, a "full selection instruction" may be sent to the electronic device in the first embodiment, for example, the instruction may be sent by hitting an enter key on a keyboard. Then, the display screen displays an interface for inputting category information, the annotator inputs the category information corresponding to the predicted target area through a keyboard, such as football, and then the edge coordinate set and the category information are stored as json files, so that semi-automatic annotation is completed. Alternatively, when the annotator considers that the prediction target area obtained this time is not ideal, the annotator may send an "incomplete selection instruction" to the electronic device in the first embodiment, for example, the annotator may send the instruction to the electronic device by hitting a space bar on a keyboard, and after receiving the instruction, the electronic device re-executes step S1. Therefore, the semi-automatic image segmentation data annotation method provided by the embodiment can improve the annotation efficiency and allow an annotator to control the annotation precision.
In the semi-automatic image segmentation data annotation method according to the third embodiment of the present invention, based on the second embodiment, the step S7 of "generating an edge coordinate set from a plurality of outermost coordinate points" further includes the following steps:
step S7.1: establishing a set A of all the outermost coordinate points, establishing a set A', and establishing any one of the outermost coordinate points p in the set A0Adding into the set A', except p0The outermost coordinate points other than the outermost coordinate points establish a set A2Setting the first mark coordinate point p as p0；
Step S7.2: judgment set A2Whether the number of the middle elements is zero or not, if not, executing a step S7.2a, and if so, executing a step S7.2b;
step S7.2a: compute set A2The distance d between all the outermost coordinate points in (a) and the first mark coordinate point p, and the first mark coordinate point p is set as a set A2Corresponding to the minimum value of the distance d, set a2The outermost peripheral coordinate point of (d) corresponding to the minimum value of the distance d is added to and from the set a2Deleting, returning to step S7.2;
step S7.2b: sorting the outermost coordinate points in the order in which they were added to the set a';
step S7.3: establishing an edge coordinate set, and adding p0Adding the mark coordinate point into the edge coordinate set, deleting the mark coordinate point from the set A ', and setting a second mark coordinate point p' as p0；
Step S7.4: judging whether the number of elements in the set A' is one, if not, executing a step S7.4a, and if so, executing a step S7.4b;
step S7.4a: judging whether three points of the second mark coordinate point p 'and two outermost peripheral coordinate points which are sequenced at the first two positions in the set A' are collinear, if so, executing a step S7.4a1, and if not, executing a step S7.4a 2;
step S7.4a1: deleting the outermost peripheral coordinate point which is ranked at the top one in the set A 'from the set A', and returning to the step S7.4;
step S7.4a2: setting the second mark coordinate point p 'as the outermost peripheral coordinate point ranked at the first position in the set a', adding the outermost peripheral coordinate point ranked at the first position in the set a 'to the edge coordinate set and deleting the outermost peripheral coordinate point from the set a', and returning to step S7.4;
step S7.4b: and adding the outermost peripheral coordinate point in the set A' into the edge coordinate set, and outputting the edge coordinate set.
When the label is purely manually labeled, for the case that the partial outline of the real target area is a straight line, for example, for the case that the real target area is a square, the label usually only clicks four vertexes of the square with a mouse, and a straight line is pulled between two adjacent vertexes. Therefore, the coordinates representing the square only need four pixel points, and the data volume is greatly reduced. When the semi-automatic labeling mode is used, the edge coordinates of the predicted target area are obtained through an edge tracking algorithm and are composed of a series of pixel points which are adjacent to each other, so that the data volume is large.
Based on this, the present embodiment provides an algorithm that simplifies obtaining the outermost peripheral coordinate point of the prediction target region. The algorithm comprises two parts, the first part being step S7.1 to step S7.2, which sequence the obtained outermost coordinate points of the prediction target area in the order in which they were added to the set a'. If the outermost coordinate points are sequentially passed through in the order in which they are added to the set a', it is exactly the outline that encloses the prediction target area. Therefore, the second part consisting of step S7.3 to step S7.4 is to sequentially check whether three adjacent points on the contour are collinear according to the order in which the outermost coordinate point is added to the set a', and if so, remove the middle point, and only reserve the first and last two points, thereby realizing the effect during manual labeling and reducing the data amount generated by semi-automatic labeling.
Based on the second embodiment and the third embodiment, the semi-automatic image segmentation data annotation method according to the fourth embodiment of the present invention, in the step S8, "generate the contour of the prediction target region based on the edge coordinate set, and highlight the contour on the image to be annotated," further includes the following steps:
step S8.1: adding adjacent sequences to two outermost coordinate points in the edge coordinate set on the image to be marked, and connecting the two outermost coordinate points by using a straight line;
step S8.2: adding the last sequence to the outermost peripheral coordinate point and p in the edge coordinate set on the image to be labeled0Are connected by straight lines;
step S8.3: and generating the outline of the pixel point of the straight line passing through the image to be marked, and highlighting the pixel point corresponding to the straight line.
Based on the third embodiment, it can be seen that the outline of the prediction target area is just surrounded by the outermost coordinate points that are sequentially passed through in the order in which the outermost coordinate points are added to the set a'. Therefore, when the third embodiment is used for reducing the semi-automatic labeling data quantity, the generation of the outline of the prediction target area is facilitated, the operation time for generating the outline is reduced, and the algorithm efficiency is improved. Meanwhile, the operation of brightness increase and color white adjustment is carried out on the pixels of the outline, so that the user can conveniently identify the edge of the currently selected area.
Based on the fourth embodiment, the semi-automatic image segmentation data annotation method according to the fifth embodiment of the present invention, in step S8, further includes:
step S8.4: and carrying out shadow processing on the prediction target area on the image to be marked.
In this embodiment, the predicted target area is subjected to shading processing, and is output as an image with a darkened local area, so that a user can conveniently identify the selected local area in the target object.
A computer-readable storage medium according to a fifth embodiment of the present invention stores computer-executable instructions for performing the semi-automatic image segmentation data annotation method according to any one of the second to fifth embodiments.
Fig. 2 is a diagram illustrating an effect of processing an image to be annotated by using the semi-automatic image segmentation data annotation method according to the embodiment of the invention.
Firstly, an image to be marked is displayed on a computer screen, a real target area on the image to be marked is a football, and a real background area of the image to be marked is a lawn.
Secondly, moving the mouse to the football by the marker, clicking a left mouse button without putting the mouse, and dragging the mouse to draw a stroke on the football;
thirdly, the annotator moves the mouse to the lawn, clicks the left mouse button to be not placed and then drags the mouse to draw a stroke on the lawn;
fourthly, automatically acquiring outline coordinates of the football through image segmentation and processing football shadows;
and fifthly, combining the points of the football contour coordinates automatically obtained by dividing the figure on the same execution by executing the simplified algorithm of the third embodiment of the invention, thereby reducing the data volume.
The embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the gist of the present invention.
Claims (5)
1. A semi-automatic image segmentation data annotation method is characterized by comprising the following steps:
step S1, displaying an image to be annotated, wherein the image to be annotated comprises a real target area and a real background area;
step S2, acquiring a plurality of first pixel coordinates located in the real target area, and generating a target area seed point set by the first pixel coordinates;
step S3, acquiring a plurality of second pixel coordinates located in the real background area, and generating a background area seed point set by the plurality of second pixel coordinates;
step S4, establishing a target seed gray value histogram based on the target region seed point set;
step S5, establishing a background seed gray value histogram based on the background region seed point set;
step S6, establishing an undirected graph for representing the image to be labeled, constructing an energy function, and carrying out image segmentation on the image to be labeled based on a minimum segmentation algorithm to obtain a divided binary picture of the image to be labeled, wherein the binary picture comprises a prediction target area and a prediction background area;
step S7, acquiring the outermost peripheral coordinate points of the prediction target area on the binary image based on an edge tracking algorithm, and generating an edge coordinate set from a plurality of outermost peripheral coordinate points;
step S8, generating the contour of the prediction target area based on the edge coordinate set, and highlighting the contour on the image to be annotated;
step S9, judging whether a complete selection instruction is received, if so, acquiring the category information of the target area, and saving the edge coordinate set and the category information as a json file; if not, the process returns to step S1.
2. The semi-automated image segmentation data annotation method according to claim 1, wherein said step S7 of "generating edge coordinate sets from a plurality of said outermost coordinate points" further comprises the steps of:
step S7.1: establishing a set A of all the outermost coordinate points, establishing a set A', and establishing any one of the outermost coordinate points p in the set A0Adding into the set A', except p0Other than said outermost peripheral coordinatesPoint build set A2Setting the first mark coordinate point p as p0；
Step S7.2: judgment set A2Whether the number of the middle elements is zero or not, if not, executing a step S7.2a, and if so, executing a step S7.2b;
step S7.2a: compute set A2The distance d between all the outermost coordinate points in (a) and the first mark coordinate point p, and the first mark coordinate point p is set as a set A2Corresponding to the minimum value of the distance d, set a2The outermost peripheral coordinate point of (d) corresponding to the minimum value of the distance d is added to and from the set a2Deleting, returning to step S7.2;
step S7.2b: sorting the outermost coordinate points in the order in which they were added to the set a';
step S7.3: establishing an edge coordinate set, and adding p0Adding the mark coordinate point into the edge coordinate set, deleting the mark coordinate point from the set A ', and setting a second mark coordinate point p' as p0；
Step S7.4: judging whether the number of elements in the set A' is one, if not, executing a step S7.4a, and if so, executing a step S7.4b;
step S7.4a: judging whether three points of the second mark coordinate point p 'and two outermost peripheral coordinate points which are sequenced at the first two positions in the set A' are collinear, if so, executing a step S7.4a1, and if not, executing a step S7.4a 2;
step S7.4a1: deleting the outermost peripheral coordinate point which is ranked at the top one in the set A 'from the set A', and returning to the step S7.4;
step S7.4a2: setting the second mark coordinate point p 'as the outermost peripheral coordinate point ranked at the first position in the set a', adding the outermost peripheral coordinate point ranked at the first position in the set a 'to the edge coordinate set and deleting the outermost peripheral coordinate point from the set a', and returning to step S7.4;
step S7.4b: and adding the outermost peripheral coordinate point in the set A' into the edge coordinate set, and outputting the edge coordinate set.
3. The semi-automated image segmentation data annotation method according to claim 3, wherein said step S8 further comprises:
step S8.4: and carrying out shadow processing on the prediction target area on the image to be marked.
4. An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein: the processor, when executing the program, implements the semi-automated image segmentation data annotation method of any one of claims 1 to 3.
5. A computer-readable storage medium storing computer-executable instructions, characterized in that: the computer-executable instructions for performing the semi-automated image segmentation data annotation method of any one of claims 1 to 3.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911052198.7A CN110910401A (en) | 2019-10-31 | 2019-10-31 | Semi-automatic image segmentation data annotation method, electronic device and storage medium |
| PCT/CN2020/100347 WO2021082507A1 (en) | 2019-10-31 | 2020-07-06 | Semi-automated image segmentation and data annotation method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911052198.7A CN110910401A (en) | 2019-10-31 | 2019-10-31 | Semi-automatic image segmentation data annotation method, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110910401A true CN110910401A (en) | 2020-03-24 |
Family
ID=69814888
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911052198.7A Pending CN110910401A (en) | 2019-10-31 | 2019-10-31 | Semi-automatic image segmentation data annotation method, electronic device and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110910401A (en) |
| WO (1) | WO2021082507A1 (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111985394A (en) * | 2020-08-19 | 2020-11-24 | 东南大学 | Semi-automatic instance labeling method and system for KITTI data set |
| WO2021082507A1 (en) * | 2019-10-31 | 2021-05-06 | 五邑大学 | Semi-automated image segmentation and data annotation method, electronic device and storage medium |
| CN113111716A (en) * | 2021-03-15 | 2021-07-13 | 中国科学院计算机网络信息中心 | Remote sensing image semi-automatic labeling method and device based on deep learning |
| CN113806573A (en) * | 2021-09-15 | 2021-12-17 | 上海商汤科技开发有限公司 | Labeling method, labeling device, electronic equipment, server and storage medium |
| CN114485717A (en) * | 2021-12-30 | 2022-05-13 | 中智行(苏州)科技有限公司 | Road image ground marking method |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110254950A1 (en) * | 2008-10-09 | 2011-10-20 | Isis Innovation Limited | Visual tracking of objects in images, and segmentation of images |
| CN103227864A (en) * | 2013-04-10 | 2013-07-31 | 深圳市金立通信设备有限公司 | Method of detecting image edge and mobile terminal |
| CN103345743A (en) * | 2013-06-18 | 2013-10-09 | 宁波成电泰克电子信息技术发展有限公司 | Image segmentation method for intelligent flaw detection of cell tail end |
| CN104463067A (en) * | 2014-12-04 | 2015-03-25 | 四川大学 | Method for extracting macro blocks of Grid Matrix two-dimensional bar code |
| CN106203377A (en) * | 2016-07-20 | 2016-12-07 | 西安科技大学 | A kind of coal dust image-recognizing method |
| CN106875397A (en) * | 2017-01-04 | 2017-06-20 | 努比亚技术有限公司 | A kind of method for realizing interactive image segmentation, device and terminal |
| US20170367685A1 (en) * | 2015-02-16 | 2017-12-28 | Shenzhen Mindray Bio-Medical Electronics Co., Ltd. | Method for processing 3d image data and 3d ultrasonic imaging method and system |
| CN108960011A (en) * | 2017-05-23 | 2018-12-07 | 湖南生物机电职业技术学院 | The citrusfruit image-recognizing method of partial occlusion |
| CN109614871A (en) * | 2018-11-13 | 2019-04-12 | 远景能源（南京）软件技术有限公司 | Photovoltaic roof and photovoltaic barrier automatic identification algorithm |
| CN110020657A (en) * | 2019-01-15 | 2019-07-16 | 浙江工业大学 | A kind of bitmap silhouettes coordinate extraction method of cutting |
| CN110119741A (en) * | 2019-04-08 | 2019-08-13 | 浙江大学宁波理工学院 | A kind of card card image information recognition methods having powerful connections |
| CN110232365A (en) * | 2019-06-19 | 2019-09-13 | 清华大学 | The balancing method of cell outline bending degree, system and medium in cell image |
| CN110355365A (en) * | 2019-06-13 | 2019-10-22 | 西安交通大学 | A kind of EBM cooperateing with 840D digital control system changes direction and shapes Dynamic Slicing method |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915950B (en) * | 2015-04-29 | 2017-10-20 | 华南理工大学 | A kind of region based on energy constraint increases ultrasonoscopy automatic division method |
| CN105761266B (en) * | 2016-02-26 | 2018-09-28 | 民政部国家减灾中心 | The method that Rectangle building is extracted from remote sensing images |
| US10420523B2 (en) * | 2016-03-21 | 2019-09-24 | The Board Of Trustees Of The Leland Stanford Junior University | Adaptive local window-based methods for characterizing features of interest in digital images and systems for practicing same |
| CN106373168A (en) * | 2016-11-24 | 2017-02-01 | 北京三体高创科技有限公司 | Medical image based segmentation and 3D reconstruction method and 3D printing system |
| CN110910401A (en) * | 2019-10-31 | 2020-03-24 | 五邑大学 | Semi-automatic image segmentation data annotation method, electronic device and storage medium |
-
2019
- 2019-10-31 CN CN201911052198.7A patent/CN110910401A/en active Pending
-
2020
- 2020-07-06 WO PCT/CN2020/100347 patent/WO2021082507A1/en active Application Filing
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110254950A1 (en) * | 2008-10-09 | 2011-10-20 | Isis Innovation Limited | Visual tracking of objects in images, and segmentation of images |
| CN103227864A (en) * | 2013-04-10 | 2013-07-31 | 深圳市金立通信设备有限公司 | Method of detecting image edge and mobile terminal |
| CN103345743A (en) * | 2013-06-18 | 2013-10-09 | 宁波成电泰克电子信息技术发展有限公司 | Image segmentation method for intelligent flaw detection of cell tail end |
| CN104463067A (en) * | 2014-12-04 | 2015-03-25 | 四川大学 | Method for extracting macro blocks of Grid Matrix two-dimensional bar code |
| US20170367685A1 (en) * | 2015-02-16 | 2017-12-28 | Shenzhen Mindray Bio-Medical Electronics Co., Ltd. | Method for processing 3d image data and 3d ultrasonic imaging method and system |
| CN106203377A (en) * | 2016-07-20 | 2016-12-07 | 西安科技大学 | A kind of coal dust image-recognizing method |
| CN106875397A (en) * | 2017-01-04 | 2017-06-20 | 努比亚技术有限公司 | A kind of method for realizing interactive image segmentation, device and terminal |
| CN108960011A (en) * | 2017-05-23 | 2018-12-07 | 湖南生物机电职业技术学院 | The citrusfruit image-recognizing method of partial occlusion |
| CN109614871A (en) * | 2018-11-13 | 2019-04-12 | 远景能源（南京）软件技术有限公司 | Photovoltaic roof and photovoltaic barrier automatic identification algorithm |
| CN110020657A (en) * | 2019-01-15 | 2019-07-16 | 浙江工业大学 | A kind of bitmap silhouettes coordinate extraction method of cutting |
| CN110119741A (en) * | 2019-04-08 | 2019-08-13 | 浙江大学宁波理工学院 | A kind of card card image information recognition methods having powerful connections |
| CN110355365A (en) * | 2019-06-13 | 2019-10-22 | 西安交通大学 | A kind of EBM cooperateing with 840D digital control system changes direction and shapes Dynamic Slicing method |
| CN110232365A (en) * | 2019-06-19 | 2019-09-13 | 清华大学 | The balancing method of cell outline bending degree, system and medium in cell image |
Non-Patent Citations (2)
| Title |
|---|
| Y.Y. BOYKOV ET AL: "Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images", 《PROCEEDINGS EIGHTH IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION. ICCV 2001》 * |
| 陈学峰等: "《微计算机信息》", 《微计算机信息》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021082507A1 (en) * | 2019-10-31 | 2021-05-06 | 五邑大学 | Semi-automated image segmentation and data annotation method, electronic device and storage medium |
| CN111985394A (en) * | 2020-08-19 | 2020-11-24 | 东南大学 | Semi-automatic instance labeling method and system for KITTI data set |
| CN111985394B (en) * | 2020-08-19 | 2021-05-28 | 东南大学 | Semi-automatic instance labeling method and system for KITTI data set |
| CN113111716A (en) * | 2021-03-15 | 2021-07-13 | 中国科学院计算机网络信息中心 | Remote sensing image semi-automatic labeling method and device based on deep learning |
| CN113111716B (en) * | 2021-03-15 | 2023-06-23 | 中国科学院计算机网络信息中心 | Remote sensing image semiautomatic labeling method and device based on deep learning |
| CN113806573A (en) * | 2021-09-15 | 2021-12-17 | 上海商汤科技开发有限公司 | Labeling method, labeling device, electronic equipment, server and storage medium |
| CN114485717A (en) * | 2021-12-30 | 2022-05-13 | 中智行(苏州)科技有限公司 | Road image ground marking method |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021082507A1 (en) | 2021-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110910401A (en) | Semi-automatic image segmentation data annotation method, electronic device and storage medium | |
| CN110309824B (en) | Character detection method and device and terminal | |
| EP3745339A1 (en) | Method for implanting advertisements in video, and computer device | |
| CN110992384B (en) | Semi-automatic image data labeling method, electronic device and storage medium | |
| CN110210387B (en) | Method, system and device for detecting insulator target based on knowledge graph | |
| CN110796046A (en) | Intelligent steel slag detection method and system based on convolutional neural network | |
| CN111028261B (en) | High-precision semi-automatic image data annotation method, electronic device and storage medium | |
| CN110163198A (en) | A kind of Table recognition method for reconstructing, device and storage medium | |
| JP6089886B2 (en) | Region dividing method and inspection apparatus | |
| US8542951B2 (en) | Image retrieval device and computer program for image retrieval applicable to the image retrieval device | |
| JP2013125322A (en) | Learning device, program and learning method | |
| CN110910414B (en) | Image contour generation method, image labeling method, electronic device and storage medium | |
| CN105260428A (en) | Picture processing method and apparatus | |
| CN110992365A (en) | Loss function based on image semantic segmentation and design method thereof | |
| CN112102929A (en) | Medical image labeling method and device, storage medium and electronic equipment | |
| CN109064525A (en) | A kind of picture format conversion method, device, equipment and storage medium | |
| CN107274425B (en) | A kind of color image segmentation method and device based on Pulse Coupled Neural Network | |
| WO2023151299A1 (en) | Data generation method and apparatus, device, and storage medium | |
| CN115830470A (en) | Method, device and equipment for detecting weak-strength small-scale target of remote sensing image | |
| JP2012003358A (en) | Background determination device, method, and program | |
| CN114511862A (en) | Form identification method and device and electronic equipment | |
| CN113487697A (en) | Method and device for generating simple strokes, electronic equipment and storage medium | |
| CN109242750B (en) | Picture signature method, picture matching method, device, equipment and storage medium | |
| CN113361530A (en) | Image semantic accurate segmentation and optimization method using interaction means | |
| CN109871910A (en) | A kind of hand-written character recognition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200324 |
|
| RJ01 | Rejection of invention patent application after publication |