CN113838061A - Method and device for image annotation and storage medium - Google Patents
Method and device for image annotation and storage medium Download PDFInfo
- Publication number
- CN113838061A CN113838061A CN202110857660.1A CN202110857660A CN113838061A CN 113838061 A CN113838061 A CN 113838061A CN 202110857660 A CN202110857660 A CN 202110857660A CN 113838061 A CN113838061 A CN 113838061A
- Authority
- CN
- China
- Prior art keywords
- pixel
- superpixel
- point
- block
- blocks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Abstract
The invention discloses a method and a device for image annotation and a storage medium. The method comprises the following steps: acquiring an image to be marked; performing superpixel segmentation on the image to generate a plurality of superpixel blocks; respectively extracting the characteristics of a plurality of superpixel blocks; inputting the characteristics of a plurality of superpixel blocks into a neighbor propagation clustering algorithm to obtain a pixel block region; and acquiring the marking information of the pixel block area. The method determines the pixel block area based on the super-pixel segmentation technology and the neighbor propagation clustering algorithm, does not need to appoint the category number in advance, and has high clustering performance and efficiency, thereby shortening the period of image labeling, saving the time and labor cost and improving the efficiency of image labeling.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for image annotation, and a storage medium.
Background
In the deep learning semantic segmentation task, the processing of data is very important. In data processing, labeling of semantic segmentation data is very complicated and time-consuming, and a labeling person needs to label the category of each pixel, belonging to labeling at a pixel level. The existing image annotation is assisted by annotation software, and comprises the following steps: selecting a marked object picture, loading the picture to marking software, manually marking, adding a data set after marking is finished, and using the data set for subsequent model training. In the prior art, only the marking is carried out manually, which wastes time and labor and has low input-output ratio, thereby causing the efficiency of image marking to be lower.
Disclosure of Invention
The invention aims to provide a method, a device and a storage medium for image annotation, which are used for solving the problem of low efficiency of image annotation in the prior art.
In order to achieve the above object, a first aspect of the present invention provides a method for image annotation, the method comprising:
acquiring an image to be marked;
performing superpixel segmentation on the image to generate a plurality of superpixel blocks;
respectively extracting the characteristics of a plurality of superpixel blocks;
inputting the characteristics of a plurality of superpixel blocks into a neighbor propagation clustering algorithm to obtain a pixel block region;
and acquiring the marking information of the pixel block area.
In an embodiment of the present invention, super-pixel segmenting the image to generate a plurality of super-pixel blocks comprises:
initializing a superpixel block to obtain an initialized center point;
in the neighborhood of each initialization center point, calculating the gradient value of each pixel point, and taking the pixel point with the minimum gradient value as a new center point;
marking all pixel points belonging to the same central point neighborhood as the same superpixel label;
calculating the distance between each searched pixel point and all the central points;
updating the central point closest to each pixel point to be a corresponding clustering central point, and updating the super pixel label of each pixel point;
iterating each clustering central point and each super-pixel label until the clustering central point and the super-pixel label corresponding to each pixel point are not changed any more;
forming a superpixel block by each clustering central point and pixel points containing the same superpixel labels with the clustering central point to form a plurality of superpixel blocks;
the connectivity of a plurality of superpixel blocks is optimized.
In an embodiment of the present invention, initializing the super pixel block to obtain the initialization center point comprises:
and uniformly distributing the initialization center points in the image according to the set number of the super pixel blocks.
In an embodiment of the present invention, calculating the distance between each searched pixel point and all the center points comprises:
respectively calculating the color distance and the space distance between each searched pixel point and all the central points;
the distances of the pixel points from all the center points are determined based on the color distances and the spatial distances.
In an embodiment of the invention, the characteristics of the plurality of super pixel blocks comprise at least one of:
color, luminance, shape, texture, variance, covariance, and gradient.
In an embodiment of the present invention, inputting the characteristics of the plurality of superpixel blocks to a neighbor propagation clustering algorithm to obtain a pixel block region comprises:
establishing a similarity matrix according to a plurality of super pixel blocks;
updating the attraction information of each superpixel block relative to other superpixel blocks in the similarity matrix under the condition of giving attribution;
under the condition of giving attraction, updating attribution degree information of each super-pixel block relative to other super-pixel blocks in the similarity matrix;
summing and iterating the attraction degree information and the attribution degree information of each superpixel block;
determining a target superpixel block serving as a clustering center and a sub superpixel block corresponding to the target superpixel block based on an iteration result;
and determining a region formed by the target super pixel block and the sub super pixel block as a pixel block region.
In an embodiment of the present invention, further comprising:
before acquiring the marking information of the pixel block area, acquiring the modification information of the pixel block area.
The second aspect of the present invention provides an apparatus for image annotation, configured to execute the above method for image annotation.
A third aspect of the invention provides a machine-readable storage medium having stored thereon instructions for causing a machine to perform the above-described method for image annotation.
A fourth aspect of the invention provides a computer program product comprising a computer program which, when executed by a processor, implements the method for image annotation described above.
According to the technical scheme, the pixel block area is obtained based on the super-pixel segmentation technology and the neighbor propagation clustering algorithm, the pixel block area is labeled in an auxiliary manual labeling mode, the neighbor propagation clustering algorithm does not need to appoint the category number in advance, and the clustering performance and efficiency are high, so that the image labeling period is shortened, the time and labor cost is saved, and the image labeling efficiency is improved.
Additional features and advantages of the invention will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
FIG. 1 is a flow chart of a method for image annotation according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a method for generating a plurality of superpixel blocks according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating a method for obtaining a pixel block region according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an apparatus for image annotation according to an embodiment of the present invention.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present invention, are given by way of illustration and explanation only, not limitation.
It should be noted that, if directional indications (such as up, down, left, right, front, and back … …) are involved in the embodiment of the present invention, the directional indications are only used to explain the relative positional relationship between the components, the movement situation, and the like in a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indications are changed accordingly.
In addition, if there is a description of "first", "second", etc. in an embodiment of the present invention, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.
Fig. 1 is a flowchart illustrating a method for image annotation according to an embodiment of the present invention. As shown in fig. 1, an embodiment of the present invention provides a method for image annotation, which may include the following steps.
In step S11, an image to be annotated is acquired. In the embodiment of the invention, the image to be annotated can be acquired by the image acquisition device. The image capturing device may be a handheld camera device or a camera device disposed at a fixed position, including but not limited to a camera, a mobile phone, a tablet, a camera, and the like. The image to be annotated is acquired through the image acquisition equipment, the image is input into the computer equipment, and the computer equipment receives the image to be annotated sent by the image acquisition equipment and stores the received image to be annotated.
In step S12, the image is superpixel divided to generate a plurality of superpixel blocks. In the embodiment of the present invention, the super-pixel segmentation refers to an irregular pixel block with certain visual significance, which is composed of adjacent pixels with similar texture, color, brightness and other characteristics. The method groups pixels by utilizing the similarity of the features between the pixels, replaces a large number of pixels with a small number of super pixels to express the picture features, and greatly reduces the complexity of image post-processing. In embodiments of the invention, Simple Linear Iterative Clustering (SLIC) superpixel segmentation may be used. SLIC is a process of converting a color image into a 5-dimensional feature vector in a CIELAB color space and XY coordinates, then constructing a distance measurement standard for the 5-dimensional feature vector, and performing local clustering on image pixels. The SLIC algorithm can generate compact and approximately uniform superpixels, and has high comprehensive evaluation in the aspects of operation speed, object contour maintenance and superpixel shapes. By performing superpixel segmentation on the image, a superpixel block which is as compact and regular as a cell can be generated, the neighborhood characteristics are relatively easy to express, and not only a color image but also a gray image can be segmented. Compared with other super-pixel segmentation methods, the SLIC is more ideal in the aspects of running speed, compactness of super-pixel generation and contour preservation.
In step S13, the features of the plurality of superpixel blocks are extracted, respectively. In embodiments of the present invention, the characteristics of the superpixel blocks may include, but are not limited to, color, luminance, shape, texture, mean, variance, covariance, gradient, and the like. Similar superpixel blocks may have similar characteristics, such as similar color, similar brightness, similar texture, and so on. Extracting the characteristics of each super pixel block can facilitate the subsequent clustering of the pixel block areas.
In step S14, the features of the plurality of superpixel blocks are input to a neighbor propagation clustering algorithm to obtain a pixel block region. In the embodiment of the present invention, an Affinity Propagation (AP) clustering algorithm refers to continuously transmitting information between different points, so as to select a final clustering center to complete clustering. The AP clustering algorithm does not need to specify the number of final clusters, and the existing data points are used as final clustering centers, so that no requirement is made on the symmetry of the initial similarity matrix data, and the square error of the result is small. Inputting the characteristics of the multiple superpixel blocks into an AP clustering algorithm, and clustering the multiple superpixel blocks into a large pixel block region through the AP clustering algorithm, so as to obtain multiple pixel block regions, and facilitate marking personnel to mark the multiple pixel block regions.
In step S15, label information of the pixel block region is acquired. In the embodiment of the invention, after a plurality of pixel block regions are obtained through the AP clustering algorithm, the image to be marked is displayed on the terminal. The marking personnel can mark the plurality of pixel block areas presented by the picture respectively, and the computer equipment can acquire marking information of the pixel block areas through the marked picture transmitted by the terminal.
The embodiment of the invention obtains the pixel block region based on the super-pixel segmentation technology and the neighbor propagation clustering algorithm, labels the pixel block region in an auxiliary manual labeling mode, and the neighbor propagation clustering algorithm does not need to appoint the category number in advance, so that the clustering performance and efficiency are high, the image labeling period is shortened, the time and labor cost is saved, and the image labeling efficiency is improved.
FIG. 2 is a flow chart of a method for generating a plurality of superpixel blocks according to an embodiment of the present invention. As shown in fig. 2, the step S12 of performing superpixel segmentation on the image to generate a plurality of superpixel blocks may include:
step S21, initializing a superpixel block to obtain an initialized central point;
step S22, calculating the gradient value of each pixel point in the neighborhood of each initialization center point, and taking the pixel point with the minimum gradient value as a new center point;
step S23, marking all pixel points belonging to the same central point neighborhood as the same superpixel label;
step S24, calculating the distance between each searched pixel point and all the center points;
step S25, updating the central point nearest to each pixel point to the corresponding clustering central point, and updating the super pixel label of each pixel point;
step S26, iterating each clustering central point and each super-pixel label until the clustering central point and the super-pixel label corresponding to each pixel point are not changed any more;
step S27, forming a superpixel block by each clustering center point and the pixel points containing the same superpixel labels with the clustering center points to form a plurality of superpixel blocks;
and step S28, optimizing the connectivity of the plurality of superpixel blocks.
In an embodiment of the present invention, SLIC superpixel segmentation may be performed on an image to be annotated to obtain a plurality of superpixel blocks. SLIC is a process of converting a color image into a 5-dimensional feature vector in a CIELAB color space and XY coordinates, then constructing a distance measurement standard for the 5-dimensional feature vector, and performing local clustering on image pixels.
In an embodiment of the present invention, the initialized center point is an initialized cluster center. Initializing the super pixel block to obtain the initialized center point may include: and uniformly distributing the initialization center points in the image according to the set number of the super pixel blocks. For example, if the set number k of superpixels is input and pre-divided into k superpixels of the same size, and the size of each superpixel is n/k, the distance (step size) between adjacent initialization center points is approximately S ═ sqrt (n/k), so that the initialization center points can be evenly distributed in the image to be labeled. And calculating the gradient value of each pixel point in the n-n neighborhood of each initialization center point, and taking the pixel point with the minimum gradient value as a new center point. The purpose of this is to avoid the center point falling on the contour boundary with larger gradient so as not to affect the subsequent clustering effect. Each pixel point is assigned a superpixel label (i.e., to which center point it belongs) in the neighborhood around each center point. The search range of the SLIC is limited to 2S by 2S, which can accelerate the convergence of the algorithm, thereby marking all the pixel points belonging to the same center point neighborhood as the same superpixel label. Because each pixel point can be searched by a plurality of central points, for each searched pixel point, the distance between each pixel point and all the central points needs to be calculated, the central point corresponding to the minimum value is taken as the clustering central point of the pixel point, and the super pixel label of each pixel point is updated. And continuously iterating each clustering central point and each super-pixel label, namely repeating the steps until the error is converged, namely the clustering central point and the super-pixel label corresponding to each pixel point are not changed any more. Multiple connectivity situations, superpixel undersize, situations where a single superpixel is cut into multiple non-contiguous superpixels, etc. may occur over an iteration. Therefore, it is also necessary to optimize the connectivity of multiple superpixel blocks, and label some multiple connected or connected blocks with proper domain labels through small equal pixel blocks. For example, a marking table is newly created, elements in the table are all-1, discontinuous superpixels and undersize superpixels are redistributed to adjacent superpixels according to the Z-shaped trend (from left to right and from top to bottom), and traversed pixel points are distributed to corresponding labels until all pixel points are traversed.
Compact and approximately uniform superpixels can be generated by SLIC superpixel segmentation on an image to be annotated, and the method has high comprehensive evaluation in the aspects of operation speed, object contour maintenance and superpixel shape. By performing superpixel segmentation on the image, a superpixel block which is as compact and regular as a cell can be generated, the neighborhood characteristics are relatively easy to express, and not only a color image but also a gray image can be segmented. Compared with other super-pixel segmentation methods, the SLIC is more ideal in the aspects of running speed, compactness of super-pixel generation and contour preservation.
In an embodiment of the present invention, calculating the distance between each searched pixel point and all the center points may include:
respectively calculating the color distance and the space distance between each searched pixel point and all the central points;
the distances of the pixel points from all the center points are determined based on the color distances and the spatial distances.
Specifically, the distance between the pixel point and the central point in the embodiment of the present invention may include a color distance and a spatial distance. And for each searched pixel point, respectively calculating the color distance and the space distance between the pixel point and all the central points to determine the distance between the pixel point and all the central points.
In the embodiment of the invention, the color distance calculation method comprises the following steps:
wherein d iscIndicating the color distance,/j-liDenotes the lightness difference, a, between the j-th point and the i-th pointj-aiRepresenting the difference in red and green of the j-th and i-th points, bj-biIndicating the difference in yellow-blue color between the j-th and i-th points.
The calculation method of the spatial distance is as follows:
wherein d issRepresenting spatial distance, xj-xiRepresents the distance difference between the j-th point and the i-th point on the horizontal axis, yj-yiRepresenting the difference in distance of the j-th point and the i-th point on the vertical axis.
The distance is calculated as follows:
wherein d iscIndicating the color distance, dsRepresenting spatial distance, NsIs the maximum spatial distance, NcThe maximum color distance. N is a radical ofsIs defined as NsMaximum color distance N ═ sqrt (N/k)cThe method is different not only with different pictures, but also with different clusters, and a fixed constant can be used for substitution.
Fig. 3 is a flowchart illustrating a method for obtaining a pixel block region according to an embodiment of the present invention. As shown in fig. 3, the step S14 of inputting the features of the plurality of superpixel blocks into the neighbor propagation clustering algorithm to obtain the pixel block region may include:
step S31, establishing a similarity matrix according to a plurality of superpixel blocks;
step S32, under the condition of giving attribution degree, updating the attraction degree information of each superpixel block relative to other superpixel blocks in the similarity matrix;
step S33, under the condition of giving attraction, updating the attribution degree information of each superpixel block relative to other superpixel blocks in the similarity matrix;
step S34, summing and iterating the attraction degree information and the attribution degree information of each superpixel block;
step S35, determining a target superpixel block serving as a clustering center and a sub superpixel block corresponding to the target superpixel block based on an iteration result;
and step S36, determining the area formed by the target superpixel block and the sub superpixel block as a pixel block area.
In the embodiment of the present invention, an AP clustering algorithm may be used to cluster a plurality of super pixel blocks into a large pixel block region, so that a labeling person labels each pixel block region. The purpose of the AP clustering algorithm is to find the cluster center of each superpixel block and determine the superpixel blocks of the same cluster center as the points of the same pixel block region. Firstly, establishing a similarity matrix according to a plurality of superpixel blocks, and then determining the clustering center of the superpixel blocks based on the similarity matrix, namely the superpixel block with the largest attraction degree and attribution degree for the current pixel block. Assuming that two superpixel blocks i and k exist, the attribution a (i, k) is the attribution sent from the candidate clustering center k to the point i, and reflects the appropriateness of the point i to select the point k as the clustering center after considering the support of other point-to-point k as the clustering center. The attraction degree r (i, k) is an attraction degree transmitted from the candidate cluster center i to the point k, and reflects a degree to which the point k is suitable as a cluster center of the point i in consideration of potential cluster centers of other points. Updating the attraction information of each superpixel block relative to other superpixel blocks in the similarity matrix under the condition of giving attribution; and under the condition of giving the attraction degree, updating the attribution degree information of each superpixel block relative to other superpixel blocks in the similarity matrix, and summing and iterating the attraction degree information and the attribution degree information of each superpixel block. At any time, the attraction degree and the attribution degree can be added to obtain a clustering center. If the preset iteration number is reached, or the clustering center does not change any more, or the decision of the sample point in one sub-area does not change any more after several iterations, the iteration can be terminated. And determining a target superpixel block serving as a clustering center and a sub superpixel block corresponding to the target superpixel block based on the terminated iteration result, and determining a region formed by the target superpixel block and the sub superpixel block as a pixel block region. Therefore, the superpixel blocks with the same semantics can be clustered into pixel block regions through the AP clustering algorithm, and labeling personnel can conveniently label the superpixel blocks.
In the embodiment of the present invention, the method may further include:
before acquiring the marking information of the pixel block area, acquiring the modification information of the pixel block area.
Specifically, after a plurality of pixel block regions are obtained through the AP clustering algorithm, an image to be labeled can be displayed on the terminal. The marking personnel can correspondingly modify a plurality of pixel block areas presented by the picture, and then mark the pixel block areas according to the modified marking information, and the computer equipment can acquire the marking information of the pixel block areas through the marked picture transmitted by the terminal. Therefore, the work of manually selecting the pixel block area is reduced, the accuracy of the labeling information is improved, and the labeling efficiency and accuracy are improved.
Fig. 4 is a schematic structural diagram of an apparatus for image annotation according to an embodiment of the present invention. As shown in fig. 4, the present invention provides an apparatus for image annotation, configured to execute the above method for image annotation. In an embodiment of the application, the apparatus may include a processor 410 and a memory 420. The memory 420 may store instructions that, when executed by the processor 410, may cause the processor 410 to perform the method for image annotation described in the previous embodiments.
Specifically, in one embodiment of the present invention, the processor 410 is configured to:
acquiring an image to be marked;
performing superpixel segmentation on the image to generate a plurality of superpixel blocks;
respectively extracting the characteristics of a plurality of superpixel blocks;
inputting the characteristics of a plurality of superpixel blocks into a neighbor propagation clustering algorithm to obtain a pixel block region;
and acquiring the marking information of the pixel block area.
In the embodiment of the invention, the image to be annotated can be acquired by the image acquisition device. The image capturing device may be a handheld camera device or a camera device disposed at a fixed position, including but not limited to a camera, a mobile phone, a tablet, a camera, and the like. The image to be annotated is acquired through the image acquisition equipment, the image is input into the computer equipment, and the computer equipment receives the image to be annotated sent by the image acquisition equipment and stores the received image to be annotated.
In the embodiment of the present invention, the super-pixel segmentation refers to an irregular pixel block with certain visual significance, which is composed of adjacent pixels with similar texture, color, brightness and other characteristics. The method groups pixels by utilizing the similarity of the features between the pixels, replaces a large number of pixels with a small number of super pixels to express the picture features, and greatly reduces the complexity of image post-processing. In embodiments of the invention, Simple Linear Iterative Clustering (SLIC) superpixel segmentation may be used. SLIC is a process of converting a color image into a 5-dimensional feature vector in a CIELAB color space and XY coordinates, then constructing a distance measurement standard for the 5-dimensional feature vector, and performing local clustering on image pixels. The SLIC algorithm can generate compact and approximately uniform superpixels, and has high comprehensive evaluation in the aspects of operation speed, object contour maintenance and superpixel shapes. By performing superpixel segmentation on the image, a superpixel block which is as compact and regular as a cell can be generated, the neighborhood characteristics are relatively easy to express, and not only a color image but also a gray image can be segmented. Compared with other super-pixel segmentation methods, the SLIC is more ideal in the aspects of running speed, compactness of super-pixel generation and contour preservation.
In embodiments of the present invention, the characteristics of the superpixel blocks may include, but are not limited to, color, luminance, shape, texture, mean, variance, covariance, gradient, and the like. Similar superpixel blocks may have similar characteristics, such as similar color, similar brightness, similar texture, and so on. Extracting the characteristics of each super pixel block can facilitate the subsequent clustering of the pixel block areas.
In the embodiment of the invention, the AP clustering algorithm is to select a final clustering center by continuously transmitting information between different points so as to complete clustering. The AP clustering algorithm does not need to specify the number of final clusters, and the existing data points are used as final clustering centers, so that no requirement is made on the symmetry of the initial similarity matrix data, and the square error of the result is small. Inputting the characteristics of the multiple superpixel blocks into an AP clustering algorithm, and clustering the multiple superpixel blocks into a large pixel block region through the AP clustering algorithm, so as to obtain multiple pixel block regions, and facilitate marking personnel to mark the multiple pixel block regions.
In the embodiment of the invention, after a plurality of pixel block regions are obtained through the AP clustering algorithm, the image to be marked is displayed on the terminal. The marking personnel can mark the plurality of pixel block areas presented by the picture respectively, and the computer equipment can acquire marking information of the pixel block areas through the marked picture transmitted by the terminal.
The embodiment of the invention obtains the pixel block region based on the super-pixel segmentation technology and the neighbor propagation clustering algorithm, labels the pixel block region in an auxiliary manual labeling mode, and the neighbor propagation clustering algorithm does not need to appoint the category number in advance, so that the clustering performance and efficiency are high, the image labeling period is shortened, the time and labor cost is saved, and the image labeling efficiency is improved.
Further, the processor 410 is further configured to:
super-pixel segmenting the image to generate a plurality of super-pixel blocks may include:
initializing a superpixel block to obtain an initialized center point;
in the neighborhood of each initialization center point, calculating the gradient value of each pixel point, and taking the pixel point with the minimum gradient value as a new center point;
marking all pixel points belonging to the same central point neighborhood as the same superpixel label;
calculating the distance between each searched pixel point and all the central points;
updating the central point closest to each pixel point to be a corresponding clustering central point, and updating the super pixel label of each pixel point;
iterating each clustering central point and each super-pixel label until the clustering central point and the super-pixel label corresponding to each pixel point are not changed any more;
forming a superpixel block by each clustering central point and pixel points containing the same superpixel labels with the clustering central point to form a plurality of superpixel blocks;
the connectivity of a plurality of superpixel blocks is optimized.
In an embodiment of the present invention, SLIC superpixel segmentation may be performed on an image to be annotated to obtain a plurality of superpixel blocks. SLIC is a process of converting a color image into a 5-dimensional feature vector in a CIELAB color space and XY coordinates, then constructing a distance measurement standard for the 5-dimensional feature vector, and performing local clustering on image pixels.
In an embodiment of the present invention, the initialized center point is an initialized cluster center. Initializing the super pixel block to obtain the initialized center point may include: and uniformly distributing the initialization center points in the image according to the set number of the super pixel blocks. For example, if the set number k of superpixels is input and pre-divided into k superpixels of the same size, and the size of each superpixel is n/k, the distance (step size) between adjacent initialization center points is approximately S ═ sqrt (n/k), so that the initialization center points can be evenly distributed in the image to be labeled. And calculating the gradient value of each pixel point in the n-n neighborhood of each initialization center point, and taking the pixel point with the minimum gradient value as a new center point. The purpose of this is to avoid the center point falling on the contour boundary with larger gradient so as not to affect the subsequent clustering effect. Each pixel point is assigned a superpixel label (i.e., to which center point it belongs) in the neighborhood around each center point. The search range of the SLIC is limited to 2S by 2S, which can accelerate the convergence of the algorithm, thereby marking all the pixel points belonging to the same center point neighborhood as the same superpixel label. Because each pixel point can be searched by a plurality of central points, for each searched pixel point, the distance between each pixel point and all the central points needs to be calculated, the central point corresponding to the minimum value is taken as the clustering central point of the pixel point, and the super pixel label of each pixel point is updated. And continuously iterating each clustering central point and each super-pixel label, namely repeating the steps until the error is converged, namely the clustering central point and the super-pixel label corresponding to each pixel point are not changed any more. Multiple connectivity situations, superpixel undersize, situations where a single superpixel is cut into multiple non-contiguous superpixels, etc. may occur over an iteration. Therefore, it is also necessary to optimize the connectivity of multiple superpixel blocks, and label some multiple connected or connected blocks with proper domain labels through small equal pixel blocks. For example, a marking table is newly created, elements in the table are all-1, discontinuous superpixels and undersize superpixels are redistributed to adjacent superpixels according to the Z-shaped trend (from left to right and from top to bottom), and traversed pixel points are distributed to corresponding labels until all pixel points are traversed.
Compact and approximately uniform superpixels can be generated by SLIC superpixel segmentation on an image to be annotated, and the method has high comprehensive evaluation in the aspects of operation speed, object contour maintenance and superpixel shape. By performing superpixel segmentation on the image, a superpixel block which is as compact and regular as a cell can be generated, the neighborhood characteristics are relatively easy to express, and not only a color image but also a gray image can be segmented. Compared with other super-pixel segmentation methods, the SLIC is more ideal in the aspects of running speed, compactness of super-pixel generation and contour preservation.
In an embodiment of the present invention, calculating the distance between each searched pixel point and all the center points may include:
respectively calculating the color distance and the space distance between each searched pixel point and all the central points;
the distances of the pixel points from all the center points are determined based on the color distances and the spatial distances.
Specifically, the distance between the pixel point and the central point in the embodiment of the present invention may include a color distance and a spatial distance. And for each searched pixel point, respectively calculating the color distance and the space distance between the pixel point and all the central points to determine the distance between the pixel point and all the central points.
In the embodiment of the invention, the color distance calculation method comprises the following steps:
wherein d iscIndicating the color distance,/j-liDenotes the lightness difference, a, between the j-th point and the i-th pointj-aiRepresenting the difference in red and green of the j-th and i-th points, bj-biIndicating the difference in yellow-blue color between the j-th and i-th points.
The calculation method of the spatial distance is as follows:
wherein d issRepresenting spatial distance, xj-xiRepresents the distance difference between the j-th point and the i-th point on the horizontal axis, yj-yiRepresenting the difference in distance of the j-th point and the i-th point on the vertical axis.
The distance is calculated as follows:
wherein d iscIndicating the color distance, dsRepresenting spatial distance, NsIs the maximum spatial distance, NcThe maximum color distance. N is a radical ofsIs defined as NsMaximum color distance N ═ sqrt (N/k)cThe method is different not only with different pictures, but also with different clusters, and a fixed constant can be used for substitution.
Further, the processor 410 is further configured to:
inputting the features of the plurality of super-pixel blocks into a neighbor propagation clustering algorithm to obtain pixel block regions may include:
establishing a similarity matrix according to a plurality of super pixel blocks;
updating the attraction information of each superpixel block relative to other superpixel blocks in the similarity matrix under the condition of giving attribution;
under the condition of giving attraction, updating attribution degree information of each super-pixel block relative to other super-pixel blocks in the similarity matrix;
summing and iterating the attraction degree information and the attribution degree information of each superpixel block;
determining a target superpixel block serving as a clustering center and a sub superpixel block corresponding to the target superpixel block based on an iteration result;
and determining a region formed by the target super pixel block and the sub super pixel block as a pixel block region.
In the embodiment of the present invention, an AP clustering algorithm may be used to cluster a plurality of super pixel blocks into a large pixel block region, so that a labeling person labels each pixel block region. The purpose of the AP clustering algorithm is to find the cluster center of each superpixel block and determine the superpixel blocks of the same cluster center as the points of the same pixel block region. Firstly, establishing a similarity matrix according to a plurality of superpixel blocks, and then determining the clustering center of the superpixel blocks based on the similarity matrix, namely the superpixel block with the largest attraction degree and attribution degree for the current pixel block. Assuming that two superpixel blocks i and k exist, the attribution a (i, k) is the attribution sent from the candidate clustering center k to the point i, and reflects the appropriateness of the point i to select the point k as the clustering center after considering the support of other point-to-point k as the clustering center. The attraction degree r (i, k) is an attraction degree transmitted from the candidate cluster center i to the point k, and reflects a degree to which the point k is suitable as a cluster center of the point i in consideration of potential cluster centers of other points. Updating the attraction information of each superpixel block relative to other superpixel blocks in the similarity matrix under the condition of giving attribution; and under the condition of giving the attraction degree, updating the attribution degree information of each superpixel block relative to other superpixel blocks in the similarity matrix, and summing and iterating the attraction degree information and the attribution degree information of each superpixel block. At any time, the attraction degree and the attribution degree can be added to obtain a clustering center. If the preset iteration number is reached, or the clustering center does not change any more, or the decision of the sample point in one sub-area does not change any more after several iterations, the iteration can be terminated. And determining a target superpixel block serving as a clustering center and a sub superpixel block corresponding to the target superpixel block based on the terminated iteration result, and determining a region formed by the target superpixel block and the sub superpixel block as a pixel block region. Therefore, the superpixel blocks with the same semantics can be clustered into pixel block regions through the AP clustering algorithm, and labeling personnel can conveniently label the superpixel blocks.
In the embodiment of the present invention, the method may further include:
before acquiring the marking information of the pixel block area, acquiring the modification information of the pixel block area.
Specifically, after a plurality of pixel block regions are obtained through the AP clustering algorithm, an image to be labeled can be displayed on the terminal. The marking personnel can correspondingly modify a plurality of pixel block areas presented by the picture, and then mark the pixel block areas according to the modified marking information, and the computer equipment can acquire the marking information of the pixel block areas through the marked picture transmitted by the terminal. Therefore, the work of manually selecting the pixel block area is reduced, the accuracy of the labeling information is improved, and the labeling efficiency and accuracy are improved.
Examples of processor 410 may include, but are not limited to, a general purpose processor, a special purpose processor, a conventional processor, a Digital Signal Processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of Integrated Circuit (IC), a state machine, and the like. The processor may perform signal encoding, data processing, power control, input/output processing.
Examples of memory 420 may include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that may be used to store information that may be accessed by a processor.
The present invention also provides a machine-readable storage medium having stored thereon instructions for causing a machine to perform the above-described method for image annotation.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, implements the method for image annotation described above.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
The preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, however, the present invention is not limited to the specific details of the above embodiments, and various simple modifications can be made to the technical solution of the present invention within the technical idea of the present invention, and these simple modifications are within the protective scope of the present invention.
It should be noted that the various features described in the above embodiments may be combined in any suitable manner without departing from the scope of the invention. The invention is not described in detail in order to avoid unnecessary repetition.
In addition, any combination of the various embodiments of the present invention is also possible, and the same should be considered as the disclosure of the present invention as long as it does not depart from the spirit of the present invention.
Claims (10)
1. A method for image annotation, the method comprising:
acquiring an image to be marked;
performing superpixel segmentation on the image to generate a plurality of superpixel blocks;
extracting features of the plurality of super pixel blocks respectively;
inputting the characteristics of the plurality of superpixel blocks into a neighbor propagation clustering algorithm to obtain a pixel block region;
and acquiring the marking information of the pixel block area.
2. The method of claim 1, wherein said superpixel segmenting said image to generate a plurality of superpixel blocks comprises:
initializing a superpixel block to obtain an initialized center point;
in the neighborhood of each initialization center point, calculating the gradient value of each pixel point, and taking the pixel point with the minimum gradient value as a new center point;
marking all pixel points belonging to the same central point neighborhood as the same superpixel label;
calculating the distance between each searched pixel point and all the central points;
updating the central point closest to each pixel point to be a corresponding clustering central point, and updating the super pixel label of each pixel point;
iterating each clustering central point and each super-pixel label until the clustering central point and the super-pixel label corresponding to each pixel point are not changed any more;
forming a superpixel block by each clustering central point and pixel points containing the same superpixel labels with the clustering central point to form a plurality of superpixel blocks;
optimizing connectivity of the plurality of superpixel blocks.
3. The method of claim 2, wherein initializing the superpixel block to obtain an initialization center point comprises:
and uniformly distributing the initialization center points in the image according to the set number of the super pixel blocks.
4. The method of claim 2, wherein calculating the distance between each searched pixel point and all the center points comprises:
respectively calculating the color distance and the space distance between each searched pixel point and all the central points;
determining distances of the pixel point from all the center points based on the color distances and the spatial distances.
5. The method of claim 1, wherein the characteristics of the plurality of superpixel blocks comprise at least one of:
color, luminance, shape, texture, variance, covariance, and gradient.
6. The method of claim 1, wherein inputting the features of the plurality of superpixel blocks to a neighbor propagation clustering algorithm to obtain pixel block regions comprises:
establishing a similarity matrix according to the plurality of super pixel blocks;
updating the attraction information of each superpixel block relative to other superpixel blocks in the similarity matrix under the condition of giving attribution;
under the condition of giving attraction, updating attribution degree information of each super-pixel block relative to other super-pixel blocks in the similarity matrix;
summing and iterating the attraction degree information and the attribution degree information of each superpixel block;
determining a target superpixel block serving as a clustering center and a sub superpixel block corresponding to the target superpixel block based on an iteration result;
and determining a region formed by the target super pixel block and the sub super pixel block as a pixel block region.
7. The method of claim 1, further comprising:
and acquiring modification information of the pixel block region before acquiring the marking information of the pixel block region.
8. An apparatus for image annotation, characterized in that it is configured to carry out the method for image annotation according to any one of claims 1 to 7.
9. A machine-readable storage medium having stored thereon instructions for causing a machine to perform the method for image annotation according to any one of claims 1 to 7.
10. A computer program product comprising a computer program, characterized in that the computer program realizes the method for image annotation according to any one of claims 1 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110857660.1A CN113838061A (en) | 2021-07-28 | 2021-07-28 | Method and device for image annotation and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110857660.1A CN113838061A (en) | 2021-07-28 | 2021-07-28 | Method and device for image annotation and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113838061A true CN113838061A (en) | 2021-12-24 |
Family
ID=78962931
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110857660.1A Pending CN113838061A (en) | 2021-07-28 | 2021-07-28 | Method and device for image annotation and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113838061A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114549703A (en) * | 2022-01-25 | 2022-05-27 | 华南理工大学 | Method, system, device and storage medium for generating fast animation image |
| CN114677499A (en) * | 2022-04-11 | 2022-06-28 | 成都信息工程大学 | Cloud microparticle image particle region positioning method |
| CN116452613A (en) * | 2023-06-14 | 2023-07-18 | 山东省国土空间生态修复中心(山东省地质灾害防治技术指导中心、山东省土地储备中心) | Crack contour extraction method in geological survey |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105118049A (en) * | 2015-07-22 | 2015-12-02 | 东南大学 | Image segmentation method based on super pixel clustering |
| CN107886512A (en) * | 2016-09-29 | 2018-04-06 | 法乐第(北京)网络科技有限公司 | A kind of method for determining training sample |
| CN109598726A (en) * | 2018-10-26 | 2019-04-09 | 哈尔滨理工大学 | A kind of adapting to image target area dividing method based on SLIC |
| CN110796181A (en) * | 2019-06-28 | 2020-02-14 | 北京建筑大学 | Cultural relic disease high-precision automatic extraction method based on texture |
| CN111292341A (en) * | 2020-02-03 | 2020-06-16 | 北京海天瑞声科技股份有限公司 | Image annotation method, image annotation device and computer storage medium |
| CN112102929A (en) * | 2020-09-11 | 2020-12-18 | 沈阳东软智能医疗科技研究院有限公司 | Medical image labeling method and device, storage medium and electronic equipment |
| WO2021082168A1 (en) * | 2019-11-01 | 2021-05-06 | 南京原觉信息科技有限公司 | Method for matching specific target object in scene image |
-
2021
- 2021-07-28 CN CN202110857660.1A patent/CN113838061A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105118049A (en) * | 2015-07-22 | 2015-12-02 | 东南大学 | Image segmentation method based on super pixel clustering |
| CN107886512A (en) * | 2016-09-29 | 2018-04-06 | 法乐第(北京)网络科技有限公司 | A kind of method for determining training sample |
| CN109598726A (en) * | 2018-10-26 | 2019-04-09 | 哈尔滨理工大学 | A kind of adapting to image target area dividing method based on SLIC |
| CN110796181A (en) * | 2019-06-28 | 2020-02-14 | 北京建筑大学 | Cultural relic disease high-precision automatic extraction method based on texture |
| WO2021082168A1 (en) * | 2019-11-01 | 2021-05-06 | 南京原觉信息科技有限公司 | Method for matching specific target object in scene image |
| CN111292341A (en) * | 2020-02-03 | 2020-06-16 | 北京海天瑞声科技股份有限公司 | Image annotation method, image annotation device and computer storage medium |
| CN112102929A (en) * | 2020-09-11 | 2020-12-18 | 沈阳东软智能医疗科技研究院有限公司 | Medical image labeling method and device, storage medium and electronic equipment |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114549703A (en) * | 2022-01-25 | 2022-05-27 | 华南理工大学 | Method, system, device and storage medium for generating fast animation image |
| CN114549703B (en) * | 2022-01-25 | 2024-04-12 | 华南理工大学 | Quick-action image generation method, system, device and storage medium |
| CN114677499A (en) * | 2022-04-11 | 2022-06-28 | 成都信息工程大学 | Cloud microparticle image particle region positioning method |
| CN116452613A (en) * | 2023-06-14 | 2023-07-18 | 山东省国土空间生态修复中心(山东省地质灾害防治技术指导中心、山东省土地储备中心) | Crack contour extraction method in geological survey |
| CN116452613B (en) * | 2023-06-14 | 2023-08-29 | 山东省国土空间生态修复中心(山东省地质灾害防治技术指导中心、山东省土地储备中心) | Crack contour extraction method in geological survey |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113838061A (en) | Method and device for image annotation and storage medium | |
| CN106204522B (en) | Joint depth estimation and semantic annotation of a single image | |
| CN110610166B (en) | Text region detection model training method and device, electronic equipment and storage medium | |
| US20210183097A1 (en) | Spare Part Identification Using a Locally Learned 3D Landmark Database | |
| US9025863B2 (en) | Depth camera system with machine learning for recognition of patches within a structured light pattern | |
| WO2020087564A1 (en) | Three-dimensional object reconstruction method, computer device and storage medium | |
| US20200410688A1 (en) | Image Segmentation Method, Image Segmentation Apparatus, Image Segmentation Device | |
| CN113704531A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN107784671A (en) | A kind of method and system positioned immediately for vision with building figure | |
| CN110648397A (en) | Scene map generation method and device, storage medium and electronic equipment | |
| CN109993753B (en) | Method and device for segmenting urban functional area in remote sensing image | |
| US20210099310A1 (en) | Image processing method, image matching method, device and storage medium | |
| CN112102929A (en) | Medical image labeling method and device, storage medium and electronic equipment | |
| CN113177592B (en) | Image segmentation method and device, computer equipment and storage medium | |
| US20210166058A1 (en) | Image generation method and computing device | |
| CN114782530A (en) | Three-dimensional semantic map construction method, device, equipment and medium under indoor scene | |
| Tao et al. | Indoor 3D semantic robot VSLAM based on mask regional convolutional neural network | |
| CN107533760B (en) | Image segmentation method and device | |
| CN110717405B (en) | Face feature point positioning method, device, medium and electronic equipment | |
| CN106056575B (en) | A kind of image matching method based on like physical property proposed algorithm | |
| CN112906517A (en) | Self-supervision power law distribution crowd counting method and device and electronic equipment | |
| CN117237681A (en) | Image processing method, device and related equipment | |
| JP2011258036A (en) | Three-dimensional shape search device, three-dimensional shape search method, and program | |
| CN113139540B (en) | Backboard detection method and equipment | |
| CN116401326A (en) | Road identification updating method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |