CN115690112A - Image segmentation method, device, equipment and storage medium - Google Patents
Image segmentation method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115690112A CN115690112A CN202211203759.0A CN202211203759A CN115690112A CN 115690112 A CN115690112 A CN 115690112A CN 202211203759 A CN202211203759 A CN 202211203759A CN 115690112 A CN115690112 A CN 115690112A
- Authority
- CN
- China
- Prior art keywords
- image
- segmented
- model
- image segmentation
- segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003709 image segmentation Methods 0.000 title claims abstract description 218
- 238000000034 method Methods 0.000 title claims abstract description 65
- 230000002452 interceptive effect Effects 0.000 claims abstract description 244
- 230000011218 segmentation Effects 0.000 claims abstract description 72
- 230000004044 response Effects 0.000 claims abstract description 16
- 238000013507 mapping Methods 0.000 claims abstract description 12
- 238000012549 training Methods 0.000 claims description 64
- 230000003993 interaction Effects 0.000 claims description 30
- 238000012545 processing Methods 0.000 claims description 27
- 238000005520 cutting process Methods 0.000 claims description 22
- 238000000605 extraction Methods 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 16
- 230000004927 fusion Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims 1
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Landscapes
- Image Processing (AREA)
Abstract
The invention relates to the field of artificial intelligence, and discloses an image segmentation method, an image segmentation device, image segmentation equipment and a storage medium, wherein the method comprises the following steps: in response to the current interactive operation aiming at the image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result; judging whether the output result of the model meets the segmentation standard or not; if not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard; and if so, outputting the model output result as an image segmentation result of the image to be segmented. According to the method, the images are divided circularly by referring to the image division of the last interactive operation each time, so that the dynamic expansion of a focusing area can be realized, and the division precision of targets with different scales is improved.
Description
Technical Field
The present invention relates to the field of artificial intelligence, and in particular, to an image segmentation method, apparatus, device, and storage medium.
Background
The image segmentation is an important link of image processing, and the traditional image segmentation methods comprise a threshold value method, a region growing method, a level set method and the like, which are simple to implement, but have low segmentation precision and are difficult to process complex segmentation tasks. In recent years, the segmentation accuracy of images is greatly improved by deep learning techniques including convolutional neural networks.
The convolutional neural network is a machine learning model and has wide application in the field of image processing. Network models such as FCN, U-Net and the like can be used for solving the image segmentation problem. However, in order to ensure the safety of automatic driving, the existing fully-automatic segmentation method based on the convolutional neural network still cannot achieve high enough precision, and needs to manually and further repair segmentation errors.
In recent years, a deep learning method is used to realize interactive image segmentation and editing with little work, improve image segmentation precision through user interaction, and reduce the workload required by the user interaction. However, in this interactive mode, the image segmentation is performed according to the current interactive mode each time, and the previous image segmentation result is not referred to, so that the accuracy of the interactive image segmentation is still not high enough.
Disclosure of Invention
The invention mainly aims to solve the technical problem of low accuracy of the conventional interactive image segmentation.
The invention provides an image segmentation method in a first aspect, which comprises the following steps:
responding to the current interactive operation aiming at the image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result;
judging whether the output result of the model meets the segmentation standard or not;
if not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard;
and if so, outputting the model output result as an image segmentation result of the image to be segmented.
Optionally, in a first implementation manner of the first aspect of the present invention, the inputting, in response to a current interactive operation on an image to be segmented, the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result includes:
responding to a current interactive operation aiming at an image to be segmented, and determining whether the current interactive operation is the first interaction of the image to be segmented;
if so, adjusting the image to be segmented to a preset size, and inputting the image to be segmented after size adjustment into the interactive image segmentation model to obtain a first model output result;
and if not, generating a focusing region of the image to be segmented according to a model output result corresponding to the last interactive operation, and inputting the focusing region into the interactive image segmentation model to obtain a second model output result.
Optionally, in a second implementation manner of the first aspect of the present invention, the interactive image segmentation model includes a fusion network, a feature extraction network, and a segmentation network; inputting the image to be segmented after the size adjustment into the interactive image segmentation model, and obtaining a first model output result comprises:
identifying foreground points and background points in the image to be segmented according to the current interactive operation through a fusion network in the interactive image segmentation model, and fusing the foreground points and the background points with the image to be segmented;
performing feature extraction on the fused image to be segmented through a feature extraction network to obtain corresponding image features;
and carrying out image segmentation on the image to be segmented according to the image characteristics through the segmentation network to obtain a first model output result.
Optionally, in a third implementation manner of the first aspect of the present invention, the generating a focus area of the image to be segmented according to a model output result corresponding to a previous interactive operation includes:
generating a corresponding maximum external frame in the image to be segmented according to a model output result corresponding to the last operation;
judging whether the position of the current interactive operation in the image to be segmented is within the maximum external frame;
if not, expanding the maximum external frame based on the position of the current interactive operation in the image to be segmented;
and generating a focusing area of the image to be segmented according to the maximum external frame.
Optionally, in a fourth implementation manner of the first aspect of the present invention, after the expanding the maximum bounding box based on the position of the current interactive operation in the image to be segmented, the method further includes:
judging whether the maximum external frame is smaller than the minimum cutting size or not;
and if so, adjusting the focus area according to the minimum cutting size.
Optionally, in a fifth implementation manner of the first aspect of the present invention, after the generating a focus area of the image to be segmented according to a model output result corresponding to a last interactive operation, the method further includes:
expanding the focusing area according to a preset expansion ratio;
and cutting the expanded focusing area to the input size according with the interactive image segmentation model according to a preset cutting algorithm.
Optionally, in a sixth implementation manner of the first aspect of the present invention, before the responding to the current interactive operation on the image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result, the method includes:
acquiring an initialized interactive image segmentation model and a training set containing training images, wherein the training images are images containing true foreground points and true background points;
responding to the interactive operation aiming at the training images, inputting the training images in the training set and the positions of the corresponding interactive operation in the training images into the initialized interactive image segmentation model, and processing the training images through the initialized interactive image segmentation model to obtain output processing images, wherein the output processing images comprise predicted foreground points and predicted background points;
calculating a preset loss function according to the true value foreground point and the true value background point of the training image and the predicted foreground point and the predicted background point of the output processing image to obtain a loss function value;
judging whether the loss value is larger than a preset loss threshold value or not;
if so, performing back propagation on the initialized interactive image segmentation model according to the loss function, adjusting network parameters of the initialized interactive image segmentation model, and inputting the training images in the training set into the initialized interactive image segmentation model again;
if not, the network training is finished, and the interactive image segmentation model is obtained.
A second aspect of the present invention provides an image segmentation apparatus comprising:
the image segmentation device comprises an input module, a segmentation module and a segmentation module, wherein the input module is used for responding to the current interactive operation aiming at an image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result;
the judging module is used for judging whether the output result of the model meets the segmentation standard or not;
the iteration module is used for mapping the model output result to the image to be segmented when the model output result does not meet the segmentation standard, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard;
and the output module is used for outputting the model output result as the image segmentation result of the image to be segmented when the model output result meets the segmentation standard.
Optionally, in a first implementation manner of the second aspect of the present invention, the input module specifically includes:
the response unit is used for responding to the current interactive operation aiming at the image to be segmented, and determining whether the current interactive operation is the first interaction of the image to be segmented;
the first input unit is used for adjusting the image to be segmented to a preset size when the current interactive operation is the first interaction of the image to be segmented, and inputting the image to be segmented after size adjustment into the interactive image segmentation model to obtain a first model output result;
and the second input unit is used for generating a focusing area of the image to be segmented according to a model output result corresponding to the last interactive operation when the current interactive operation is not the first interactive operation of the image to be segmented, and inputting the focusing area into the interactive image segmentation model to obtain a second model output result.
Optionally, in a second implementation manner of the second aspect of the present invention, the interactive image segmentation model includes a fusion network, a feature extraction network, and a segmentation network; the first input unit is used for:
identifying foreground points and background points in the image to be segmented according to the current interactive operation through a fusion network in the interactive image segmentation model, and fusing the foreground points and the background points with the image to be segmented;
performing feature extraction on the fused image to be segmented through a feature extraction network to obtain corresponding image features;
and carrying out image segmentation on the image to be segmented according to the image characteristics through the segmentation network to obtain a first model output result.
Optionally, in a third implementation manner of the second aspect of the present invention, the second input module is specifically configured to: generating a corresponding maximum external frame in the image to be segmented according to the model output result corresponding to the previous operation;
judging whether the position of the current interactive operation in the image to be segmented is within the maximum external frame;
if not, expanding the maximum external frame based on the position of the current interactive operation in the image to be segmented;
and generating a focusing area of the image to be segmented according to the maximum external frame.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the image segmentation apparatus further includes an outer frame adjustment module, where the outer frame adjustment module is specifically configured to:
judging whether the maximum external frame is smaller than the minimum cutting size or not;
and if so, adjusting the focus area according to the minimum cutting size.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the image segmentation apparatus further includes a focus adjustment module, where the focus adjustment module is specifically configured to:
expanding the focusing area according to a preset expansion ratio;
and cutting the expanded focusing area to the input size according with the interactive image segmentation model according to a preset cutting algorithm.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the image segmentation apparatus further includes a model training module, where the model training module is specifically configured to:
acquiring an initialized interactive image segmentation model and a training set containing training images, wherein the training images are images containing true foreground points and true background points;
responding to the interactive operation aiming at the training images, inputting the training images in the training set and the positions of the corresponding interactive operation in the training images into the initialized interactive image segmentation model, and processing the training images through the initialized interactive image segmentation model to obtain output processing images, wherein the output processing images comprise predicted foreground points and predicted background points;
calculating a preset loss function according to the true foreground point and the true background point of the training image and the predicted foreground point and the predicted background point of the output processing image to obtain a loss function value;
judging whether the loss value is larger than a preset loss threshold value or not;
if so, performing back propagation on the initialized interactive image segmentation model according to the loss function, adjusting network parameters of the initialized interactive image segmentation model, and inputting the training images in the training set into the initialized interactive image segmentation model again;
if not, the network training is finished, and the interactive image segmentation model is obtained.
A third aspect of the present invention provides an image segmentation apparatus comprising: a memory having instructions stored therein and at least one processor, the memory and the at least one processor interconnected by a line; the at least one processor invokes the instructions in the memory to cause the image segmentation apparatus to perform the steps of the image segmentation method described above.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to perform the steps of the image segmentation method described above.
According to the technical scheme, the image to be segmented is input into a preset interactive image segmentation model in response to the current interactive operation aiming at the image to be segmented, and the image to be segmented is segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result; judging whether the output result of the model meets the segmentation standard or not; if not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard; and if so, outputting the model output result as an image segmentation result of the image to be segmented. According to the method, the images are circularly segmented by referring to the image segmentation of the last interactive operation each time, so that the dynamic expansion of a focusing area can be realized, and the influence of the manually set minimum bounding box size on the effect of the whole method is eliminated, and the segmentation precision of targets with different scales is improved.
Drawings
FIG. 1 is a schematic diagram of a first embodiment of an image segmentation method according to an embodiment of the present invention;
FIG. 2 is a diagram of a second embodiment of an image segmentation method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an embodiment of an image segmentation apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another embodiment of an image segmentation apparatus according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an embodiment of an image segmentation apparatus in an embodiment of the present invention.
Detailed Description
The embodiment of the application provides an image segmentation method, an image segmentation device, image segmentation equipment and a storage medium, which are used for solving the technical problem of low accuracy of the conventional interactive image segmentation.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Moreover, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For ease of understanding, a specific flow of an embodiment of the present invention is described below, and referring to fig. 1, a first embodiment of an image segmentation method according to an embodiment of the present invention includes:
101. in response to the current interactive operation aiming at the image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result;
it is to be understood that the execution subject of the present invention may be an image segmentation apparatus, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
In this embodiment, the interaction mode includes three forms, namely frame selection, smearing and clicking. The frame selection can provide the most abundant information, a user uses a rectangular frame to designate an area where a target object exists, the user can also designate a specific area as a reliable foreground by using a smearing mode, the accuracy of the information is guaranteed while a large number of foreground areas are provided, but the size of the smearing area influences the segmentation effect to a great extent, complicated interaction is needed, and an ideal effect is difficult to achieve under the conditions of more object blocks or rich textures; the click mode provides less accurate information, is a convenient interactive mode and is generally used for the specification of the segmentation boundary or the further optimization of the segmentation result. The three interaction modes have respective advantages and disadvantages, and the interaction with the image to be segmented can be realized by using one or more combinations of frame selection, smearing and clicking.
In this embodiment, the interactive image segments the model to use arbitrary models, such as: any one of f-BRS, RITM and EdgeFlow, the invention is not limited, and any one of HRNet, OCRNet, segFormer and HRViT can be selected for the feature extraction network in the interactive image segmentation model, and the invention is not limited.
102. Judging whether the output result of the model meets the segmentation standard or not;
in this embodiment, it is mainly determined by the user whether the model output is satisfactory, if not, it is determined that the model output result meets the segmentation standard, and if not, it is determined that the model output result meets the non-segmentation standard, the user may perform a corresponding click operation on the system page, and the system determines whether the user is satisfactory to the model output result in response to the click operation. In addition, in this embodiment, it may also be determined in advance whether the type of the target object in the input image to be segmented determines that the model output result meets the segmentation standard, for example, it is determined in advance that the target object in the input image to be segmented is an automobile, when the system obtains the model output result, each component serving as an automobile in the model output result is identified, and the size ratio of each component and the entire automobile are calculated to determine whether the component is within a reasonable range, if not, it is determined that the model output result is not reasonable, and therefore, the segmentation standard is not met, and if within the reasonable range, it is determined that the model output result is reasonable, and therefore, the segmentation standard is met.
In this embodiment, the model output result is a result of image segmentation performed on a target object in an image to be segmented, and after the image is segmented, a mask map is generated in the image to be segmented according to pixels of the segmented image, where the mask map is a labeled map generated after foreground and background segmentation is performed on part or all of pixels of one image, each pixel on the mask map is labeled as a foreground point or a background point, and a user can determine whether the model output result meets a segmentation standard by looking at the mask map.
103. If not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard;
104. and if so, outputting the model output result as an image segmentation result of the image to be segmented.
In this embodiment, after obtaining the model output result, mapping the model output result to the image to be segmented, specifically, determining that each pixel in the image to be segmented is marked as a foreground point or a background point according to the mask map, and screening out image parts of the foreground points for processing, for example, generating a maximum external frame of the foreground points, and displaying the maximum external frame in the system image, where the user can view the maximum external frame and perform a second interactive operation with reference to the maximum external frame, for example, the maximum external frame generated by the model processing result does not include a complete target object, and can click on a target object part outside the maximum external frame, so that the maximum external frame expands again to include the complete target object, and the final maximum external frame is intercepted and then input into the interactive image segmentation model, and the cycle is repeated until the output result meets the segmentation standard, and the model output result is output as the image segmentation result of the image to be segmented.
In this embodiment, in response to a current interactive operation on an image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result; judging whether the output result of the model meets the segmentation standard or not; if not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard; and if so, outputting the model output result as an image segmentation result of the image to be segmented. According to the method, the images are circularly segmented by referring to the image segmentation of the last interactive operation each time, so that the dynamic expansion of a focusing area can be realized, and the influence of the manually set minimum bounding box size on the effect of the whole method is eliminated, and the segmentation precision of targets with different scales is improved.
Referring to fig. 2, a second embodiment of the image segmentation method according to the embodiment of the present invention includes:
201. responding to the current interactive operation aiming at the image to be segmented, and determining whether the current interactive operation is the first interaction of the image to be segmented;
202. if so, adjusting the image to be segmented to a preset size, and inputting the image to be segmented after size adjustment into the interactive image segmentation model to obtain a first model output result;
in this embodiment, since the first interaction is performed and the output result of the model corresponding to the last click operation is not referred to, the first model processing can only generate the maximum circumscribed frame or perform size adjustment on the image to be segmented according to the position where the first interaction operation is performed in the image to be segmented, so that the image to be segmented can conform to the input condition of the interactive image segmentation model. Specifically, if the maximum circumscribing frame is selected to be generated according to the position where the first interaction operation is performed in the image to be segmented, the position where the first interaction operation is performed in the image to be segmented is used as a central point to generate the circumscribing frame with a preset size, if the border of the circumscribing frame exceeds the edge of the image, the circumscribing frame is truncated according to the image boundary, but the maximum circumscribing frame is selected to be generated according to the position where the first interaction operation is performed in the image to be segmented by setting the minimum bounding frame size S min Prevent the region from being too small every time of cropping, however, when S min When the setting is larger, the detail processing on small targets or local edges is not good enough; when S is min When the setting is small, for a large target, the whole information of the target cannot be sensed in the first few times of interaction, so that the number of interaction required by a user is increased, and the size of the image to be segmented can be adjusted optionally, namely, the size of the image to be segmented is adjustedDirectly setting the focus area as the original image by first interaction, and then removing S min This parameter, too, can be understood as meaning S min Is set to 0.
In practical applications, the interactive image segments the model to use arbitrary models, such as: any one of f-BRS, RITM, edgeFlow, in this embodiment, taking the interactive image segmentation model as f-BRS as an example, the f-BRS may perform not only backward transfer from the input but also backward transfer optimization network from the corresponding feature extraction module node in the process of propagation under the click of obtaining the interactive foreground and background, which may achieve a large speed increase on reasoning, and when the interactive image segmentation model is f-BRS, the interactive image segmentation model includes a fusion network, a feature extraction network, and a segmentation network; inputting the image to be segmented after the size adjustment into the interactive image segmentation model, and obtaining a first model output result comprises: identifying foreground points and background points in the image to be segmented according to the current interactive operation through a fusion network in the interactive image segmentation model, and fusing the foreground points and the background points with the image to be segmented; performing feature extraction on the fused image to be segmented through a feature extraction network to obtain corresponding image features; and carrying out image segmentation on the image to be segmented according to the image characteristics through the segmentation network to obtain a first model output result.
In this embodiment, before the responding to the current interactive operation on the image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result, the method includes: acquiring an initialized interactive image segmentation model and a training set containing training images, wherein the training images are images containing true foreground points and true background points; responding to the interactive operation aiming at the training images, inputting the training images in the training set and the positions of the corresponding interactive operation in the training images into the initialized interactive image segmentation model, and processing the training images through the initialized interactive image segmentation model to obtain output processing images, wherein the output processing images comprise predicted foreground points and predicted background points; calculating a preset loss function according to the true foreground point and the true background point of the training image and the predicted foreground point and the predicted background point of the output processing image to obtain a loss function value; judging whether the loss value is larger than a preset loss threshold value or not; if so, performing back propagation on the initialized interactive image segmentation model according to the loss function, adjusting network parameters of the initialized interactive image segmentation model, and inputting the training images in the training set into the initialized interactive image segmentation model again; if not, the network training is finished, and an interactive image segmentation model is obtained.
203. If not, generating a focus area of the image to be segmented according to a model output result corresponding to the last interactive operation, and inputting the focus area into the interactive image segmentation model to obtain a second model output result;
in this embodiment, the generating the focus area of the image to be segmented according to the output result of the model corresponding to the previous interactive operation includes: generating a corresponding maximum external frame in the image to be segmented according to a model output result corresponding to the last operation; judging whether the position of the current interactive operation in the image to be segmented is within the maximum external frame; if not, expanding the maximum external frame based on the position of the current interactive operation in the image to be segmented; and generating a focusing area of the image to be segmented according to the maximum external frame.
Specifically, after each interactive operation, the interactive image segmentation model outputs a corresponding model output result, when the model output result does not meet the standard, the interaction is performed again, the maximum external frame of the target segmentation is calculated according to the segmentation result obtained by the previous interaction in the next interactive operation, and if the user clicks outside the boundary frame in the current interaction, the boundary frame is expanded until the user clicks inside the boundary frame in the current interaction.
In this embodiment, after the expanding the maximum bounding box based on the position of the current interactive operation in the image to be segmented, the method further includes: judging whether the maximum external frame is smaller than the minimum cutting size or not; and if so, adjusting the focus area according to the minimum cutting size.
Specifically, if the size of the outer frame is smaller than the minimum size S set by human min Then adjust the size of the bounding box to S min 。
In this embodiment, after the generating the focus area of the image to be segmented according to the model output result corresponding to the previous interactive operation, the method further includes: expanding the focusing area according to a preset expansion ratio; and cutting the expanded focusing area to the input size according with the interactive image segmentation model according to a preset cutting algorithm.
Specifically, in order to ensure that the context and the boundary information are not lost, the boundary frame is extended by a certain proportion along the edge to obtain a focus area, the area is cut and adjusted to a certain size, and then the focus area is input into the interactive segmentation model, wherein in the cutting process, the values of the area are directly taken out from the original image according to the position of the area to be cut. Assume that the original image matrix size is C × H × W (C is the channel dimension, H is the image height, and W is the image width), and the extracted region size is C × H '× W' (H 'is the region height, and W' is the region width).
In particular, the focus area needs to be expanded and cropped, respectively, because if the bounding box is expanded directly to fit the input size of the interactive segmentation model, it is likely that the small objects will only occupy a small portion within this box. On the contrary, expansion is performed first (here the expansion ratio is artificially set to r, and the expanded size is S min * r) cutting and taking out the expanded region (focus region) and adjusting the cut region to the size of the model input, wherein the focus region can better contain the concerned target, for example, the small target may occupy a large part in the region at the moment, which is equivalent to artificial magnification resolution and is beneficial to improving the segmentation effect.
204. Judging whether the output result of the first model or the output result of the second model meets the segmentation standard or not;
205. if not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard;
206. and if so, outputting the model output result as an image segmentation result of the image to be segmented.
On the basis of the previous embodiment, the present embodiment describes in detail a process of inputting an image to be segmented into a preset interactive image segmentation model in response to a current interactive operation on the image to be segmented, performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result, and determining whether the current interactive operation is the first interaction of the image to be segmented in response to the current interactive operation on the image to be segmented; if so, adjusting the image to be segmented to a preset size, and inputting the image to be segmented after size adjustment into the interactive image segmentation model to obtain a first model output result; if not, generating a focus area of the image to be segmented according to a model output result corresponding to the last interactive operation, and inputting the focus area into the interactive image segmentation model to obtain a second model output result. According to the method, the images are circularly segmented by referring to the image segmentation of the last interactive operation each time, so that the dynamic expansion of a focusing area can be realized, and the influence of the manually set minimum bounding box size on the effect of the whole method is eliminated, and the segmentation precision of targets with different scales is improved.
With reference to fig. 3, the image segmentation method in the embodiment of the present invention is described above, and an image segmentation apparatus in the embodiment of the present invention is described below, where an embodiment of the image segmentation apparatus in the embodiment of the present invention includes:
the input module 301 is configured to input, in response to a current interactive operation for an image to be segmented, the image to be segmented into a preset interactive image segmentation model, perform image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model, and obtain a model output result;
a judging module 302, configured to judge whether the model output result meets a segmentation standard;
an iteration module 303, configured to map the model output result to the image to be segmented when the model output result does not meet a segmentation standard, and return to the step of responding to the current interactive operation on the image to be segmented until a model output result corresponding to the current interactive operation meets the segmentation standard;
an output module 304, configured to output the model output result as an image segmentation result of the image to be segmented when the model output result meets the segmentation criteria.
In the embodiment of the invention, the image segmentation device runs the image segmentation method, inputs the image to be segmented into a preset interactive image segmentation model by responding to the current interactive operation aiming at the image to be segmented, and performs image segmentation on the image to be segmented according to the current interactive operation by the interactive image segmentation model to obtain a model output result; judging whether the output result of the model meets the segmentation standard or not; if not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard; and if so, outputting the model output result as the image segmentation result of the image to be segmented. According to the method, the image is divided circularly by referring to the image division of the last interactive operation each time, so that the dynamic expansion of a focusing area can be realized, and the influence of the manually set minimum bounding box size on the effect of the overall method is eliminated, and the division precision of the targets with different scales is improved.
Referring to fig. 4, a second embodiment of the image segmentation apparatus according to the embodiment of the present invention includes:
the input module 301 is configured to input, in response to a current interactive operation for an image to be segmented, the image to be segmented into a preset interactive image segmentation model, perform image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model, and obtain a model output result;
a judging module 302, configured to judge whether the model output result meets a segmentation standard;
an iteration module 303, configured to map the model output result to the image to be segmented when the model output result does not meet a segmentation standard, and return to the step of responding to the current interactive operation on the image to be segmented until a model output result corresponding to the current interactive operation meets the segmentation standard;
an output module 304, configured to output the model output result as an image segmentation result of the image to be segmented when the model output result meets the segmentation criteria.
Optionally, in a first implementation manner of the second aspect of the present invention, the input module 301 specifically includes:
a response unit 3011, configured to determine, in response to a current interaction operation for an image to be segmented, whether the current interaction operation is a first interaction of the image to be segmented;
the first input unit 3012, configured to, when the current interaction operation is a first interaction of the image to be segmented, adjust the image to be segmented to a preset size, and input the image to be segmented after size adjustment to the interactive image segmentation model, so as to obtain a first model output result;
a second input unit 3013, configured to generate a focus area of the image to be segmented according to a model output result corresponding to a last interactive operation when the current interactive operation is not the first interaction of the image to be segmented, and input the focus area into the interactive image segmentation model to obtain a second model output result.
Optionally, in a second implementation manner of the second aspect of the present invention, the interactive image segmentation model includes a fusion network, a feature extraction network, and a segmentation network; the first input unit 3012 is configured to:
identifying foreground points and background points in the image to be segmented according to the current interactive operation through a fusion network in the interactive image segmentation model, and fusing the foreground points and the background points with the image to be segmented;
performing feature extraction on the fused image to be segmented through a feature extraction network to obtain corresponding image features;
and carrying out image segmentation on the image to be segmented according to the image characteristics through the segmentation network to obtain a first model output result.
Optionally, in a third implementation manner of the second aspect of the present invention, the second input module 3013 is specifically configured to: generating a corresponding maximum external frame in the image to be segmented according to the model output result corresponding to the previous operation;
judging whether the position of the current interactive operation in the image to be segmented is within the maximum external frame or not;
if not, expanding the maximum external frame based on the position of the current interactive operation in the image to be segmented;
and generating a focusing area of the image to be segmented according to the maximum external frame.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the image segmentation apparatus further includes an outer frame adjustment module 305, where the outer frame adjustment module 305 is specifically configured to:
judging whether the maximum external frame is smaller than the minimum cutting size or not;
and if so, adjusting the focus area according to the minimum cutting size.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the image segmentation apparatus further includes a focus adjustment module 306, where the focus adjustment module 306 is specifically configured to:
expanding the focusing area according to a preset expansion ratio;
and cutting the expanded focusing area to the input size according with the interactive image segmentation model according to a preset cutting algorithm.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the image segmentation apparatus further includes a model training module 307, where the model training module 307 is specifically configured to:
acquiring an initialized interactive image segmentation model and a training set containing training images, wherein the training images are images containing true foreground points and true background points;
responding to the interactive operation aiming at the training images, inputting the training images in the training set and the positions of the corresponding interactive operation in the training images into the initialized interactive image segmentation model, and processing the training images through the initialized interactive image segmentation model to obtain output processing images, wherein the output processing images comprise predicted foreground points and predicted background points;
calculating a preset loss function according to the true foreground point and the true background point of the training image and the predicted foreground point and the predicted background point of the output processing image to obtain a loss function value;
judging whether the loss value is larger than a preset loss threshold value or not;
if so, performing back propagation on the initialized interactive image segmentation model according to the loss function, adjusting network parameters of the initialized interactive image segmentation model, and inputting the training images in the training set into the initialized interactive image segmentation model again;
if not, the network training is finished, and an interactive image segmentation model is obtained.
In this implementation, the specific functions of each module and the unit configuration of a part of modules of the image segmentation apparatus are described in detail, through each module and each unit of the apparatus, in response to the current interactive operation on the image to be segmented, the image to be segmented is input into a preset interactive image segmentation model, and the image to be segmented is segmented according to the current interactive operation through the interactive image segmentation model, so as to obtain a model output result; judging whether the output result of the model meets the segmentation standard or not; if not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard; and if so, outputting the model output result as an image segmentation result of the image to be segmented. According to the method, the image is divided circularly by referring to the image division of the last interactive operation each time, so that the dynamic expansion of a focusing area can be realized, and the influence of the manually set minimum bounding box size on the effect of the overall method is eliminated, and the division precision of the targets with different scales is improved.
Fig. 3 and 4 describe the image segmentation apparatus in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the image segmentation apparatus in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 5 is a schematic structural diagram of an image segmentation apparatus 500 according to an embodiment of the present invention, where the image segmentation apparatus 500 may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored on the storage medium 530 may include one or more modules (not shown), each of which may include a sequence of instructions operating on the image segmentation apparatus 500. Still further, the processor 510 may be configured to communicate with the storage medium 530, and execute a series of instruction operations in the storage medium 530 on the image segmentation apparatus 500 to implement the steps of the image segmentation method described above.
The present invention also provides a computer readable storage medium, which may be a non-volatile computer readable storage medium, and which may also be a volatile computer readable storage medium, having stored therein instructions, which, when run on a computer, cause the computer to perform the steps of the image segmentation method.
It can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working process of the system, the apparatus, and the unit described above may refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. An image segmentation method, characterized in that the image segmentation method comprises:
responding to the current interactive operation aiming at the image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented through the interactive image segmentation model according to the current interactive operation to obtain a model output result;
judging whether the output result of the model meets the segmentation standard or not;
if not, mapping the model output result to the image to be segmented, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard;
and if so, outputting the model output result as an image segmentation result of the image to be segmented.
2. The image segmentation method according to claim 1, wherein the inputting the image to be segmented into a preset interactive image segmentation model in response to a current interactive operation on the image to be segmented, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result comprises:
responding to a current interactive operation aiming at an image to be segmented, and determining whether the current interactive operation is the first interaction of the image to be segmented;
if so, adjusting the image to be segmented to a preset size, and inputting the image to be segmented after size adjustment into the interactive image segmentation model to obtain a first model output result;
and if not, generating a focusing region of the image to be segmented according to a model output result corresponding to the last interactive operation, and inputting the focusing region into the interactive image segmentation model to obtain a second model output result.
3. The image segmentation method according to claim 1, wherein the interactive image segmentation model includes a fusion network, a feature extraction network, and a segmentation network; the step of inputting the image to be segmented after the size adjustment into the interactive image segmentation model to obtain a first model output result comprises the following steps:
identifying foreground points and background points in the image to be segmented according to the current interactive operation through a fusion network in the interactive image segmentation model, and fusing the foreground points and the background points with the image to be segmented;
performing feature extraction on the fused image to be segmented through a feature extraction network to obtain corresponding image features;
and carrying out image segmentation on the image to be segmented according to the image characteristics through the segmentation network to obtain a first model output result.
4. The image segmentation method according to claim 2, wherein the generating the focus area of the image to be segmented according to the model output result corresponding to the previous interactive operation comprises:
generating a corresponding maximum external frame in the image to be segmented according to a model output result corresponding to the last operation;
judging whether the position of the current interactive operation in the image to be segmented is within the maximum external frame;
if not, expanding the maximum external frame based on the position of the current interactive operation in the image to be segmented;
and generating a focusing area of the image to be segmented according to the maximum external frame.
5. The image segmentation method according to claim 4, wherein after the expanding the maximum bounding box based on the position of the current interactive operation in the image to be segmented, the method further comprises:
judging whether the maximum external frame is smaller than the minimum cutting size;
and if so, adjusting the focus area according to the minimum cutting size.
6. The image segmentation method according to claim 2, wherein after the generating the focus area of the image to be segmented according to the output result of the model corresponding to the previous interactive operation, the method further comprises:
expanding the focusing area according to a preset expansion ratio;
and cutting the expanded focusing area to the input size according with the interactive image segmentation model according to a preset cutting algorithm.
7. The image segmentation method according to any one of claims 1 to 6, before the step of inputting the image to be segmented into a preset interactive image segmentation model in response to the current interactive operation on the image to be segmented, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result, the method comprises:
acquiring an initialized interactive image segmentation model and a training set containing training images, wherein the training images are images containing true foreground points and true background points;
responding to the interactive operation aiming at the training images, inputting the training images in the training set and the positions of the corresponding interactive operation in the training images into the initialized interactive image segmentation model, and processing the training images through the initialized interactive image segmentation model to obtain output processing images, wherein the output processing images comprise predicted foreground points and predicted background points;
calculating a preset loss function according to the true value foreground point and the true value background point of the training image and the predicted foreground point and the predicted background point of the output processing image to obtain a loss function value;
judging whether the loss value is larger than a preset loss threshold value or not;
if so, performing back propagation on the initialized interactive image segmentation model according to the loss function, adjusting network parameters of the initialized interactive image segmentation model, and inputting the training images in the training set into the initialized interactive image segmentation model again;
if not, the network training is finished, and an interactive image segmentation model is obtained.
8. An image segmentation apparatus, characterized in that the image segmentation apparatus comprises:
the image segmentation device comprises an input module, a segmentation module and a segmentation module, wherein the input module is used for responding to the current interactive operation aiming at an image to be segmented, inputting the image to be segmented into a preset interactive image segmentation model, and performing image segmentation on the image to be segmented according to the current interactive operation through the interactive image segmentation model to obtain a model output result;
the judging module is used for judging whether the output result of the model meets the segmentation standard or not;
the iteration module is used for mapping the model output result to the image to be segmented when the model output result does not meet the segmentation standard, and returning to the step of responding to the current interactive operation aiming at the image to be segmented until the model output result corresponding to the current interactive operation meets the segmentation standard;
and the output module is used for outputting the model output result as an image segmentation result of the image to be segmented when the model output result meets the segmentation standard.
9. An image segmentation apparatus characterized by comprising: a memory having instructions stored therein and at least one processor, the memory and the at least one processor interconnected by a line;
the at least one processor invokes the instructions in the memory to cause the image segmentation apparatus to perform the steps of the image segmentation method of any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the image segmentation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211203759.0A CN115690112A (en) | 2022-09-29 | 2022-09-29 | Image segmentation method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211203759.0A CN115690112A (en) | 2022-09-29 | 2022-09-29 | Image segmentation method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115690112A true CN115690112A (en) | 2023-02-03 |
Family
ID=85065436
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211203759.0A Pending CN115690112A (en) | 2022-09-29 | 2022-09-29 | Image segmentation method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115690112A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116342629A (en) * | 2023-06-01 | 2023-06-27 | 深圳思谋信息科技有限公司 | Image interaction segmentation method, device, equipment and storage medium |
-
2022
- 2022-09-29 CN CN202211203759.0A patent/CN115690112A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116342629A (en) * | 2023-06-01 | 2023-06-27 | 深圳思谋信息科技有限公司 | Image interaction segmentation method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111640125B (en) | Aerial photography graph building detection and segmentation method and device based on Mask R-CNN | |
| CN110852949B (en) | Point cloud data completion method and device, computer equipment and storage medium | |
| CN111583097A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN112508975A (en) | Image identification method, device, equipment and storage medium | |
| CN104134234A (en) | Full-automatic three-dimensional scene construction method based on single image | |
| CN112330815B (en) | Three-dimensional point cloud data processing method, device and equipment based on obstacle fusion | |
| EP4239583A1 (en) | Three-dimensional reconstruction method, apparatus and system, and storage medium | |
| CN116052094B (en) | Ship detection method, system and computer storage medium | |
| CN115690112A (en) | Image segmentation method, device, equipment and storage medium | |
| CN115810133B (en) | Welding control method based on image processing and point cloud processing and related equipment | |
| US20230237777A1 (en) | Information processing apparatus, learning apparatus, image recognition apparatus, information processing method, learning method, image recognition method, and non-transitory-computer-readable storage medium | |
| CN111744199A (en) | Image processing method and device, computer readable storage medium and electronic device | |
| CN114689036A (en) | Map updating method, automatic driving method, electronic device and storage medium | |
| CN111659123A (en) | Navigation data processing method and device, storage medium and electronic equipment | |
| CN112819937B (en) | Self-adaptive multi-object light field three-dimensional reconstruction method, device and equipment | |
| CN110717141A (en) | Lane line optimization method and device and storage medium | |
| CN113724383A (en) | Robot topology map generation system, method, computer device and storage medium | |
| CN116936116A (en) | Intelligent medical data analysis method and system | |
| CN115830046A (en) | Interactive image segmentation method, device, equipment and storage medium | |
| CN111739062B (en) | Target detection method and system based on feedback mechanism | |
| CN113359693A (en) | Robot working method, device, equipment and storage medium | |
| CN111735463B (en) | Picture construction method and device | |
| KR102661488B1 (en) | Servers, systems, methods and programs that provide special effect creation and 3d model creation services using generative ai models | |
| CN112435293B (en) | Method and device for determining structural parameter representation of lane line | |
| CN116612269B (en) | Interactive segmentation labeling method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |