CN113409224A - Image target pertinence enhancing method, device, equipment and storage medium - Google Patents
Image target pertinence enhancing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113409224A CN113409224A CN202110779988.6A CN202110779988A CN113409224A CN 113409224 A CN113409224 A CN 113409224A CN 202110779988 A CN202110779988 A CN 202110779988A CN 113409224 A CN113409224 A CN 113409224A
- Authority
- CN
- China
- Prior art keywords
- target
- image
- model
- module
- mask
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The application discloses a method, a device, equipment and a storage medium for enhancing image target pertinence, which comprise the following steps: constructing and training an automatic cutout model; the model comprises a segmentation module for generating a target Mask, a Trimap generation module for generating Trimap, and a matting module for performing boundary correction on the target Mask generated by the segmentation module according to the Trimap; extracting a target Mask in the original image by using an automatic cutout model; enhancing the image information characteristics of the extracted target Mask; and assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image. Therefore, the automatic cutout model is used for extracting rough target masks, correcting the rough target masks by cutout, and then pertinently enhancing the target to be observed in the image, and reserving the original background information, so that the visual effect of the target in the whole image can be improved under the condition that the whole image is not distorted.
Description
Technical Field
The present invention relates to the field of image enhancement technologies, and in particular, to a method, an apparatus, a device, and a storage medium for enhancing image target pertinence.
Background
In many real-world scenarios, image quality can be affected by adverse factors such as noise, aperture size, and shutter speed. To enable the contrast and detail features of these images to be well revealed and maintained, image enhancement techniques aimed at improving the interpretability and perceptibility of information in the images to the viewer are needed.
By adopting the existing image global enhancement mode, although the effects of highlighting texture, form, color and the like can be achieved on a foreground target in an image, certain influence can be generated on the image background. If the image is excessively enhanced, the characteristics of both the target and the background are enhanced, so that the problems of overlarge noise, distortion of pixel colors and partial textures and the like of the whole image occur, the target enhancement effect is weakened due to the excessive enhancement of the background, the visual effect of highlighting the target cannot be achieved, or the visual effect of the whole image is distorted or excessively changed, so that the observation and judgment of a user on the image are influenced.
Disclosure of Invention
In view of the above, the present invention provides a method, an apparatus, a device and a storage medium for enhancing image target pertinence, which can improve the visual effect of a target in the whole image without distorting the whole image. The specific scheme is as follows:
an image target pertinence enhancement method comprising:
constructing and training an automatic cutout model; the automatic cutout model comprises a segmentation module for generating a target Mask, a Trimap generation module for generating a Trimap capable of being cutout according to the target Mask generated by the segmentation module, and a cutout module for performing boundary correction on the target Mask generated by the segmentation module according to the Trimap generated by the Trimap generation module;
extracting a target Mask in the original image by using the trained automatic cutout model;
enhancing the image information characteristics of the target Mask in the extracted original image to obtain an enhanced target part;
and assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image.
Preferably, in the above method for enhancing image target pertinence provided by the embodiment of the present invention, the segmentation module is composed of a first deep learning segmentation model Unet;
the first deep learning segmentation model Unet is trained by taking an image sample and a corresponding labeled Mask sample as a training set.
Preferably, in the above image target targeting enhancement method provided by the embodiment of the present invention, the first deep learning segmentation model Unet includes a feature extraction part and an upsampling part;
in the feature extraction part, an input picture is subjected to one-layer feature extraction through one convolution layer, and each passage through one pooling layer is one scale, so that multi-scale feature identification of image features on different scales is realized;
in the up-sampling part, the up-sampling output and the corresponding feature output of the feature extraction part are spliced and fused on a channel in each up-sampling.
Preferably, in the method for enhancing image target pertinence provided by the embodiment of the present invention, the Trimap generation module includes an expansion processing portion, a skeleton extraction portion, and a Trimap generation portion;
in the expansion processing part, performing expansion operation on the target Mask generated by the segmentation module to obtain a first foreground;
in the skeleton extraction part, performing skeleton extraction and expansion operation on the target Mask generated by the segmentation module to obtain a second prospect;
and in the Trimap generation part, according to the first foreground and the second foreground, a Trimap which is used for matting and has the same scale with the target Mask is generated.
Preferably, in the above method for enhancing pertinence of an image object provided by the embodiment of the present invention, the matting module is composed of a second deep learning segmentation model Unet;
the second deep learning segmentation model Unet takes an image obtained by channel merging of an input image and the Trimap generated by the Trimap generation module as an input, and takes a target Mask after boundary correction as an output.
Preferably, in the above method for enhancing image target pertinence provided by the embodiment of the present invention, the step of constructing and training an automatic matting model includes:
pre-creating a model which is the same as the segmentation module, and pre-training the model to obtain segmentation module parameters;
constructing the automatic matting module comprising the segmentation module, the Trimap generation module and the matting module, initializing the segmentation module by using the segmentation module parameters, and solidifying the segmentation module parameters;
training the matting module by adopting a randomly initialized matting module parameter;
inputting the collected image samples and the corresponding labeled Mask samples into the automatic cutout model for forward calculation, and outputting a predicted Mask;
calculating the error between the prediction Mask and the labeling Mask;
and judging whether the automatic cutout model is trained or not according to the calculated error.
Preferably, in the method for enhancing pertinence of an image object provided by an embodiment of the present invention, the determining whether the automatic matting model is trained based on the calculated error includes:
when the calculated error is larger than the expected value, calculating the error and the error gradient of each layer in the automatic cutout model, converting the gradient into a weight increment, updating the weight of the network layer, and training again;
and when the calculated error is less than or equal to the expected value, finishing the training, and fixing the weight and the threshold parameter of the automatic cutout model.
The embodiment of the invention also provides an image target pertinence enhancing device, which comprises:
the model training unit is used for constructing and training an automatic cutout model; the automatic cutout model comprises a segmentation module for generating a target Mask, a Trimap generation module for generating a Trimap capable of being cutout according to the target Mask generated by the segmentation module, and a cutout module for performing boundary correction on the target Mask generated by the segmentation module according to the Trimap generated by the Trimap generation module;
the target extraction unit is used for extracting a target Mask in the original image by using the trained automatic cutout model;
the target enhancing unit is used for enhancing the image information characteristics of the extracted target Mask in the original image to obtain an enhanced target part;
and the image generation unit is used for assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image.
The embodiment of the present invention further provides an image target pertinence enhancement apparatus, including a processor and a memory, where the processor implements the above image target pertinence enhancement method provided in the embodiment of the present invention when executing a computer program stored in the memory.
Embodiments of the present invention further provide a computer-readable storage medium for storing a computer program, where the computer program is executed by a processor to implement the above-mentioned image target pertinence enhancement method provided by the embodiments of the present invention.
According to the technical scheme, the image target pertinence enhancing method provided by the invention comprises the following steps: constructing and training an automatic cutout model; the automatic cutout model comprises a segmentation module for generating a target Mask, a Trimap generation module for generating a Trimap capable of being cutout according to the target Mask generated by the segmentation module, and a cutout module for performing boundary correction on the target Mask generated by the segmentation module according to the Trimap generated by the Trimap generation module; extracting a target Mask in the original image by using the trained automatic cutout model; enhancing the image information characteristics of the target Mask in the extracted original image to obtain an enhanced target part; and assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image.
According to the method, the rough target Mask is extracted by using the automatic cutout model, the cutout is used for correcting the target Mask, only the target to be observed in the image is pertinently enhanced, the difference between the target and the background in the image is expanded, and the original information of the background is kept, so that the target can be more highlighted under the condition that the whole image is not distorted, the target in the image can be observed more clearly and accurately, and the visual effect of the target in the whole image is improved. In addition, the invention also provides a corresponding device, equipment and a computer readable storage medium for the image target pertinence enhancement method, so that the method is further more practical, and the device, the equipment and the computer readable storage medium have corresponding advantages.
Drawings
In order to more clearly illustrate the embodiments of the present invention or technical solutions in related arts, the drawings used in the description of the embodiments or related arts will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flowchart of an image target pertinence enhancement method provided by an embodiment of the present invention;
fig. 2 is a schematic flowchart of an image target pertinence enhancement method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of an automatic cutout model provided by an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a partitioning module according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a process of outputting a target Mask after an original image is processed by a segmentation module according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a Trimap generation module according to an embodiment of the present invention;
fig. 7 shows a segmented ductal Mask and a corresponding first foreground Mask0 according to an embodiment of the present invention;
fig. 8 shows a segmented ductal Mask and a corresponding second foreground Mask1 according to an embodiment of the present invention;
fig. 9 is a Trimap generated by the Trimap generation module according to the embodiment of the present invention;
FIG. 10 is a diagram of an automatic matting model generating a target-enhanced image according to an embodiment of the present invention;
FIG. 11 is a final accurate duct division Mask provided by an embodiment of the present invention;
fig. 12 to fig. 16 are comparison diagrams of the ductal original image and the corresponding labeled Mask provided by the embodiment of the present invention;
FIGS. 17 and 18 are diagrams illustrating a model training process according to an embodiment of the present invention;
FIG. 19 is a flowchart of a model training process provided by an embodiment of the present invention;
fig. 20 to 27 are comparison diagrams of the original image and the effect diagram of the glandular tube after the targeted enhancement provided by the embodiment of the invention;
fig. 28 is a schematic structural diagram of an image target pertinence enhancing apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention provides an image target pertinence enhancing method, as shown in fig. 1, comprising the following steps:
s101, constructing and training an automatic cutout model; as shown in fig. 2 and fig. 3, the automatic cutout model includes a segmentation module for generating an object Mask, a Trimap generation module for generating a Trimap capable of being cutout according to the object Mask generated by the segmentation module, and a cutout module for performing boundary correction on the object Mask generated by the segmentation module according to the Trimap generated by the Trimap generation module;
s102, extracting a target Mask in the original image by using the trained automatic cutout model;
s103, enhancing image information characteristics of the target Mask in the extracted original image to obtain an enhanced target part; it should be noted that, this step only enhances the image information characteristics such as color, form, etc. of the extracted target portion, so that the original unclear target becomes clear, the features such as texture form, etc. of the target are emphasized, the difference between the target and the background in the image is enlarged, but the original information of the background can be retained;
and S104, assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image.
In the method for enhancing the pertinence of the image target provided by the embodiment of the invention, the automatic cutout model is used for extracting the rough target Mask, then the cutout is used for correcting the target Mask, and then only the target to be observed in the image is pertinently enhanced, so that the difference between the target and the background in the image is enlarged, and meanwhile, the original information of the background is kept, so that the target can be more highlighted under the condition that the whole image is not distorted, the target in the image can be observed more clearly and accurately, and the visual effect of the target in the whole image is improved.
In the following, taking an endoscopic image as an example, the enhancement is performed by targeting the ductus gland in the endoscopic image. The invention can extract the gland Mask based on the automatic cutout model, and then assigns pixels to the original image after the gland part is enhanced, thereby obtaining the enhanced image only aiming at the enhancement of the gland.
In specific implementation, in the above image target pertinence enhancement method provided in the embodiment of the present invention, the segmentation module may be composed of a first deep learning segmentation model Unet; the first deep learning segmentation model Unet is trained by taking image samples and corresponding labeled Mask samples as a training set.
When the duct in the endoscopic image is taken as a target, duct Mask is extracted in a segmentation module based on a first deep learning segmentation model Unet, the model is trained according to a certain duct image and duct labeling Mask samples, and after a certain segmentation precision is obtained, module parameters are fixed and used for extracting the Mask of the target object. That is to say, the model is stored after the pre-training data preparation (the collection of glandular vessel image samples and the labeling of glandular vessel masks) and the training and parameter adjustment optimization of the Unet segmentation model, and the trained model has the function of segmenting the glandular vessel masks with certain precision.
Further, in specific implementation, in the above-mentioned image target-specific enhancement method provided by the embodiment of the present invention, as shown in fig. 4, the first deep learning segmentation model Unet may include a feature extraction part (left part) and an upsampling part (right part). In the feature extraction part, an input picture extracts a layer of features through one convolution layer every time, and a feature graph of one scale can be output every time the input picture passes through one pooling layer, so that multi-scale feature recognition of image features on different scales is realized; in particular, a total of 5 scales in the model, i.e. the model, enables multi-scale feature recognition of image features on 5 different scales. In the up-sampling part, the U-net adopts a fusion mode of splicing the features together in the channel dimension, and the output of the up-sampling and the features of the corresponding feature extraction part are output in each up-sampling and spliced on the channel to form thicker features.
As shown in fig. 5, the first deep learning segmentation model Unet training is To train an Image-To-Image model according To an existing picture and a corresponding Mask, the model generates a corresponding predicted Mask through learning of an original Image, then the labeled Mask is used as a Mask, the recognition segmentation capability of the model is improved by minimizing the error between the actual labeled Mask and the model predicted Mask, and the pixels belonging To the ductus glandulae are recognized and segmented.
It can be understood that after the present invention adopts the segmentation module to segment and extract the ductal Mask, the extracted ductal Mask will inevitably contain partial non-ductal partial content and impurities due to the characteristics of unclear visible boundary, fuzzy boundary, etc. of partial ductal, and the ductal boundary has a certain error. Therefore, the invention further adopts a trigap for matting based on the matting module to further optimize the gland Mask, and aims to remove part of non-gland and precise gland boundaries.
In specific implementation, in the image target pertinence enhancement method provided by the embodiment of the present invention, the Trimap generation module may include an expansion processing part, a skeleton extraction part, and a Trimap generation part; as shown in fig. 6, in the expansion processing part, the target Mask generated by the segmentation module is subjected to an expansion operation to obtain a first foreground Mask0 (i.e., a possible foreground); in the skeleton extraction part, skeleton extraction and expansion operations are carried out on the target Mask generated by the segmentation module to obtain a second foreground Mask1 (namely the determined foreground); in the Trimap generation section, trimaps on the same scale as the target masks for matting are generated based on the first foreground Mask0 and the second foreground Mask 1.
Specifically, the Trimap generation module may generate Trimap for matting through image processing operations such as morphology according to the glandular tube Mask generated by the segmentation module. As shown in fig. 7, Mask0 after the dilation operation of the segmented ductal Mask is a possible prospect. As shown in fig. 8, the segmented glandular duct Mask is expanded after the skeleton extraction operation, and a new Mask1 is generated, Mask1 is used as a confirmation prospect. And skeleton extraction, namely binary image refinement, namely refining a connected region into the width of one pixel for feature extraction and target topology representation. The invention may employ the Skeletonize function in the morphology submodule of the Skymage library of Python.
And (3) newly building a Trimap which is used for matting and has the same scale with the segmented glandular Mask and is all 0, wherein in the Trimap, the pixel values of the corresponding points of the white part in the Mask0 and the black part in the Mask1 are set to be 3 (namely, the corresponding points may be foreground), and the pixel values of the corresponding points of the white part in the Mask1 are set to be 1 (namely, the corresponding points must be foreground). Trimap is shown in fig. 9, where the black part is the background and the white part is the determined foreground.
In specific implementation, in the above method for enhancing image target pertinence provided in the embodiment of the present invention, the matting module may be formed by a second deep learning segmentation model Unet; the second deep learning segmentation model Unet takes an image obtained by channel merging of an input image and the Trimap generated by the Trimap generation module as an input, and takes a target Mask after boundary correction as an output.
Specifically, an accurate target Mask is calculated and obtained based on the second deep learning segmentation model Unet and the generated Trimap. The matting module is used for carrying out boundary correction on the Mask obtained by the segmentation module and refining the glandular tube shape. The second deep learning segmentation model Unet may be identical in structure to the first deep learning segmentation model Unet.
As shown in fig. 10, the automatic matting model uses a depth-learning based segmentation model Unet to matte the image object. The matting module carries out channel combination according to the input image and the Trimap generated by the segmentation module to generate input of a thought channel, and the matting module is input to extract the gland duct Mask for gland duct boundary contour refinement and precision as shown in fig. 11.
In specific implementation, in the image target pertinence enhancement method provided in the embodiment of the present invention, the step of constructing and training the automatic matting model in step S101 may specifically include the following steps:
firstly, a model which is the same as a segmentation module is created in advance, and the model is pre-trained to obtain parameters of the segmentation module;
it should be noted that before step one is performed, training data needs to be prepared. Fig. 12 to 16 show ductal samples, the left image of each group being the original captured image, the right image being the corresponding labeled Mark, the white areas representing the ductal duct, and the black areas representing the background. In the process of executing the step one, a segmentation module Unet model is created, parameters are initialized, pre-training is carried out according to later ductal samples, and module parameters are stored after the precision reaches a certain degree.
Constructing an automatic matting module comprising a segmentation module, a Trimap generation module and a matting module, initializing the segmentation module by using segmentation module parameters, and solidifying the segmentation module parameters; training a matting module by adopting a random initialization matting module parameter;
specifically, the segmentation module parameters obtained by training in the step one are adopted to initialize the segmentation module of the automatic cutout model, the module parameters are solidified and kept unchanged during training, and the cutout module parameters are initialized randomly.
Inputting the collected image samples and the corresponding marked Mask samples into an automatic cutout model for forward calculation, and outputting a predicted Mask;
specifically, as shown in fig. 17 and 18, a picture of BatchSize and an actual labeled Mask are read, and the picture input model is subjected to forward calculation;
step four, calculating and outputting a prediction Mask by the Unet model, and calculating an error between the prediction Mask and an actual labeled Mask;
and step five, judging whether the automatic cutout model is trained or not according to the calculated error.
In a specific implementation, in the method for enhancing image target pertinence provided by the embodiment of the present invention, the fifth step determines whether the automatic cutout model is trained according to the calculated error, as shown in fig. 19, and may specifically include: when the calculated error is larger than the expected value, calculating the error and the error gradient of each layer in the automatic cutout model according to the Adam optimizer, the learning rate and the historical weight, converting the gradient into a weight increment, updating the weight of the network layer in the automatic cutout model, training again, namely skipping to the second step; and when the calculated error is less than or equal to the expected value, finishing the training, fixing the weight and the threshold parameter of the automatic cutout model, and storing the current automatic cutout model.
In specific implementation, when step S102 is executed, after the automatic cutout model is created, the weight parameters in the trained model are loaded, then the ductal image is read by using the model, and the calculation processing is started, where the model calculates an output ductal Mask (that is, a black region is a background, and a white region is a ductal region), and the Mask is a binary image with the same size as the input image.
In specific implementation, when step S103 is executed, only the enhancement effect in the ductal Mask is superimposed on the original image according to the extracted ductal Mask, so that not only the background effect of the original image is retained, but also the texture structure of the ductal Mask is more obvious, and the ductal Mask which is not obvious originally is visually enhanced.
Glandular duct enhancement was performed using the following formula:
ImageO＝α·ImageI-β·ImageGSF
wherein, the image is input by ImageI, the ImageGSF is a processed image of the input image by the fastGloblal SmootherFilter algorithm of Opencv, alpha can be 2.0, and beta can be 1.0. It should be noted that Opencv is a cross-platform computer vision library issued based on BSD license (open source), is lightweight and efficient, provides interfaces of languages such as Python, Ruby, MATLAB, and the like, and implements many general algorithms in image processing and computer vision.
Next, step S104 is executed to assign the pixel values of the ductal portion in the enhanced image to the original image, and generate an enhanced effect map for the target ductal enhancement.
Fig. 20 to 27 are graphs showing the comparison between the original and the duct-specific enhanced effect, in which the original is shown on the left side and the duct-enhanced image is shown on the right side. Therefore, the invention can clearly and accurately observe the glandular duct in the image and retain the original information of the background.
Based on the same inventive concept, the embodiment of the present invention further provides an image target pertinence enhancement apparatus, and as the principle of the apparatus for solving the problem is similar to that of the aforementioned image target pertinence enhancement method, the implementation of the apparatus may refer to the implementation of the image target pertinence enhancement method, and repeated details are not repeated.
In specific implementation, the image target pertinence enhancing apparatus provided in the embodiment of the present invention, as shown in fig. 28, specifically includes:
the model training unit 11 is used for constructing and training an automatic cutout model; the automatic cutout model comprises a segmentation module for generating a target Mask, a Trimap generation module for generating a Trimap capable of being cutout according to the target Mask generated by the segmentation module, and a cutout module for performing boundary correction on the target Mask generated by the segmentation module according to the Trimap generated by the Trimap generation module;
the target extraction unit 12 is used for extracting a target Mask in the original image by using the trained automatic matting model;
a target enhancing unit 13, configured to enhance image information characteristics of a target Mask in the extracted original image to obtain an enhanced target portion;
and the image generating unit 14 is used for assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image.
In the image target pertinence enhancement device provided by the embodiment of the invention, a rough target Mask can be extracted firstly through the interaction of the four modules, then the target Mask is corrected by matting, and then only a target to be observed in the image is pertinently enhanced, so that the difference between the target and the background in the image is enlarged, and meanwhile, original background information is kept, so that the target can be more highlighted under the condition that the whole image is not distorted, the target in the image can be observed more clearly and accurately, and the visual effect of the target in the whole image is improved.
For more specific working processes of the modules, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
Correspondingly, the embodiment of the invention also discloses image target pertinence enhancing equipment, which comprises a processor and a memory; wherein the processor, when executing the computer program stored in the memory, implements the image target specific enhancement method disclosed in the foregoing embodiments.
For more specific processes of the above method, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
Further, the present invention also discloses a computer readable storage medium for storing a computer program; the computer program when executed by a processor implements the image object targeting enhancement method disclosed previously.
For more specific processes of the above method, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device, the equipment and the storage medium disclosed by the embodiment correspond to the method disclosed by the embodiment, so that the description is relatively simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
To sum up, an image target pertinence enhancing method provided by the embodiment of the present invention includes: constructing and training an automatic cutout model; the automatic cutout model comprises a segmentation module for generating a target Mask, a Trimap generation module for generating a Trimap capable of being cutout according to the target Mask generated by the segmentation module, and a cutout module for performing boundary correction on the target Mask generated by the segmentation module according to the Trimap generated by the Trimap generation module; extracting a target Mask in the original image by using the trained automatic cutout model; enhancing the image information characteristics of the target Mask in the extracted original image to obtain an enhanced target part; and assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image. Therefore, the automatic cutout model is used for extracting rough target masks firstly, then the cutout is used for correcting the target masks, and then only the target to be observed in the image is pertinently enhanced, so that the difference between the target in the image and the background is enlarged, and meanwhile, the original information of the background is kept, so that the target can be more highlighted under the condition that the whole image is not distorted, the target in the image can be observed more clearly and accurately, and the visual effect of the target in the whole image is improved. In addition, the invention also provides a corresponding device, equipment and a computer readable storage medium for the image target pertinence enhancement method, so that the method is further more practical, and the device, the equipment and the computer readable storage medium have corresponding advantages.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above detailed description of the method, apparatus, device and storage medium for enhancing image target pertinence provided by the present invention is provided, and a specific example is applied in the present disclosure to explain the principle and the implementation of the present invention, and the above description of the embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. An image target pertinence enhancement method, comprising:
constructing and training an automatic cutout model; the automatic cutout model comprises a segmentation module for generating a target Mask, a Trimap generation module for generating a Trimap capable of being cutout according to the target Mask generated by the segmentation module, and a cutout module for performing boundary correction on the target Mask generated by the segmentation module according to the Trimap generated by the Trimap generation module;
extracting a target Mask in the original image by using the trained automatic cutout model;
enhancing the image information characteristics of the target Mask in the extracted original image to obtain an enhanced target part;
and assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image.
2. The image target targeting enhancement method according to claim 1, characterized in that said segmentation module is constituted by a first deep learning segmentation model Unet;
the first deep learning segmentation model Unet is trained by taking an image sample and a corresponding labeled Mask sample as a training set.
3. The image target-specific enhancement method according to claim 2, wherein the first deep learning segmentation model Unet includes a feature extraction part and an upsampling part;
in the feature extraction part, an input picture is subjected to one-layer feature extraction through one convolution layer, and each passage through one pooling layer is one scale, so that multi-scale feature identification of image features on different scales is realized;
in the up-sampling part, the up-sampling output and the corresponding feature output of the feature extraction part are spliced and fused on a channel in each up-sampling.
4. The image target targeting enhancement method of claim 3, wherein the Trimap generation module comprises a dilation processing part, a skeleton extraction part and a Trimap generation part;
in the expansion processing part, performing expansion operation on the target Mask generated by the segmentation module to obtain a first foreground;
in the skeleton extraction part, performing skeleton extraction and expansion operation on the target Mask generated by the segmentation module to obtain a second prospect;
and in the Trimap generation part, according to the first foreground and the second foreground, a Trimap which is used for matting and has the same scale with the target Mask is generated.
5. The image target-specific enhancement method of claim 4, wherein the matting module is composed of a second deep learning segmentation model Unet;
the second deep learning segmentation model Unet takes an image obtained by channel merging of an input image and the Trimap generated by the Trimap generation module as an input, and takes a target Mask after boundary correction as an output.
6. The image object targeting enhancement method of claim 5, wherein the step of constructing and training an automatic matting model comprises:
pre-creating a model which is the same as the segmentation module, and pre-training the model to obtain segmentation module parameters;
constructing the automatic matting module comprising the segmentation module, the Trimap generation module and the matting module, initializing the segmentation module by using the segmentation module parameters, and solidifying the segmentation module parameters;
training the matting module by adopting a randomly initialized matting module parameter;
inputting the collected image samples and the corresponding labeled Mask samples into the automatic cutout model for forward calculation, and outputting a predicted Mask;
calculating the error between the prediction Mask and the labeling Mask;
and judging whether the automatic cutout model is trained or not according to the calculated error.
7. The method for image object-specific enhancement according to claim 6, wherein said determining whether the automatic matting model is trained based on the calculated error comprises:
when the calculated error is larger than the expected value, calculating the error and the error gradient of each layer in the automatic cutout model, converting the gradient into a weight increment, updating the weight of the network layer, and training again;
and when the calculated error is less than or equal to the expected value, finishing the training, and fixing the weight and the threshold parameter of the automatic cutout model.
8. An image target pertinence enhancement apparatus, comprising:
the model training unit is used for constructing and training an automatic cutout model; the automatic cutout model comprises a segmentation module for generating a target Mask, a Trimap generation module for generating a Trimap capable of being cutout according to the target Mask generated by the segmentation module, and a cutout module for performing boundary correction on the target Mask generated by the segmentation module according to the Trimap generated by the Trimap generation module;
the target extraction unit is used for extracting a target Mask in the original image by using the trained automatic cutout model;
the target enhancing unit is used for enhancing the image information characteristics of the extracted target Mask in the original image to obtain an enhanced target part;
and the image generation unit is used for assigning the pixel value of the enhanced target part to the original image to generate a target enhanced image.
9. An image object targeting enhancement device comprising a processor and a memory, wherein the processor, when executing a computer program stored in the memory, implements the image object targeting enhancement method according to any one of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the image object-specific enhancement method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110779988.6A CN113409224B (en) | 2021-07-09 | 2021-07-09 | Image target pertinence enhancement method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110779988.6A CN113409224B (en) | 2021-07-09 | 2021-07-09 | Image target pertinence enhancement method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113409224A true CN113409224A (en) | 2021-09-17 |
| CN113409224B CN113409224B (en) | 2023-07-04 |
Family
ID=77685683
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110779988.6A Active CN113409224B (en) | 2021-07-09 | 2021-07-09 | Image target pertinence enhancement method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113409224B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116843708A (en) * | 2023-08-30 | 2023-10-03 | 荣耀终端有限公司 | Image processing method and device |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8175384B1 (en) * | 2008-03-17 | 2012-05-08 | Adobe Systems Incorporated | Method and apparatus for discriminative alpha matting |
| US20180253865A1 (en) * | 2017-03-02 | 2018-09-06 | Adobe Systems Incorporated | Image matting using deep learning |
| CN110400323A (en) * | 2019-07-30 | 2019-11-01 | 上海艾麒信息科技有限公司 | It is a kind of to scratch drawing system, method and device automatically |
| US20200193609A1 (en) * | 2018-12-18 | 2020-06-18 | Qualcomm Incorporated | Motion-assisted image segmentation and object detection |
| CN111429452A (en) * | 2020-04-15 | 2020-07-17 | 深圳市嘉骏实业有限公司 | Bladder ultrasonic image segmentation method and device based on UNet convolutional neural network |
| CN112446380A (en) * | 2019-09-02 | 2021-03-05 | 华为技术有限公司 | Image processing method and device |
| CN112581480A (en) * | 2020-12-22 | 2021-03-30 | 深圳市雄帝科技股份有限公司 | Automatic image matting method, system and readable storage medium thereof |
| CN112634314A (en) * | 2021-01-19 | 2021-04-09 | 深圳市英威诺科技有限公司 | Target image acquisition method and device, electronic equipment and storage medium |
| CN112801896A (en) * | 2021-01-19 | 2021-05-14 | 西安理工大学 | Backlight image enhancement method based on foreground extraction |
| CN113052755A (en) * | 2019-12-27 | 2021-06-29 | 杭州深绘智能科技有限公司 | High-resolution image intelligent matting method based on deep learning |
-
2021
- 2021-07-09 CN CN202110779988.6A patent/CN113409224B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8175384B1 (en) * | 2008-03-17 | 2012-05-08 | Adobe Systems Incorporated | Method and apparatus for discriminative alpha matting |
| US20180253865A1 (en) * | 2017-03-02 | 2018-09-06 | Adobe Systems Incorporated | Image matting using deep learning |
| US20200193609A1 (en) * | 2018-12-18 | 2020-06-18 | Qualcomm Incorporated | Motion-assisted image segmentation and object detection |
| CN110400323A (en) * | 2019-07-30 | 2019-11-01 | 上海艾麒信息科技有限公司 | It is a kind of to scratch drawing system, method and device automatically |
| CN112446380A (en) * | 2019-09-02 | 2021-03-05 | 华为技术有限公司 | Image processing method and device |
| CN113052755A (en) * | 2019-12-27 | 2021-06-29 | 杭州深绘智能科技有限公司 | High-resolution image intelligent matting method based on deep learning |
| CN111429452A (en) * | 2020-04-15 | 2020-07-17 | 深圳市嘉骏实业有限公司 | Bladder ultrasonic image segmentation method and device based on UNet convolutional neural network |
| CN112581480A (en) * | 2020-12-22 | 2021-03-30 | 深圳市雄帝科技股份有限公司 | Automatic image matting method, system and readable storage medium thereof |
| CN112634314A (en) * | 2021-01-19 | 2021-04-09 | 深圳市英威诺科技有限公司 | Target image acquisition method and device, electronic equipment and storage medium |
| CN112801896A (en) * | 2021-01-19 | 2021-05-14 | 西安理工大学 | Backlight image enhancement method based on foreground extraction |
Non-Patent Citations (1)
| Title |
|---|
| 白杨;姚桂林;: "一种基于KNN后处理的鲁棒性抠图方法", 计算机应用与软件, no. 09, pages 176 - 181 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116843708A (en) * | 2023-08-30 | 2023-10-03 | 荣耀终端有限公司 | Image processing method and device |
| CN116843708B (en) * | 2023-08-30 | 2023-12-12 | 荣耀终端有限公司 | Image processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113409224B (en) | 2023-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110163219B (en) | Target detection method based on image edge recognition | |
| JP5007953B2 (en) | Image processing apparatus, image processing method, and program | |
| CN109903294B (en) | Image processing method and device, electronic equipment and readable storage medium | |
| CN107871319B (en) | Method and device for detecting beam limiter area, X-ray system and storage medium | |
| CN110390643B (en) | License plate enhancement method and device and electronic equipment | |
| CN111275034B (en) | Method, device, equipment and storage medium for extracting text region from image | |
| CN110147816B (en) | Method and device for acquiring color depth image and computer storage medium | |
| CN102930515A (en) | Automatic geometric distortion correction method of digital image | |
| JP5534411B2 (en) | Image processing device | |
| CN113780110A (en) | Method and device for detecting weak and small targets in image sequence in real time | |
| JP6914734B2 (en) | Silhouette extractor, method and program | |
| CN113409224A (en) | Image target pertinence enhancing method, device, equipment and storage medium | |
| JP2008146278A (en) | Cell outline extraction device, cell outline extraction method and program | |
| CN104036481A (en) | Multi-focus image fusion method based on depth information extraction | |
| CN115578289A (en) | Defocused image deblurring method based on boundary neighborhood gradient difference | |
| CN113421210B (en) | Surface point Yun Chong construction method based on binocular stereoscopic vision | |
| CN111932594B (en) | Billion pixel video alignment method and device based on optical flow and medium | |
| CN117132503A (en) | Method, system, equipment and storage medium for repairing local highlight region of image | |
| CN112949453A (en) | Training method of smoke and fire detection model, smoke and fire detection method and smoke and fire detection equipment | |
| EP3018626B1 (en) | Apparatus and method for image segmentation | |
| CN112288680A (en) | Automatic defect area extraction method and system for automobile hub X-ray image | |
| CN110602476A (en) | Hole filling method of Gaussian mixture model based on depth information assistance | |
| JP5761353B2 (en) | Ridge direction extraction device, ridge direction extraction method, ridge direction extraction program | |
| WO2021009804A1 (en) | Method for learning threshold value | |
| JP6811679B2 (en) | Silhouette extractor, method and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |