CN112348755A - Image content restoration method, electronic device and storage medium - Google Patents
Image content restoration method, electronic device and storage medium Download PDFInfo
- Publication number
- CN112348755A CN112348755A CN202011196606.9A CN202011196606A CN112348755A CN 112348755 A CN112348755 A CN 112348755A CN 202011196606 A CN202011196606 A CN 202011196606A CN 112348755 A CN112348755 A CN 112348755A
- Authority
- CN
- China
- Prior art keywords
- image
- processed
- repaired
- area
- filling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 230000000873 masking effect Effects 0.000 claims abstract description 26
- 239000011159 matrix material Substances 0.000 claims description 30
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 29
- 230000009466 transformation Effects 0.000 claims description 29
- 238000012545 processing Methods 0.000 claims description 20
- 238000013507 mapping Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 230000008439 repair process Effects 0.000 claims description 6
- 239000000945 filler Substances 0.000 claims description 5
- 238000013519 translation Methods 0.000 claims description 5
- 238000003708 edge detection Methods 0.000 claims description 4
- 230000007547 defect Effects 0.000 abstract description 14
- 230000000694 effects Effects 0.000 abstract description 7
- 230000004927 fusion Effects 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 14
- 210000000887 face Anatomy 0.000 description 10
- 238000004891 communication Methods 0.000 description 6
- 206010000496 acne Diseases 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000001815 facial effect Effects 0.000 description 4
- 208000002874 Acne Vulgaris Diseases 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 210000001061 forehead Anatomy 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- 230000037303 wrinkles Effects 0.000 description 2
- 208000007256 Nevus Diseases 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/02—Affine transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration using local operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the invention provides an image content restoration method, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a to-be-repaired area masking layout mapped on an image to be processed, and determining a filling image of the to-be-repaired area in the to-be-repaired area masking layout; and repairing the area to be repaired on the image to be processed according to the filling image. The image content restoration method, the electronic device and the storage medium provided by the embodiment of the invention are used for acquiring the mask image of the area to be restored mapped on the image to be processed, determining the filling image of the area to be restored in the mask image of the area to be restored, restoring the area to be restored on the image to be processed by the filling image, realizing the reasonable fusion of the pixel value on the filling image and the pixel value outside the area to be restored on the face image to be processed, and ensuring that the defect restoration effect on the image is better.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to an image content restoration method, an electronic device, and a storage medium.
Background
Images taken by users sometimes require a restoration process to present a better picture state. Especially, the user needs to perform image restoration for self-photographing. For this reason, in the field of digital image processing, an image repair technique is required for repairing defects on an image after a user takes a picture.
The currently adopted image restoration technology mainly fills the area to be restored with the surrounding images directly, which easily causes local blurring of the restored area, or adopts texture matching, which causes the situation that matching fails and defects may still exist. Therefore, the repairing effect cannot be better.
Disclosure of Invention
In order to solve the problems in the prior art, embodiments of the present invention provide an image content restoration method, an electronic device, and a storage medium.
In a first aspect, an embodiment of the present invention provides an image content repairing method, including:
acquiring a to-be-repaired area masking layout mapped on an image to be processed, and determining a filling image of the to-be-repaired area in the to-be-repaired area masking layout;
and repairing the area to be repaired on the image to be processed according to the filling image.
Further, the acquiring a masking map of the region to be repaired mapped on the image to be processed includes:
deriving a standard Mongolian layout based on the marked region to be repaired in the standard image;
acquiring key point data of a standard image and an image to be processed, and determining an affine transformation matrix according to the key point data;
and mapping the standard mask graph by adopting the affine transformation matrix to obtain a mask graph of the area to be repaired.
Further, the obtaining of the key point data of the standard image and the image to be processed and determining the mapping transformation matrix according to the key point data includes:
performing triangulation processing according to the key point data of the standard image and the image to be processed to obtain a first grid corresponding to the standard image and a second grid corresponding to the image to be processed;
establishing an affine transformation matrix between triangular faces in the first mesh and corresponding triangular faces in the second mesh.
Further, the determining of the filling image of the region to be repaired in the masking layout of the region to be repaired includes:
and moving the masking layout of the area to be repaired in a preset direction on the image to be processed, and selecting the image corresponding to the position as a filling image of the area to be repaired based on the position of the area to be repaired on the image to be processed in the masking layout of the area to be repaired after translation.
Further, repairing the region to be repaired on the image to be processed according to the filling image includes:
determining that pixel points in the filling image correspond to matching pixel points on the image to be processed, and updating pixel values of the pixel points in the filling image according to the pixel values of the matching pixel points;
and repairing the area to be repaired on the image to be processed according to the filling image after the pixel value is updated.
Further, the determining that the pixel points in the filling image correspond to the matching pixel points on the image to be processed includes:
traversing pixel points in the filling image, selecting target pixel points, and determining the positions of the target pixel points;
constructing a texture block with a preset size by taking the position of the target pixel point as a center;
calculating and obtaining the similarity between a target pixel point and each pixel point in a preset number of pixel points based on the positions of the pixel points in the texture block and the positions of the preset number of pixel points selected on the image to be processed;
and determining the pixel point corresponding to the minimum similarity as the matching pixel point.
Further, after obtaining the mask map of the area to be repaired mapped on the image to be processed, the method further includes:
determining a minimum external rectangle of the to-be-repaired area in the to-be-repaired area mask map, performing edge detection on the to-be-repaired area on the to-be-processed image based on the minimum external rectangle, determining an updating to-be-repaired area, acquiring the updating to-be-repaired mask map mapped on the to-be-processed image according to the updating to-be-repaired area, and using the updating to-be-repaired mask map as a new to-be-repaired area mask map.
Further, after repairing the region to be repaired on the image to be processed according to the filling image after the pixel value is updated, the method further includes: and carrying out filtering processing on the image to be processed after the restoration processing.
In a second aspect, an embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the steps of the image content repairing method.
In a third aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the image content restoration method as described above.
The image content restoration method, the electronic device and the storage medium provided by the embodiment of the invention are used for acquiring the mask image of the area to be restored mapped on the image to be processed, determining the filling image of the area to be restored in the mask image of the area to be restored, restoring the area to be restored on the image to be processed by the filling image, realizing the reasonable fusion of the pixel value on the filling image and the pixel value outside the area to be restored on the face image to be processed, and ensuring that the defect restoration effect on the image is better.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a flowchart of an embodiment of an image content restoration method according to the present invention;
FIG. 2 is a schematic flow chart of an image content restoration method according to the present invention;
FIG. 3 is a block diagram of an embodiment of an image content restoration apparatus according to the present invention;
FIG. 4 is a block diagram of an embodiment of an electronic device according to the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flow chart illustrating an image content restoration method according to an embodiment of the present invention, and referring to fig. 1, the method includes:
s11, acquiring a to-be-repaired area masking layout mapped on the to-be-processed image, and determining a filling image of the to-be-repaired area in the to-be-repaired area masking layout;
and S12, repairing the region to be repaired on the image to be processed according to the filling image.
With respect to step S11 and step S12, it should be noted that, in the embodiment of the present invention, the image captured by the user sometimes needs to be subjected to a restoration process to present a better screen state. Especially, the user needs to perform image restoration for self-photographing. From the skin condition revealed by the photographs, the skin is prone to blemishes, which may include wrinkles (raised lines, french lines, etc.), spots, acne (comedones and pox), and also black nevi, as well as stains stuck to the skin. For this reason, it is more necessary to repair these defects.
In an embodiment of the present invention, the different defects are configured with different defect types. The types corresponding to the above-mentioned defects are a grain type, a raising line type, a mottled type, a acne type, and the like.
And different defect types are provided with corresponding outline areas on the image, and the outline areas are areas to be repaired which need to be repaired.
For example, a contour region of the type of a french-mark, may be a "dog-bone" region.
The head raising line type contour region may be a preset region of the forehead.
The outline region of the acne type can be a preset region of the forehead, a preset region of the left face or a preset region of the right face.
In the embodiment of the invention, the method needs to map the image to be processed to obtain the mask layout of the region to be repaired. The mask map of the area to be repaired comprises the area to be repaired.
And then, filling the image in the region to be repaired in the masking layout of the region to be repaired. The filled image is used for further perfecting as a basis for repairing the flaws on the image to be processed. The filler image may capture an image in an area on the image to be processed that is free of defects. The image belongs to a weak texture structure, has uniform color and no obvious brightness change.
And then, repairing the pixel points in the to-be-repaired area on the to-be-processed image based on the pixel points on the filling image to obtain an image with better color.
The image content restoration method provided by the embodiment of the invention obtains the mask layout of the area to be restored mapped on the image to be processed, and determines the filling image of the area to be restored in the mask layout of the area to be restored, so that the filling image restores the area to be restored on the image to be processed, the pixel value on the filling image and the pixel value outside the area to be restored on the face image to be processed are reasonably fused, and the defect restoration effect on the image is better.
In a further embodiment of the foregoing embodiment method, a process of repairing a to-be-repaired area on a to-be-processed image according to a fill image is mainly explained, which specifically includes the following steps:
determining that pixel points in the filled image correspond to matching pixel points on the image to be processed, and updating pixel values of the pixel points in the image according to pixel values of the matching pixel points;
and repairing the area to be repaired on the image to be processed according to the filling image after the pixel value is updated.
In contrast, it should be noted that the improvement of the filled image is mainly embodied in the updating of the pixel values of the pixels in the image, and the purpose of the improvement is to maximize the reasonable fusion of the filled image and the pixels around the to-be-repaired area in the to-be-processed image. Therefore, the matching position of the pixel point in the filling image on the image to be processed needs to be determined, then the pixel value of the pixel point corresponding to the matching position is made to update the pixel value of the pixel point in the filling image, and therefore the pixel perfection of the filling image is completed.
And then, repairing the image of the area to be repaired on the image to be repaired according to the filling image after the pixel value is updated to obtain a repaired image. The repaired image has better picture feeling compared with the image to be processed.
According to the embodiment, the pixel points on the filling image of the area to be repaired on the masking layout are matched with the reasonable pixel points on the image to be repaired, the pixel values of the matched pixel points are updated to the pixel values on the filling image, then the filling image with the updated pixel values is used for repairing the area to be repaired, the pixel values on the filling image and the pixel values outside the area to be repaired on the image to be repaired are reasonably fused, and the image repairing effect is better.
In a further embodiment of the method in the above embodiment, a description is mainly given to a processing procedure of obtaining a mask layout of a region to be repaired mapped on an image to be processed, specifically as follows:
and deriving a standard Mongolian picture based on the marked defective region in the standard image.
And acquiring key point data of the standard image and the image to be processed, and determining an affine transformation matrix according to the key point data.
And mapping the standard mask graph by adopting an affine transformation matrix to obtain the mask graph of the region to be repaired.
It should be noted that, in the embodiment of the present invention, for the self-photographing, the selected standard image may be a standard facial image, such as a front face or a model facial image without expression.
And automatically marking the area to be repaired (namely the area with the flaw) from the standard image according to the flaw type, and then deriving the area to be repaired in a mask mode based on the area to be repaired to obtain a standard mask layout.
And acquiring an image to be processed, and respectively carrying out key point detection on the standard image and the image to be processed to obtain corresponding key point data. The keypoint detection may employ a 68-point model or a 106-point model.
After the key point data is obtained, a corresponding relation exists between the standard image and the image to be processed, and an affine transformation relation between the two images, namely an affine transformation matrix, can be constructed for the corresponding relation.
After the affine transformation matrix is obtained, numerical calculation can be carried out on the standard Mongolian layout based on the affine transformation matrix, the standard Mongolian layout is mapped to an image to be processed, and the image is regarded as the Mongolian layout of the area to be repaired.
In this embodiment, by setting the standard image, an affine transformation matrix between the standard image and the image to be processed is established, and the standard mask map is mapped to the mask map of the area to be repaired corresponding to the image to be processed, so that the mask map of the area to be repaired, which is closer to the image to be processed, is obtained with the standard image as a reference, and subsequent image repairing processing is facilitated.
In a further embodiment of the method in the above embodiment, a description is mainly given to a processing procedure of obtaining key point data of a standard image and an image to be processed and determining a mapping transformation matrix according to the key point data, which is specifically as follows:
performing triangulation processing according to the key point data of the standard image and the image to be processed to obtain a first grid corresponding to the standard image and a second grid corresponding to the image to be processed;
establishing an affine transformation matrix between triangular faces in the first mesh and corresponding triangular faces in the second mesh.
In this respect, it should be noted that, in the embodiment of the present invention, since the key point data is a location point having a special key feature point on the image, for this reason, based on a reasonable division between these key points, a grid map may be obtained, which is regarded as a grid, that is: a first mesh corresponding to the standard image, and a second mesh corresponding to the face image to be processed.
Since the mesh is a triangular section, there are a plurality of triangular faces in the mesh. The key point data adopts a 68-point model or a 106-point model, and for this reason, corresponding numbers exist in each key point. In this case, the vertices of the triangular faces are uniquely identified by numbers. It can be determined that a certain triangular face in the first mesh corresponds to a certain triangular face in the second mesh, where an affine transformation matrix between the two triangular faces corresponding to each other needs to be established. In practice, the affine transformation matrix is an affine transformation matrix between two triangular surface vertices. After the affine transformation matrix is determined, all image pixel points in the vertex area of the triangular surface are transformed based on the affine matrix.
In addition, after the 68 point model is adopted to obtain the key point data, in order to improve the accuracy of affine, point interpolation can be performed on the key point data, and more key point data can be obtained.
In this embodiment, the two images are subjected to key point detection to determine key point data, so as to obtain a mesh, and then an affine transformation matrix between vertices is established based on the unique correspondence of the triangular surface, so as to realize accurate affine between Mongolian layouts.
Fig. 2 is a schematic specific flow chart of the image content repairing method in this embodiment, and based on the process from S21 to S24 in fig. 2, a specific example description is given below of obtaining a montage layout by taking a legal wrinkle on a face as an example, specifically as follows:
1) selecting a standard face image (face image of a front face or a model without expression) as a standard face image, and recording the standard face image as S1Marking a shape region of the statute on the standard face image, deriving the shape region in a mask form, and recording as M as a standard mask pattern1。
2) Inputting a face image to be processed, and recording as S2Respectively detecting key points of the standard face image and the face image to be processed, and obtaining key point data (for marking the position or the shape of a facial organ, such as a universal 68-point model or a 106-point model) of the face, wherein the corresponding key point data of the two images are respectively marked as L1And L2。
3) Based on the above-mentioned key point data L1And L2And generating a corresponding mesh (a mesh structure formed by connecting lines among three discrete vertexes) by using a Delaunay triangulation method, wherein the mesh is formed by a plurality of triangular surfaces, and the vertex of each triangular surface stores a key point index number (the vertex number corresponds to 0-108).
4) Traversing each triangular face in the mesh, e.g., triangular face Δ of a standard face image1Triangular face Δ of face image to be processed2Calculating an affine transformation matrix between two triangular surfaces (here, calculating an affine transformation matrix between two sets of triangular vertices, and then transforming image pixels in the vertex regions based on the affine matrix), and transforming the standard Mongolian surface M using the affine transformation matrix1Delta of1Region image mapping to delta2In the region, after traversing all the triangular faces in the grid, obtaining a region mask layout on the face image to be processed, and marking as M2-1。
In a further embodiment of the foregoing embodiment method, mainly after obtaining the mask map of the region to be repaired mapped on the image to be processed, the method further includes the following subsequent steps:
determining a minimum external rectangle of the to-be-repaired area in the to-be-repaired area mask map, performing edge detection on the to-be-repaired area on the to-be-processed image based on the minimum external rectangle, determining an updating to-be-repaired area, acquiring the updating to-be-repaired mask map mapped on the to-be-processed image according to the updating to-be-repaired area, and using the updating to-be-repaired mask map as a new to-be-repaired area mask map.
To this end, it should be noted that, taking a legal line as an example to describe continuously, the masking layout M of the region to be repaired is calculated2-1For the minimum bounding rectangle ofAnd performing texture region detection on the image to be processed in the domain by using Frangi filtering, determining a region, and regarding the region as an 'updating region to be repaired'. Then, the area to be repaired for updating is used for obtaining a mask map mapped on the image to be processed, the mask map is regarded as a mask map for updating and is marked as M2-2。
In a further embodiment of the method in the above embodiment, a process of determining a filling image of the to-be-repaired region in the to-be-repaired region masking map is mainly explained, which specifically includes the following steps:
and moving the masking layout of the region to be repaired in a preset direction (horizontal, vertical or oblique line) on the image to be processed, and selecting the image corresponding to the position as a filling image of the region to be repaired based on the position of the region to be repaired on the image to be processed in the masking layout of the region to be repaired after translation. The filler image may capture an image in an area on the image to be processed that is free of defects. The image belongs to a weak texture structure, has uniform color and no obvious brightness change.
In this embodiment, an image of a region close to the region to be repaired can be obtained by the Mongolian layout translation, so that the subsequent accurate matching of the defect repair is facilitated.
In a further embodiment of the method according to the above embodiment, the explanation is mainly given to a process of determining that a pixel point in a filled image corresponds to a matching pixel point on an image to be processed, specifically as follows:
traversing pixel points in the filling image, selecting a target pixel point, and determining the position of the target pixel point;
constructing a texture block with a preset size by taking the position of the target pixel point as a center;
calculating and obtaining the similarity between the target pixel point and each pixel point in the preset number of pixel points based on the positions of the pixel points belonging to the texture block and the positions of the preset number of pixel points selected on the image to be processed;
and determining the pixel point corresponding to the minimum similarity as a matching pixel point.
In this regard, it should be noted that, in the embodiment of the present invention, the pixel value of each pixel point in the filling image needs to be updated, and therefore, the pixel points in the filling image need to be traversed.
And selecting any pixel point as a target pixel point, and determining the position coordinate of the target pixel point. And constructing a texture block with a preset size (such as 7 x 7) by taking the position coordinates of the target pixel point as the coordinates of the central point.
Selecting a certain number of pixel points in the area to be repaired and the area nearby on the image to be processed, and determining the position coordinates of the pixel points. And calculating to obtain the similarity between the target pixel point and each pixel point in the preset number of pixel points based on the position coordinates of each pixel point belonging to the texture block and the position coordinates of a certain number of pixel points on the image to be processed.
And then selecting the pixel point corresponding to the determined minimum similarity as a matching pixel point.
Based on the process of S25-S27 in fig. 2, the following description proceeds with specific examples of the matching positions by taking the facial statutes as an example, as follows:
calendar above mask pattern M2-2And selecting the position (x, y) of any pixel point in the filling image in the area. In the initial state, selecting a face image S to be processed2The initial matching position (u, v) of (a), and (u, v) is (x, y).
At this time, the similarity s between each position and the position (u, v) on the texture block centered on the position (x, y) is calculated and is expressed as score, and score (x, y) is expressed as s. In this embodiment, the calculation process of the similarity is implemented by using a sum of squares and a distance SSD.
Then selecting a face image S to be processed2And (3) other positions adjacent to the initial matching position are updated (namely, adjacent matching is carried out, and the adjacent selected other positions are selected by moving up, down, left and right, such as values of (u-1, v), (u, v-1), (u +1, v) or (u, v +1)), then the similarity s1 between each position on the texture block with the position (x, y) as the center and the selected adjacent position is calculated, and if s1 is smaller than s, score (x, y) is updated to be s 1.
Selecting N adjacent positions according to a preset certain number N, then respectively calculating the similarity between each position on the texture block taking the position (x, y) as the center and the selected N adjacent positions, and after the calculation is finished, selecting score (x, y) as the minimum similarity.
And then determining the position of the pixel point corresponding to the minimum similarity as the finally obtained matching position.
The embodiment carries out the proximity matching search through the similarity calculation, realizes the acquisition of pixel information required by flaw repairing on the image, and improves the accurate effect for the subsequent flaw repairing treatment.
In a further embodiment of the method according to the above embodiment, after the repairing process is mainly performed on the image to be processed according to the filling image after the pixel value is updated, the method further includes the following subsequent steps:
and filtering the image to be processed after the flaw repairing processing.
The filtering processing can be realized by adopting Gaussian filtering, and is mainly characterized in that the image obtained by repairing the flaws of the image to be processed according to the filling image after the pixel value is updated is processed by adopting a Gaussian filtering mode, so that the local repair of the flaws is smoother.
Fig. 3 shows a schematic structural diagram of an image content restoration apparatus according to an embodiment of the present invention, and referring to fig. 3, the apparatus includes an acquisition module 31 and a processing module 32, where:
the acquiring module 31 is configured to acquire a mask layout of a to-be-repaired region mapped on an image to be processed, and determine a filling image of the to-be-repaired region in the mask layout of the to-be-repaired region;
and the processing module 32 is configured to repair the to-be-repaired area on the to-be-processed image according to the filling image.
In a further embodiment of the apparatus in the above embodiment, the processing module, in the process of repairing the region to be repaired on the image to be processed according to the filling image, is specifically configured to:
determining that pixel points in the filling image correspond to matching pixel points on the image to be processed, and updating pixel values of the pixel points in the filling image according to the pixel values of the matching pixel points;
and repairing the area to be repaired on the image to be processed according to the filling image after the pixel value is updated.
In a further embodiment of the apparatus in the above embodiment, the obtaining module, in the process of obtaining the mask layout of the region to be repaired mapped on the image to be processed, is specifically configured to:
deriving a standard Mongolian layout based on the marked region to be repaired in the standard image;
acquiring key point data of a standard image and an image to be processed, and determining an affine transformation matrix according to the key point data;
and mapping the standard mask graph by adopting the affine transformation matrix to obtain a mask graph of the area to be repaired.
In a further embodiment of the apparatus in the above embodiment, the obtaining module is specifically configured to, in a process of obtaining the key point data of the standard image and the image to be processed and determining the mapping transformation matrix according to the key point data:
performing triangulation processing according to the key point data of the standard image and the image to be processed to obtain a first grid corresponding to the standard image and a second grid corresponding to the image to be processed;
establishing an affine transformation matrix between triangular faces in the first mesh and corresponding triangular faces in the second mesh.
In a further embodiment of the apparatus in the above embodiment, the obtaining module, in the process of determining the filling image of the to-be-repaired region in the to-be-repaired region masking map, is specifically configured to:
and moving the mask layout of the area to be repaired in a preset direction on the image to be processed, and selecting the image corresponding to the position as a filling image of the area to be repaired based on the position of the area to be repaired in the mask layout of the area to be repaired in the image to be processed after translation.
In a further embodiment of the apparatus in the above embodiment, the processing module, in the process of determining that the pixel point in the filler image corresponds to the matching pixel point on the image to be processed, is specifically configured to:
traversing pixel points in the filling image, selecting target pixel points, and determining the positions of the target pixel points;
constructing a texture block with a preset size by taking the position of the target pixel point as a center;
calculating and obtaining the similarity between a target pixel point and each pixel point in a preset number of pixel points based on the positions of the pixel points in the texture block and the positions of the preset number of pixel points selected on the image to be processed;
and determining the pixel point corresponding to the minimum similarity as the matching pixel point.
In a further embodiment of the apparatus in the foregoing embodiment, the apparatus further includes an optimization module, configured to determine a minimum circumscribed rectangle of the to-be-repaired region in the to-be-repaired region masked map after obtaining the to-be-repaired region masked map mapped on the to-be-processed image, perform edge detection on the to-be-repaired region on the to-be-processed image based on the minimum circumscribed rectangle, determine the to-be-updated region, obtain, according to the to-be-updated region, an update mask map mapped on the to-be-processed image, and use the update mask map as a new to-be-repaired region masked map.
In a further embodiment of the apparatus in the foregoing embodiment, the apparatus further includes an adjusting module, configured to perform filtering processing on the to-be-processed image after performing the repairing processing on the to-be-processed image according to the filling image after the pixel value is updated.
Since the principle of the apparatus according to the embodiment of the present invention is the same as that of the method according to the above embodiment, further details are not described herein for further explanation.
It should be noted that, in the embodiment of the present invention, the relevant functional module may be implemented by a hardware processor (hardware processor).
The image content restoration device provided by the embodiment of the invention obtains the mask image of the area to be restored mapped on the image to be processed, and determines the filling image of the area to be restored in the mask image of the area to be restored, so that the filling image restores the area to be restored on the image to be processed, the pixel value on the filling image and the pixel value outside the area to be restored on the face image to be processed are reasonably fused, and the defect restoration effect on the image is better.
Fig. 4 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 4: a processor (processor)41, a communication Interface (communication Interface)42, a memory (memory)43 and a communication bus 44, wherein the processor 41, the communication Interface 42 and the memory 43 complete communication with each other through the communication bus 44. Processor 41 may call logic instructions in memory 43 to perform the following method: acquiring a to-be-repaired area masking layout mapped on an image to be processed, and determining a filling image of the to-be-repaired area in the to-be-repaired area masking layout; and repairing the area to be repaired on the image to be processed according to the filling image.
Furthermore, the logic instructions in the memory 43 may be implemented in the form of software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the method provided in the foregoing embodiments when executed by a processor, and the method includes: acquiring a to-be-repaired area masking layout mapped on an image to be processed, and determining a filling image of the to-be-repaired area in the to-be-repaired area masking layout; and repairing the area to be repaired on the image to be processed according to the filling image.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. An image content restoration method, comprising:
acquiring a to-be-repaired area masking layout mapped on an image to be processed, and determining a filling image of the to-be-repaired area in the to-be-repaired area masking layout;
and repairing the area to be repaired on the image to be processed according to the filling image.
2. The image content restoration method according to claim 1, wherein the obtaining of the mask map of the region to be restored mapped on the image to be processed includes:
deriving a standard Mongolian layout based on the marked region to be repaired in the standard image;
acquiring key point data of a standard image and an image to be processed, and determining an affine transformation matrix according to the key point data;
and mapping the standard mask graph by adopting the affine transformation matrix to obtain a mask graph of the area to be repaired.
3. The image content restoration method according to claim 2, wherein the obtaining of the key point data of the standard image and the image to be processed and the determining of the mapping transformation matrix according to the key point data comprises:
performing triangulation processing according to the key point data of the standard image and the image to be processed to obtain a first grid corresponding to the standard image and a second grid corresponding to the image to be processed;
establishing an affine transformation matrix between triangular faces in the first mesh and corresponding triangular faces in the second mesh.
4. The image content restoration method according to claim 1, wherein the determining of the filling image of the region to be restored in the masking layout of the region to be restored comprises:
and moving the masking layout of the area to be repaired in a preset direction on the image to be processed, and selecting the image corresponding to the position as a filling image of the area to be repaired based on the position of the area to be repaired on the image to be processed in the masking layout of the area to be repaired after translation.
5. The image content restoration method according to claim 1, wherein the restoring the area to be restored on the image to be processed according to the filling image comprises:
determining that pixel points in the filling image correspond to matching pixel points on the image to be processed, and updating pixel values of the pixel points in the filling image according to pixel values of the matching pixel points;
and repairing the area to be repaired on the image to be processed according to the filling image after the pixel value is updated.
6. The method according to claim 5, wherein said determining that the pixel points in the filler image correspond to matching pixel points on the image to be processed comprises:
traversing pixel points in the filling image, selecting target pixel points, and determining the positions of the target pixel points;
constructing a texture block with a preset size by taking the position of the target pixel point as a center;
calculating and obtaining the similarity between a target pixel point and each pixel point in a preset number of pixel points based on the positions of the pixel points in the texture block and the positions of the preset number of pixel points selected on the image to be processed;
and determining the pixel point corresponding to the minimum similarity as the matching pixel point.
7. The image content restoration method according to claim 1, wherein after acquiring the mask map of the area to be restored mapped on the image to be processed, the method further comprises:
determining a minimum external rectangle of the to-be-repaired area in the to-be-repaired area mask map, performing edge detection on the to-be-repaired area on the to-be-processed image based on the minimum external rectangle, determining an updating to-be-repaired area, acquiring the updating to-be-repaired mask map mapped on the to-be-processed image according to the updating to-be-repaired area, and using the updating to-be-repaired mask map as a new to-be-repaired area mask map.
8. The image content restoration method according to claim 1, wherein after restoring the area to be restored on the image to be processed according to the filler image after the pixel value update, the method further comprises: and carrying out filtering processing on the image to be processed after the restoration processing.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the image content repair method according to any one of claims 1 to 8 are implemented when the processor executes the program.
10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the image content restoration method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011196606.9A CN112348755A (en) | 2020-10-30 | 2020-10-30 | Image content restoration method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011196606.9A CN112348755A (en) | 2020-10-30 | 2020-10-30 | Image content restoration method, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112348755A true CN112348755A (en) | 2021-02-09 |
Family
ID=74355276
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011196606.9A Pending CN112348755A (en) | 2020-10-30 | 2020-10-30 | Image content restoration method, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112348755A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113222981A (en) * | 2021-06-01 | 2021-08-06 | 山东贝特建筑项目管理咨询有限公司 | Processing method and system for anchor bolt image in heat-insulation board |
| CN113469923A (en) * | 2021-05-28 | 2021-10-01 | 北京达佳互联信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105741231A (en) * | 2016-02-02 | 2016-07-06 | 深圳中博网络技术有限公司 | Skin beautifying processing method and device of image |
| US20180005448A1 (en) * | 2016-06-30 | 2018-01-04 | Fittingbox | Method of hiding an object in an image or video and associated augmented reality process |
| CN111311521A (en) * | 2020-03-12 | 2020-06-19 | 京东方科技集团股份有限公司 | Image restoration method and device and electronic equipment |
| CN111462007A (en) * | 2020-03-31 | 2020-07-28 | 北京百度网讯科技有限公司 | Image processing method, device, equipment and computer storage medium |
| CN111836058A (en) * | 2019-04-22 | 2020-10-27 | 腾讯科技（深圳）有限公司 | Method, device and equipment for real-time video playing and storage medium |
-
2020
- 2020-10-30 CN CN202011196606.9A patent/CN112348755A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105741231A (en) * | 2016-02-02 | 2016-07-06 | 深圳中博网络技术有限公司 | Skin beautifying processing method and device of image |
| US20180005448A1 (en) * | 2016-06-30 | 2018-01-04 | Fittingbox | Method of hiding an object in an image or video and associated augmented reality process |
| CN111836058A (en) * | 2019-04-22 | 2020-10-27 | 腾讯科技（深圳）有限公司 | Method, device and equipment for real-time video playing and storage medium |
| CN111311521A (en) * | 2020-03-12 | 2020-06-19 | 京东方科技集团股份有限公司 | Image restoration method and device and electronic equipment |
| CN111462007A (en) * | 2020-03-31 | 2020-07-28 | 北京百度网讯科技有限公司 | Image processing method, device, equipment and computer storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 沈焕锋 等: "ENVI遥感影像处理方法", 武汉大学出版社, pages: 419 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113469923A (en) * | 2021-05-28 | 2021-10-01 | 北京达佳互联信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN113469923B (en) * | 2021-05-28 | 2024-05-24 | 北京达佳互联信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN113222981A (en) * | 2021-06-01 | 2021-08-06 | 山东贝特建筑项目管理咨询有限公司 | Processing method and system for anchor bolt image in heat-insulation board |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109859098B (en) | Face image fusion method and device, computer equipment and readable storage medium | |
| CN109697688B (en) | Method and device for image processing | |
| CN107507217B (en) | Method and device for making certificate photo and storage medium | |
| CN107610202B (en) | Face image replacement method, device and storage medium | |
| CN112348755A (en) | Image content restoration method, electronic device and storage medium | |
| KR100810326B1 (en) | Method for generation of multi-resolution 3d model | |
| JP3819662B2 (en) | Three-dimensional modeling apparatus, method and medium, and tertiary shape data recording apparatus, method and medium | |
| EP4174785A1 (en) | Method and apparatus for constructing landform map, electronic device, and readable storage medium | |
| CN116805356A (en) | Building model construction method, building model construction equipment and computer readable storage medium | |
| Wei et al. | Domain-based structure-aware image inpainting | |
| JP5295044B2 (en) | Method and program for extracting mask image and method and program for constructing voxel data | |
| CN114782645A (en) | Virtual digital person making method, related equipment and readable storage medium | |
| KR102327304B1 (en) | A method of improving the quality of 3D images acquired from RGB-depth camera | |
| CN111524171B (en) | Image processing method and device and electronic equipment | |
| CN106934846B (en) | Cloth image processing method and system | |
| CN116342519A (en) | Image processing method based on machine learning | |
| KR20240044495A (en) | Inspection support system, inspection support method, and inspection support program | |
| US20210241430A1 (en) | Methods, devices, and computer program products for improved 3d mesh texturing | |
| Huang et al. | A patch-based image inpainting based on structure consistence | |
| CN111145204B (en) | Polygonal simplifying method for profile curve with settable edge number | |
| CN113724373A (en) | Modeling method and device of GIS (geographic information System) equipment, computer equipment and storage medium | |
| CN112052859A (en) | License plate accurate positioning method and device in free scene | |
| US8363941B2 (en) | Method and program for extracting silhouette image and method and program for constructing three dimensional model | |
| CN110490877A (en) | Binocular stereo image based on Graph Cuts is to Target Segmentation method | |
| CN110751078B (en) | Method and equipment for determining non-skin color region of three-dimensional face |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |