CN116934637A - Photo processing method, device and equipment - Google Patents
Photo processing method, device and equipment Download PDFInfo
- Publication number
- CN116934637A CN116934637A CN202210355149.6A CN202210355149A CN116934637A CN 116934637 A CN116934637 A CN 116934637A CN 202210355149 A CN202210355149 A CN 202210355149A CN 116934637 A CN116934637 A CN 116934637A
- Authority
- CN
- China
- Prior art keywords
- picture
- transformation matrix
- processing
- resolution
- photo
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 89
- 238000005457 optimization Methods 0.000 claims abstract description 84
- 230000001131 transforming effect Effects 0.000 claims abstract description 24
- 230000004927 fusion Effects 0.000 claims abstract description 14
- 230000009466 transformation Effects 0.000 claims description 195
- 239000011159 matrix material Substances 0.000 claims description 135
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 70
- 238000000034 method Methods 0.000 claims description 49
- 238000004422 calculation algorithm Methods 0.000 claims description 43
- 238000007499 fusion processing Methods 0.000 claims description 11
- 238000013519 translation Methods 0.000 description 20
- 230000008569 process Effects 0.000 description 16
- 238000011282 treatment Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000002085 persistent effect Effects 0.000 description 6
- 238000000354 decomposition reaction Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 239000002355 dual-layer Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The application relates to a photo processing method, a photo processing device and photo processing equipment. The photo processing method comprises the following steps: transforming the initial picture into an intermediate picture with unchanged resolution; respectively carrying out definition optimization processing on the intermediate picture according to the portrait area and the background area; fusing the portrait area and the background area which are subjected to definition optimization respectively to obtain a fused picture; and transforming the fusion picture into a picture with specified specification. The scheme provided by the application can improve the definition of the photo.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method, an apparatus, and a device for processing a photograph.
Background
In daily life, people often use credentials. Credentials are photographs used to prove identity, and generally have high requirements for clarity of the photograph. There are various methods for generating and processing credentials in the related art, generally scaling and transforming the photos to a specified specification, and then performing a related optimization process.
However, in the related art photo processing method, since the photo is scaled to a specified specification and then optimized, the definition of the photo is inevitably degraded, thereby affecting the definition of the photo.
Disclosure of Invention
In order to solve or partially solve the problems in the related art, the application provides a photo processing method, a photo processing device and photo processing equipment, which can improve the definition of a photo.
The first aspect of the present application provides a photo processing method, including:
transforming the initial picture into an intermediate picture with unchanged resolution;
respectively carrying out definition optimization processing on the intermediate picture according to the portrait area and the background area;
fusing the portrait area and the background area which are subjected to definition optimization respectively to obtain a fused picture;
and transforming the fusion picture into a picture with specified specification.
In an embodiment, the transforming the initial photograph into an intermediate picture with unchanged resolution includes:
the initial picture is transformed into an intermediate picture of constant resolution by a first affine transformation matrix.
In one embodiment, the first affine transformation matrix is obtained as follows:
aligning the initial photo to a reference picture to obtain transformation parameters of position coordinates;
and constructing a transformation matrix according to the transformation parameters, and setting the transformation matrix to obtain a first affine transformation matrix.
In an embodiment, after the initial photo is transformed into the intermediate picture with unchanged resolution through the first affine transformation matrix, if the resolution of the picture is greater than or equal to a set threshold, a first interpolation method is adopted to perform interpolation, and if the resolution of the picture is less than the set threshold, a second interpolation method is adopted to perform interpolation.
In an embodiment, the performing sharpness optimization processing on the intermediate picture according to the image area and the background area respectively includes:
carrying out definition optimization processing on the portrait area of the intermediate picture according to a first optimization algorithm;
and carrying out definition optimization processing on the background area of the intermediate picture according to a second optimization algorithm.
In an embodiment, the transforming the fused picture into a specified specification picture includes:
transforming the fused picture into a picture with a specified specification through a second affine transformation matrix;
wherein the second affine transformation matrix is obtained as follows: aligning the intermediate picture to the specified specification picture to obtain a transformation parameter of a position coordinate; and constructing a transformation matrix according to the transformation parameters, and taking the transformation matrix as a second affine transformation matrix.
A second aspect of the present application provides a photograph processing apparatus comprising:
the first transformation module is used for transforming the initial photo into an intermediate picture with unchanged resolution;
the optimizing processing module is used for respectively carrying out definition optimizing processing on the intermediate picture transformed by the first transforming module according to the image area and the background area;
the fusion processing module is used for fusing the portrait area and the background area which are subjected to definition optimization processing by the optimization processing module respectively to obtain a fused picture;
and the second transformation module is used for transforming the fusion picture obtained by the fusion processing module into a picture with specified specification.
In an embodiment, the first transformation module transforms the initial photograph into an intermediate picture with unchanged resolution through a first affine transformation matrix; or alternatively, the first and second heat exchangers may be,
the second transformation module transforms the fused picture into a specified specification picture through a second affine transformation matrix.
A third aspect of the present application provides an electronic apparatus, comprising:
a processor; and
a memory having executable code stored thereon which, when executed by the processor, causes the processor to perform the method as described above.
A fourth aspect of the application provides a computer readable storage medium having stored thereon executable code which, when executed by a processor of an electronic device, causes the processor to perform a method as described above.
The technical scheme provided by the application can comprise the following beneficial effects:
in the related art, the initial photo is subjected to scaling post-processing, and the scaling processing leads to the loss of definition in the photo processing flow, but the technical scheme of the application converts the initial photo into the intermediate picture with unchanged resolution when converting the initial photo into the intermediate picture, so that the loss of definition in the photo processing flow can be reduced or avoided. In addition, the application respectively carries out definition optimization processing on the intermediate picture according to the image area and the background area, and then fuses the image area and the background area which are respectively subjected to the definition optimization processing to obtain a fused picture, so that different definition optimization processing can be carried out according to the image characteristics of different areas in the picture, and the definition loss in the photo processing flow is further reduced or avoided. Therefore, the technical scheme of the application can improve the definition of the photo through optimization processing.
Furthermore, according to the technical scheme, the initial photo can be transformed into the intermediate picture with unchanged resolution through the first affine transformation matrix, and the fused picture is transformed into the picture with the specified specification through the second affine transformation matrix. By using two affine transformations, sharpness loss in the photo processing flow can also be reduced.
Further, according to the technical scheme, the image area of the intermediate picture can be subjected to definition optimization according to a first optimization algorithm; and carrying out definition optimization processing on the background area of the intermediate picture according to a second optimization algorithm. Because the portrait area is subjected to independent optimization treatment, the definition of the portrait area can be improved; and other background areas except the portrait area are independently optimized by adopting other more matched algorithms, so that the definition of the background area can be improved. In this way, the sharpness of the entire photograph is higher than that obtained by the related art process.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.
Drawings
The foregoing and other objects, features and advantages of the application will be apparent from the following more particular descriptions of exemplary embodiments of the application as illustrated in the accompanying drawings wherein like reference numbers generally represent like parts throughout the exemplary embodiments of the application.
FIG. 1 is a flow chart of a photo processing method according to an embodiment of the present application;
FIG. 2 is another flow chart of a photo processing method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a method application corresponding to FIG. 2, according to an embodiment of the present application;
FIG. 4 is a schematic diagram showing a process of determining an affine transformation matrix according to an embodiment of the application;
fig. 5 is a schematic diagram of a structure of a photograph processing apparatus shown in an embodiment of the present application;
FIG. 6 is another schematic view of a photograph processing apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While embodiments of the present application are illustrated in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms "first," "second," "third," etc. may be used herein to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the application. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present application, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
The photo processing method in the related art may cause a decrease in the definition of the picture, thereby affecting the definition of the photo. In view of the above problems, embodiments of the present application provide a photo processing method, which can improve the definition of a photo.
The following describes the technical scheme of the embodiment of the present application in detail with reference to the accompanying drawings.
Fig. 1 is a flow chart of a photo processing method according to an embodiment of the present application.
Referring to fig. 1, the method includes:
in S101, the initial photograph is converted into an intermediate picture of constant resolution.
This step S101 may transform the initial picture into an intermediate picture of unchanged resolution by means of a first affine transformation matrix. The initial photograph may be, for example, a certificate photograph, but is not limited thereto. Wherein the first affine transformation matrix can be obtained as follows: aligning the initial photo to a reference picture to obtain transformation parameters of position coordinates; and constructing a transformation matrix according to the transformation parameters, and setting the transformation matrix to obtain a first affine transformation matrix. Wherein, a translation transformation matrix T, a rotation transformation matrix R and a scaling transformation matrix S can be constructed according to transformation parameters; performing a left multiplication operation on the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S to obtain a transformation matrix; and performing decomposition operation on the transformation matrix, and modifying the scaling coefficient to be 1 to obtain a first affine transformation matrix. Since the first affine transformation matrix is modified to have the scaling coefficient of 1, the resolution of the photo after the transformation is unchanged, and the loss of definition of the picture can be reduced.
In S102, the intermediate picture is subjected to sharpness optimization processing in terms of the portrait area and the background area, respectively.
Step S102 can perform definition optimization processing on the portrait area of the intermediate picture according to a first optimization algorithm; and carrying out definition optimization processing on the background area of the intermediate picture according to a second optimization algorithm. The first optimization algorithm may be a face blind restoration algorithm; the second optimization algorithm may be a picture super resolution algorithm. According to different characteristics of different areas of the picture, different optimization processing modes are adopted respectively, so that the definition of the picture can be further improved.
In S103, the portrait area and the background area after the definition optimization processing are fused, and a fused picture is obtained.
In the step S103, the image area and the background area after the definition optimization process are respectively fused by using a poisson fusion algorithm, so as to obtain a fused picture.
In S104, the fused picture is converted into a specified specification picture.
This step S104 may transform the fused picture into a specified specification picture through a second affine transformation matrix. Wherein the second affine transformation matrix can be obtained as follows: aligning the intermediate picture to a picture with a specified specification to obtain a transformation parameter of a position coordinate; and constructing a transformation matrix according to the transformation parameters, and taking the transformation matrix as a second affine transformation matrix. Wherein a translation transformation matrix T, a rotation transformation matrix R and a scaling transformation matrix S can be constructed according to transformation parameters; and carrying out left multiplication operation on the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S to obtain a transformation matrix, and taking the transformation matrix as a second affine transformation matrix.
As can be seen from this embodiment, in the related art, the initial photo is first scaled and then processed, and the scaling processing may result in loss of definition in the photo processing flow, but the technical solution of the present application transforms the initial photo into an intermediate picture, when transforming the initial photo into an intermediate picture, into an intermediate picture with unchanged resolution, so that loss of definition in the photo processing flow can be reduced or avoided. In addition, the application respectively carries out definition optimization treatment on the intermediate picture according to the image area and the background area, and then fuses the image area and the background area which are respectively subjected to the definition optimization treatment to obtain a fused picture, so that different definition optimization treatments can be carried out according to the image characteristics of different areas in the picture, and the definition loss in the photo processing flow is further reduced or avoided. Therefore, the technical scheme of the embodiment of the application can improve the definition of the photo through optimization processing.
Fig. 2 is another flow chart of a photo processing method according to an embodiment of the present application.
According to the technical scheme provided by the embodiment of the application, the initial picture is firstly transformed into an intermediate picture through the first affine transformation matrix. Because the first affine transformation matrix has modified the scaling factor to be 1, the resolution of the photo after transformation is not changed (compared with the original picture of the original photo, the resolution of the region which corresponds to the original picture is not changed, the whole picture involves rotation and cutting, and the size can be changed to a certain extent), but the rotation and translation correction is carried out; and respectively carrying out definition optimization processing on the intermediate picture according to the portrait region and the background region to obtain a fused picture, and finally transforming the fused picture to a specified specification size through a second affine transformation matrix. Through the processing, the problem of loss of definition in the processing process is reduced, the effect of optimizing definition is improved, and the overall definition of the photo is better optimized.
Referring to fig. 2 and 3, the method includes:
in S201, the initial photograph is transformed into an intermediate picture of unchanged resolution by a first affine transformation matrix.
Taking the initial photograph as a portrait photograph and the reference photograph as a frontal reference photograph as an example, a process for determining the first affine transformation matrix is described, which (see fig. 4) includes:
1) According to the face key points of the face picture and the face key points of the face reference picture, obtaining a transformation matrix tfm
1-1) aligning the initial photograph to a reference map to obtain transformation parameters of the position coordinates.
Assuming that the position coordinates of the face key points of the face image are (x, y), and the position coordinates of the face key points of the front face reference image are (x ', y'), after the face key points of the face image are paired with the face key points of the Ji Zheng face reference image, the transformation parameters for transforming the position coordinates of the face key points of the face image into the position coordinates of the face key points of the front face reference image can be calculated. The alignment includes translation, rotation, scaling and the like of the portrait pictures. Wherein the transformation parameters include translation parameters tx and ty for performing a translation process, rotation parameters θ for performing a rotation process, and scaling coefficients sx and sy for performing a scaling process.
The key points of the face may be a set number of key feature points of the face, for example, 3 key feature points or more, for example, 5 key feature points of left eye, right eye, nose, left mouth corner, right mouth corner may be selected, but not limited thereto.
1-2) constructing a translation transformation matrix T, a rotation transformation matrix R and a scaling transformation matrix S according to transformation parameters:
wherein T, R, S respectively represents a translation transformation matrix, a rotation transformation matrix and a scaling transformation matrix, t represents a parameter of translation, s represents a scaling coefficient, θ represents a rotation angle, and subscripts x and y represent corresponding components of a coordinate position in an x axis and a y axis.
Wherein translation is to move each point to (x+t, y+t); rotation is a clockwise rotation of an angle θ around the origin, scaling is an increase or decrease by sx times and an increase (decrease) by sy times in the abscissa of each point.
1-3) multiplying the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S to obtain a transformation matrix tfm:
equation (2) shows that R transformation is performed first, then S transformation is performed, and then T transformation is performed. That is, the transformation sequence is to make the following transformation in the formula and then make the preceding transformation, and the matrix multiplication is the left multiplication. Matrix multiplication by left is understood to mean the multiplication of a matrix by the left of the multiplication number.
The transformation relationship between the face key points of the face reference picture and the face key points of the portrait picture can be:
it can be seen that by the alignment process, the transformation matrix tfm for aligning the portrait picture to the frontal face reference picture can be calculated.
2) And (3) performing decomposition operation on the transformation matrix tfm, and modifying the scaling coefficient to be 1 to obtain a first affine transformation matrix tfm_stage1.
The transformation matrix tfm is decomposed in the process of S.tfm_stag1. In theory, the transformation matrix tfm can be decomposed into any number of terms, for example tfm=tfm_n·tfm_n-1·. In the technical scheme of the application, the decomposition is carried out by firstly carrying out the transformation of tfm_stag1 and then carrying out the S transformation, so that the transformation matrix tfm is equivalent to the transformation matrix tfm. Since the S transform is known, tfm_stag1 can be decomposed.
the decomposition operation of tfm is as follows:
the calculation formulas of the scaling coefficients sx and sy in the rotation matrix S may be as follows:
then, the scaling factor is modified to 1, i.e., S is modified to 1.tfm_stage1 by 1, then the diagonal matrix corresponds to no multiplication. That is, the scaling factor is 1, that is, sx and sy in the rotation transformation matrix S are changed to 1, which corresponds to not performing S transformation, and the rest is tfm_stage1.
Thus, the first affine transformation matrix required to get the transformation to the intermediate picture is as follows:
3) The portrait picture is transformed into an intermediate picture with unchanged resolution through a first affine transformation matrix tfm_stag1.
Since the scaling factor in the first affine transformation matrix tfm_stag1 is modified to 1, which is equivalent to that without S transformation, the resolution of the transformed portrait picture is not changed, and finally the portrait picture is transformed into an intermediate picture with unchanged resolution through the rotation and translation processing of the first affine transformation matrix tfm_stag1.
After the initial photo is converted into an intermediate picture with unchanged resolution through the first affine transformation matrix (compared with the original picture of the initial photo, the resolution of the region corresponding to the original picture is unchanged, the whole picture is rotated and cut, and the size of the whole picture is possibly changed to a certain extent), if the resolution of the picture after conversion (for example, after rotation cutting) is greater than or equal to a set threshold value, interpolation is performed by adopting a first interpolation method, and if the resolution of the picture is less than the set threshold value, interpolation is performed by adopting a second interpolation method. Wherein the first interpolation may be, for example, INTER-CUBIC interpolation (CUBIC spline interpolation); the second interpolation method may be, for example, an inter_area (AREA interpolation) interpolation method. For pictures with larger resolution, the inter_cubic interpolation method has better scaling effect, and for the pictures with smaller resolution obtained by compressing faces, the inter_area interpolation method can avoid granular sensation caused by inter_cubic.
Implementing affine transformation using opencv (a cross-platform computer vision and machine learning software library issued based on Apache2.0 license (open source)) typically involves a warp Affine function that can implement some simple remapping. According to the scheme, the use mode of the warp Affine function is modified, and after the warp Affine function is used for carrying out rotation and translation transformation on the portrait picture, the picture is scaled. For a picture with larger resolution, for example, if the short side of the picture is greater than or equal to 260, interpolation is performed by using an inter_unit interpolation method, and if the short side of the picture is less than 260, interpolation is performed by using an inter_area interpolation method.
Since the application modifies the scaling factor of the first affine transformation matrix tfm_stage1 to 1, the modification of the use of the warp function does not affect the use of the first affine transformation matrix tfm_stage1, but improves the effect of the second affine transformation matrix tfm_stage2 after application.
In S202, the image area of the intermediate picture is subjected to sharpness optimization processing according to a first optimization algorithm.
The human face is a region of comparative attention in the picture, which is also called a portrait region, and the embodiment of the application independently enhances the definition of the human face. In the step S202, the image area of the intermediate picture is subjected to sharpness optimization processing according to a first optimization algorithm, which may be, for example, a face blind restoration algorithm.
Face blind restoration refers to reconstructing a degraded (low-resolution, noise, blurring, picture compression and the like) face picture, and finally obtaining a less-degraded and clear face picture. Face blind restoration is to restore a face mainly by means of geometric face prior information (key points, face segmentation and face heat map). The GFPGAN is the most advanced model of a face blind restoration algorithm, and the generated face prior is used for face blind restoration. The application may use GFPGAN models for face sharpness optimization but is not limited thereto.
In S203, the background area of the intermediate picture is subjected to sharpness optimization processing according to a second optimization algorithm.
In the embodiment of the application, because the face area is independently processed and optimized, the step is mainly to enhance the definition of the background except the face. The application can carry out definition optimization processing on the background area of the intermediate picture according to the second optimization algorithm. The second optimization algorithm may be, for example, a picture super-resolution algorithm or the like. The present application uses a super resolution algorithm model, such as the RealESRGAN model, for background sharpness optimization. The RealESRGAN model is mainly used for simulating various degradation processes in the low resolution process of a high resolution image, and then the model is used for viewing a paste image and then is used for deducing a high definition image of the paste image.
The step S203 and the step S202 may be performed separately and simultaneously, and there is no sequential relationship between the two.
It should be noted that, by using the super-resolution algorithm of the picture, the picture is transformed from low resolution to high resolution, the information contained in the picture is changed from small to large, and if the information contained in the original picture is missing, the model after the super-resolution processing is finally affected after the definition is reduced. For example, the original picture is 1024x1024, and the picture is directly processed with super resolution without scaling to obtain a 2048x2048 picture A; scaling the original picture into a 512x512 picture by adopting a related technology, and performing super-resolution processing on the scaled picture to obtain a 2048x2048 picture B; since the photo B is subjected to the over-scaling treatment, the definition is reduced, and then the photo A obtained by adopting the scheme of the application can have better definition than the photo B obtained by adopting the related technology. The same holds true for the face blind restoration algorithm.
It should be noted that, if other portrait processing logic is required to be performed on the intermediate picture according to different service requirements, corresponding processing may also be performed on the intermediate picture, which is not limited in the embodiment of the present application.
In S204, the portrait area and the background area after the definition optimization process are fused by using a poisson fusion algorithm, so as to obtain a fused picture.
The application applies a poisson fusion algorithm to paste the face with enhanced definition back to the picture with enhanced background definition, thereby completing the picture definition enhancement and obtaining the processed fusion picture.
The image fusion is a key technology in the image stitching technology, and the principle is that smooth transition and seamless stitching between stitched images are realized by redefining and calculating pixels of overlapping bands in the stitched images. The poisson fusion algorithm based on poisson equation utilizes the gradient fields of two images to conduct guiding difference value on the overlapping area, the image fusion problem is changed into the problem of solving the minimization of the difference value between the gradient field of the target image block and the background guiding gradient field, and good image fusion effect can be obtained.
In S205, the fused picture is transformed into a specified specification picture by the second affine transformation matrix.
This step determines a second affine transformation matrix tfm_stage2 with intermediate pictures aligned to the specified specification pictures.
The principle of the determining process of the second affine transformation matrix tfm_stage2 and the first affine transformation matrix tfm_stage1 is the same (see fig. 4), and the transformation matrix tfm can be obtained according to the face key points of the intermediate picture and the face key points of the specified specification picture. However, in this case, the transformation matrix tfm does not need to be decomposed, the scaling factor does not need to be modified to 1, and the obtained transformation matrix tfm is only required to be directly used as the second affine transformation matrix tfm_stag2.
In step S205, the fused picture is transformed into a picture with a specified specification through the second affine transformation matrix, that is, affine transformation is applied to the fused picture obtained after the fusion processing, so as to obtain a final picture with a specified specification.
In summary, according to the technical scheme of the embodiment of the application, the scaling coefficient of the affine transformation matrix is modified to be 1, so that the resolution of the picture is not changed after the initial picture is subjected to rotation and translation correction, and the interpolation method used for scaling in the affine transformation process is specified, so that the sharpness loss in the picture processing flow can be reduced. Although the picture after the second affine transformation is adopted finally, the picture after the first affine transformation is used before the last step, and the intermediate picture and the initial picture keep the same resolution, so that the loss of definition in the processing process is avoided, and the definition of the whole picture after the processing is improved.
Corresponding to the embodiment of the application function implementation method, the embodiment of the application also provides a photo processing device, electronic equipment and corresponding embodiments.
Fig. 5 is a schematic diagram of a photograph processing apparatus according to an embodiment of the present application.
Referring to fig. 5, a photograph processing apparatus 50 includes: a first transformation module 51, an optimization processing module 52, a fusion processing module 53, a second transformation module 54.
The first transformation module 51 is configured to transform the initial picture into an intermediate picture with a constant resolution. The first transformation module 51 transforms the initial picture into an intermediate picture of unchanged resolution by means of a first affine transformation matrix. Wherein the first affine transformation matrix can be obtained as follows: aligning the initial photo to a reference picture to obtain transformation parameters of position coordinates; and constructing a transformation matrix according to the transformation parameters, and setting the transformation matrix to obtain a first affine transformation matrix. Wherein, a translation transformation matrix T, a rotation transformation matrix R and a scaling transformation matrix S can be constructed according to transformation parameters; performing a left multiplication operation on the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S to obtain a transformation matrix; and performing decomposition operation on the transformation matrix, and modifying the scaling coefficient to be 1 to obtain a first affine transformation matrix.
The optimizing processing module 52 is configured to perform sharpness optimization processing on the intermediate picture transformed by the first transforming module 51 according to the image area and the background area, respectively. The optimization processing module 52 may perform sharpness optimization processing on the portrait area of the intermediate picture according to a first optimization algorithm; and carrying out definition optimization processing on the background area of the intermediate picture according to a second optimization algorithm. The first optimization algorithm may be a face blind restoration algorithm; the second optimization algorithm may be a picture super resolution algorithm.
And the fusion processing module 53 is configured to fuse the portrait area and the background area after the definition optimization processing by the optimization processing module 52 respectively, so as to obtain a fused picture. The fusion processing module 53 may fuse the portrait area and the background area after the definition optimization processing respectively by using a poisson fusion algorithm to obtain a fused picture.
The second transformation module 54 is configured to transform the fused picture obtained by the fusion processing module 53 into a specified specification picture. The second transformation module 54 may transform the fused picture into a specified specification picture through a second affine transformation matrix. Wherein the second affine transformation matrix can be obtained as follows: aligning the intermediate picture to a picture with a specified specification to obtain a transformation parameter of a position coordinate; and constructing a transformation matrix according to the transformation parameters, and taking the transformation matrix as a second affine transformation matrix. Wherein a translation transformation matrix T, a rotation transformation matrix R and a scaling transformation matrix S can be constructed according to transformation parameters; and carrying out left multiplication operation on the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S to obtain a transformation matrix, and taking the transformation matrix as a second affine transformation matrix.
As can be seen from this embodiment, in the related art, the initial photo is first scaled and then processed, and the scaling processing may result in loss of definition in the photo processing flow, but the technical solution of the present application transforms the initial photo into an intermediate picture, when transforming the initial photo into an intermediate picture, into an intermediate picture with unchanged resolution, so that loss of definition in the photo processing flow can be reduced or avoided. In addition, the application respectively carries out definition optimization treatment on the intermediate picture according to the image area and the background area, and then fuses the image area and the background area which are respectively subjected to the definition optimization treatment to obtain a fused picture, so that different definition optimization treatments can be carried out according to the image characteristics of different areas in the picture, and the definition loss in the photo processing flow is further reduced or avoided. Therefore, the technical scheme of the application can improve the definition of the photo through optimization processing.
Fig. 6 is another schematic structural view of a photograph processing apparatus according to an embodiment of the present application.
Referring to fig. 6, a photograph processing apparatus 50 includes: a first transformation module 51, an optimization processing module 52, a fusion processing module 53, a second transformation module 54.
The functions of the first transformation module 51, the optimization processing module 52, the fusion processing module 53, and the second transformation module 54 may be described with reference to fig. 5.
Wherein the first transformation module 51 may transform the initial photograph into an intermediate picture of unchanged resolution through a first affine transformation matrix; the second transformation module 54 may transform the fused picture into a specified specification picture through a second affine transformation matrix.
The optimization processing module 52 may include: a first optimization sub-module 521, a second optimization sub-module 522.
The first optimizing sub-module 521 is configured to perform sharpness optimization processing on the portrait area of the intermediate picture according to a first optimizing algorithm;
and the second optimizing sub-module 522 is configured to perform sharpness optimization processing on the background area of the intermediate picture according to a second optimizing algorithm.
The first optimization algorithm may be a face blind restoration algorithm; the second optimization algorithm may be a picture super resolution algorithm. GFPGAN is the most advanced model for face blind restoration algorithms, which uses the generated face priors for face blind restoration. The present application may use GFPGAN for face definition optimization but is not limited thereto. The present application may use a super resolution algorithm model, such as the RealESRGAN model, for background sharpness optimization.
The specific manner in which the respective modules perform the operations in the apparatus of the above embodiments has been described in detail in the embodiments related to the method, and will not be described in detail herein.
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Referring to fig. 7, the electronic device 1000 includes a memory 1010 and a processor 1020.
The processor 1020 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Memory 1010 may include various types of storage units, such as system memory, read Only Memory (ROM), and persistent storage. Where the ROM may store static data or instructions that are required by the processor 1020 or other modules of the computer. The persistent storage may be a readable and writable storage. The persistent storage may be a non-volatile memory device that does not lose stored instructions and data even after the computer is powered down. In some embodiments, the persistent storage device employs a mass storage device (e.g., magnetic or optical disk, flash memory) as the persistent storage device. In other embodiments, the persistent storage may be a removable storage device (e.g., diskette, optical drive). The system memory may be a read-write memory device or a volatile read-write memory device, such as dynamic random access memory. The system memory may store instructions and data that are required by some or all of the processors at runtime. Furthermore, memory 1010 may comprise any combination of computer-readable storage media including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks, and/or optical disks may also be employed. In some implementations, memory 1010 may include readable and/or writable removable storage devices such as Compact Discs (CDs), digital versatile discs (e.g., DVD-ROMs, dual-layer DVD-ROMs), blu-ray discs read only, super-density discs, flash memory cards (e.g., SD cards, min SD cards, micro-SD cards, etc.), magnetic floppy disks, and the like. The computer readable storage medium does not contain a carrier wave or an instantaneous electronic signal transmitted by wireless or wired transmission.
The memory 1010 has stored thereon executable code that, when processed by the processor 1020, can cause the processor 1020 to perform some or all of the methods described above.
Furthermore, the method according to the application may also be implemented as a computer program or computer program product comprising computer program code instructions for performing part or all of the steps of the above-described method of the application.
Alternatively, the application may also be embodied as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) having stored thereon executable code (or a computer program or computer instruction code) which, when executed by a processor of an electronic device (or a server, etc.), causes the processor to perform part or all of the steps of the above-described method according to the application.
The foregoing description of embodiments of the application has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the improvement of technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (10)
1. A photo processing method, comprising:
transforming the initial picture into an intermediate picture with unchanged resolution;
respectively carrying out definition optimization processing on the intermediate picture according to the portrait area and the background area;
fusing the portrait area and the background area which are subjected to definition optimization respectively to obtain a fused picture;
and transforming the fusion picture into a picture with specified specification.
2. The method of claim 1, wherein said transforming the initial photograph into an intermediate picture of constant resolution comprises:
the initial picture is transformed into an intermediate picture of constant resolution by a first affine transformation matrix.
3. The method according to claim 2, wherein the first affine transformation matrix is obtained as follows:
aligning the initial photo to a reference picture to obtain transformation parameters of position coordinates;
and constructing a transformation matrix according to the transformation parameters, and setting the transformation matrix to obtain a first affine transformation matrix.
4. The method according to claim 2, characterized in that:
after the initial photo is converted into an intermediate picture with unchanged resolution through the first affine transformation matrix, if the resolution of the picture is larger than or equal to a set threshold value, a first interpolation method is adopted for interpolation, and if the resolution of the picture is smaller than the set threshold value, a second interpolation method is adopted for interpolation.
5. The method according to claim 1, wherein the performing sharpness optimization processing on the intermediate picture according to the portrait area and the background area respectively includes:
carrying out definition optimization processing on the portrait area of the intermediate picture according to a first optimization algorithm;
and carrying out definition optimization processing on the background area of the intermediate picture according to a second optimization algorithm.
6. The method of claim 1, wherein said transforming the fused picture into a specified specification picture comprises:
transforming the fused picture into a picture with a specified specification through a second affine transformation matrix;
wherein the second affine transformation matrix is obtained as follows: aligning the intermediate picture to the specified specification picture to obtain a transformation parameter of a position coordinate; and constructing a transformation matrix according to the transformation parameters, and taking the transformation matrix as a second affine transformation matrix.
7. A photograph processing apparatus, comprising:
the first transformation module is used for transforming the initial photo into an intermediate picture with unchanged resolution;
the optimizing processing module is used for respectively carrying out definition optimizing processing on the intermediate picture transformed by the first transforming module according to the image area and the background area;
the fusion processing module is used for fusing the portrait area and the background area which are subjected to definition optimization processing by the optimization processing module respectively to obtain a fused picture;
and the second transformation module is used for transforming the fusion picture obtained by the fusion processing module into a picture with specified specification.
8. The apparatus according to claim 7, wherein:
the first transformation module transforms the initial photo into an intermediate picture with unchanged resolution through a first affine transformation matrix; or alternatively, the first and second heat exchangers may be,
the second transformation module transforms the fused picture into a specified specification picture through a second affine transformation matrix.
9. An electronic device, comprising:
a processor; and
a memory having executable code stored thereon, which when executed by the processor, causes the processor to perform the method of any of claims 1-6.
10. A computer readable storage medium having stored thereon executable code which when executed by a processor of an electronic device causes the processor to perform the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210355149.6A CN116934637A (en) | 2022-03-31 | 2022-03-31 | Photo processing method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210355149.6A CN116934637A (en) | 2022-03-31 | 2022-03-31 | Photo processing method, device and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116934637A true CN116934637A (en) | 2023-10-24 |
Family
ID=88374305
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210355149.6A Pending CN116934637A (en) | 2022-03-31 | 2022-03-31 | Photo processing method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116934637A (en) |
-
2022
- 2022-03-31 CN CN202210355149.6A patent/CN116934637A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10762606B2 (en) | Image processing apparatus and method for generating high quality image | |
| Liu et al. | Learning raw image denoising with bayer pattern unification and bayer preserving augmentation | |
| Cheng et al. | Robust algorithm for exemplar-based image inpainting | |
| Whyte et al. | Non-uniform deblurring for shaken images | |
| JP5437311B2 (en) | Image correction method, image correction system, angle estimation method, and angle estimation device | |
| Paramanand et al. | Non-uniform motion deblurring for bilayer scenes | |
| CN110660037A (en) | Method, apparatus, system and computer program product for face exchange between images | |
| Chang et al. | Vornet: Spatio-temporally consistent video inpainting for object removal | |
| US20220207651A1 (en) | Method and apparatus for image processing | |
| EP3640886B1 (en) | Homography rectification | |
| US10275863B2 (en) | Homography rectification | |
| Song et al. | Weakly-supervised stitching network for real-world panoramic image generation | |
| Zheng et al. | Windowing decomposition convolutional neural network for image enhancement | |
| WO2024125245A1 (en) | Panoramic image processing method and apparatus, and electronic device and storage medium | |
| CN112150384A (en) | Method and system based on fusion of residual error network and dynamic convolution network model | |
| Wu et al. | Rectangling irregular videos by optimal spatio-temporal warping | |
| CN116934637A (en) | Photo processing method, device and equipment | |
| Dey | Image Processing Masterclass with Python: 50+ Solutions and Techniques Solving Complex Digital Image Processing Challenges Using Numpy, Scipy, Pytorch and Keras (English Edition) | |
| WO2023151385A1 (en) | Image processing method and device, terminal, and readable storage medium | |
| Wang et al. | An implementation method of the box filter on FPGA | |
| WO2021135676A1 (en) | Photographing background blurring method, mobile terminal, and storage medium | |
| CN110503603B (en) | Method for obtaining light field refocusing image based on guide up-sampling | |
| CN113409375A (en) | Image processing method, image processing apparatus, and non-volatile storage medium | |
| CN112001268B (en) | Face calibration method and equipment | |
| Yang et al. | A novel stereo image self-inpainting network for autonomous robots |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240509 Address after: 100102 room 1201, 12 / F, building 8, yard 34, Chuangyuan Road, Chaoyang District, Beijing Applicant after: Beijing new oxygen world wide Technology Consulting Co.,Ltd. Country or region after: China Address before: 100102 room 901, 9 / F, room 1001, 10 / F, building 8, yard 34, Chuangyuan Road, Chaoyang District, Beijing Applicant before: Beijing New Oxygen Technology Co.,Ltd. Country or region before: China |
|
| TA01 | Transfer of patent application right |