CN116934637A - Photo processing method, device and equipment - Google Patents

Abstract

The application relates to a photo processing method, a photo processing device and photo processing equipment. The photo processing method comprises the following steps: transforming the initial picture into an intermediate picture with unchanged resolution; respectively carrying out definition optimization processing on the intermediate picture according to the portrait area and the background area; fusing the portrait area and the background area which are subjected to definition optimization respectively to obtain a fused picture; and transforming the fusion picture into a picture with specified specification. The scheme provided by the application can improve the definition of the photo.

Description

Photo processing methods, devices and equipment

Technical field

The present application relates to the field of image processing technology, and in particular to a photo processing method, device and equipment.

Background technique

In daily life, people often use ID photos. ID photos are photos used to prove identity, and generally have high requirements for photo clarity. There are various methods for generating and processing ID photos in related technologies. Generally, the photos are scaled and transformed to specified specifications, and then related optimization processing is performed.

However, the photo processing method in the related art inevitably causes a decrease in image clarity due to scaling and transforming the photo to a specified specification and then optimizing the process, thus affecting the clarity of the photo.

Contents of the invention

In order to solve or partially solve the problems existing in related technologies, this application provides a photo processing method, device and equipment. The photo processing method, device and equipment can improve the clarity of photos.

The first aspect of this application provides a photo processing method, including:

Transform the initial photo into an intermediate image with the same resolution;

Perform definition optimization processing on the intermediate image according to the portrait area and background area respectively;

Fusion of the portrait area and background area that have been separately processed for sharpness optimization to obtain a fused image;

Transform the fused image into a specified specification image.

In one embodiment, transforming the initial photo into an intermediate image with unchanged resolution includes:

The initial photo is transformed into an intermediate image with constant resolution through the first affine transformation matrix.

In one implementation, the first affine transformation matrix is obtained in the following manner:

Align the initial photo to the reference image to obtain the transformation parameters of the position coordinates;

A transformation matrix is constructed according to the transformation parameters, and the transformation matrix is set to obtain a first affine transformation matrix.

In one embodiment, after the initial photo is transformed into an intermediate picture with constant resolution through the first affine transformation matrix, if the resolution of the picture is greater than or equal to the set threshold, the first interpolation method is used for interpolation, If the image resolution is smaller than the set threshold, the second interpolation method is used for interpolation.

In one embodiment, the definition optimization processing of the intermediate picture according to the portrait area and the background area includes:

Perform sharpness optimization processing on the portrait area of the intermediate picture according to the first optimization algorithm;

The background area of the intermediate picture is subjected to definition optimization processing according to the second optimization algorithm.

In one embodiment, converting the fused picture into a picture of a specified specification includes:

Transform the fused image into a specified specification image through a second affine transformation matrix;

The second affine transformation matrix is obtained in the following manner: align the intermediate picture to the specified specification picture to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, and use the transformation matrix as the second Affine transformation matrix.

A second aspect of this application provides a photo processing device, including:

The first transformation module is used to transform the initial photo into an intermediate image with unchanged resolution;

An optimization processing module, configured to optimize the definition of the intermediate picture transformed by the first transformation module according to the portrait area and the background area respectively;

A fusion processing module, used to fuse the portrait area and the background area after the definition optimization processing by the optimization processing module, respectively, to obtain a fused picture;

The second transformation module is used to transform the fused picture obtained by the fusion processing module into a picture of specified specifications.

In one embodiment, the first transformation module transforms the initial photo into an intermediate picture with constant resolution through a first affine transformation matrix; or,

The second transformation module transforms the fused picture into a picture of a specified specification through a second affine transformation matrix.

The third aspect of this application provides an electronic device, including:

processor; and

A memory has executable code stored thereon, and when the executable code is executed by the processor, causes the processor to perform the method as described above.

A fourth aspect of the present application provides a computer-readable storage medium on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

The technical solution provided by this application can include the following beneficial effects:

The related technology is to first scale the initial photo and then process it, and the scaling process will cause the loss of clarity in the photo processing process. However, the technical solution of this application transforms the initial photo into an intermediate image with the same resolution. , which can reduce or avoid sharpness loss during photo processing. In addition, this application performs sharpness optimization processing on the portrait area and background area of the intermediate picture respectively, and then fuses the portrait area and background area respectively after sharpness optimization processing to obtain a fused picture, which can be based on the picture. Different sharpness optimization processes are performed on the image characteristics of different areas in the photo, which further reduces or avoids the loss of sharpness in the photo processing process. Therefore, the technical solution of this application can improve the clarity of photos through optimization processing.

Furthermore, according to the technical solution of the present application, the initial photo can be transformed into an intermediate picture with unchanged resolution through a first affine transformation matrix, and the fused picture can be transformed into a picture of specified specifications through a second affine transformation matrix. By using two affine transformations, it is also possible to reduce the loss of sharpness in the photo processing pipeline.

Furthermore, according to the technical solution of this application, the portrait area of the middle picture can be processed for sharpness optimization according to the first optimization algorithm; and the background area of the middle picture can be processed for sharpness optimization according to the second optimization algorithm. Since the portrait area is individually optimized, the clarity of the portrait area can be improved; while other background areas other than the portrait area are also individually optimized using other more matching algorithms, and the clarity of the background area can also be improved. In this way, the clarity of the entire photo will be higher than that obtained by processing it using related technologies.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present application.

Description of the drawings

The above and other objects, features and advantages of the present application will become more apparent by describing the exemplary embodiments of the present application in more detail with reference to the accompanying drawings, in which the same reference numerals generally refer to the exemplary embodiments of the present application. Same parts.

Figure 1 is a schematic flowchart of a photo processing method according to an embodiment of the present application;

Figure 2 is another schematic flowchart of a photo processing method according to an embodiment of the present application;

Figure 3 is a schematic diagram of the method application corresponding to Figure 2 shown in the embodiment of the present application;

Figure 4 is a schematic diagram of the process of determining an affine transformation matrix according to an embodiment of the present application;

Figure 5 is a schematic structural diagram of a photo processing device according to an embodiment of the present application;

Figure 6 is another structural schematic diagram of a photo processing device according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a," "the" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present application, the first information may also be called second information, and similarly, the second information may also be called first information. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of this application, "plurality" means two or more than two, unless otherwise explicitly and specifically limited.

The photo processing methods in the related art can cause the image clarity to decrease, thereby affecting the clarity of the photo. To address the above problems, embodiments of the present application provide a photo processing method that can improve the clarity of photos.

The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of a photo processing method according to an embodiment of the present application.

Referring to Figure 1, the method includes:

In S101, the initial photo is transformed into an intermediate image with unchanged resolution.

In this step S101, the initial photo can be transformed into an intermediate picture with unchanged resolution through the first affine transformation matrix. The initial photo may be an ID photo, for example, but is not limited to this. The first affine transformation matrix can be obtained in the following manner: align the initial photo to the reference image to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, set the transformation matrix, and obtain the first affine transformation matrix. radial transformation matrix. Among them, the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameters; the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S are left multiplied to obtain the transformation matrix; the transformation matrix Perform a decomposition operation and modify the scaling coefficient to 1 to obtain the first affine transformation matrix. Since the first affine transformation matrix has modified the scaling coefficient to 1, the resolution of the photo remains unchanged after this step of transformation, which can reduce the loss of image clarity.

In S102, the intermediate picture is subjected to sharpness optimization processing according to the portrait area and the background area respectively.

In step S102, the portrait area of the middle picture may be subjected to sharpness optimization processing according to the first optimization algorithm; and the background area of the middle picture may be subjected to sharpness optimization processing according to the second optimization algorithm. Among them, the first optimization algorithm may be a blind face restoration algorithm; the second optimization algorithm may be an image super-resolution algorithm. Using different optimization processing methods according to the different characteristics of different areas of the image can further improve the clarity of the image.

In S103, the portrait area and the background area that have been separately processed for sharpness optimization are fused to obtain a fused picture.

In step S103, the portrait area and the background area that have been separately processed for sharpness optimization can be fused using a Poisson fusion algorithm to obtain a fused picture.

In S104, the fused image is transformed into an image of a specified specification.

In this step S104, the fused picture can be transformed into a picture of a specified specification through a second affine transformation matrix. The second affine transformation matrix can be obtained in the following manner: align the intermediate picture to the specified specification picture to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, and use the transformation matrix as the second affine transformation matrix. Among them, the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameters; by left-multiplying the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S, the transformation matrix is obtained, and the transformation matrix is as The second affine transformation matrix.

It can be seen from this embodiment that the related technology is to first scale the initial photo and then process it, and the scaling process will cause the loss of clarity in the photo processing process. However, the technical solution of this application transforms the initial photo into an intermediate picture. An intermediate image with the same resolution, which can reduce or avoid the loss of sharpness in the photo processing process. In addition, this application performs sharpness optimization processing on the portrait area and background area of the middle image respectively, and then fuses the portrait area and background area respectively after the sharpness optimization processing to obtain a fused image, which can be based on the image in the image. The image characteristics of different areas are optimized for different definitions, which further reduces or avoids the loss of definition during the photo processing process. Therefore, the technical solutions of the embodiments of the present application can improve the clarity of photos through optimization processing.

FIG. 2 is another schematic flowchart of a photo processing method according to an embodiment of the present application.

The technical solution of the embodiment of the present application is to first transform the initial photo into an intermediate picture through the first affine transformation matrix. Since the first affine transformation matrix has modified the scaling coefficient to 1, the resolution of the photo after this step of transformation has not changed (compared with the original image of the initial photo, the resolution obtained in the same area corresponding to the original image has not changed, and the entire image has not changed. The image involves rotation and cropping, and the size may change to a certain extent), but the rotation and translation are corrected; then the middle image is processed for sharpness optimization according to the portrait area and background area respectively to obtain the fused image, and finally the fused image is passed through the third Two affine transformation matrices are transformed to the specified size. Through the above processing, the problem of sharpness loss during the processing is reduced, the effect of optimizing the sharpness is improved, and the overall sharpness of the photo is better optimized.

Referring to Figures 2 and 3, the method includes:

In S201, the initial photo is transformed into an intermediate picture with unchanged resolution through the first affine transformation matrix.

The following takes the initial photo as a portrait picture and the reference picture as a front face reference picture as an example to introduce the determination process of the first affine transformation matrix. This process (see Figure 4) includes:

1) Obtain the transformation matrix tfm based on the key points of the face in the portrait picture and the key points of the front face reference picture.

1-1) Align the initial photo to the reference image to obtain the transformation parameters of the position coordinates.

Assume that the position coordinates of the key points of the face in the portrait picture are (x, y), and the position coordinates of the key points of the face in the front face reference picture are (x', y'). Align the key points of the face in the portrait picture to the front face. After referring to the facial key points of the reference picture, the transformation parameters that transform the position coordinates of the facial key points of the portrait picture into the position coordinates of the facial key points of the front face reference picture can be calculated. The so-called alignment includes translation, rotation and scaling of portrait images. The transformation parameters include translation parameters tx and ty for translation processing, rotation parameters θ for rotation processing, and scaling coefficients sx and sy for scaling processing.

Among them, the key points of the face can be selected from a set number of key feature points of the face, such as 3 or more key feature points, etc. For example, you can choose 5 key feature points of the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth. Not limited to this.

1-2) Construct the translation transformation matrix T, rotation transformation matrix R and scaling transformation matrix S according to the transformation parameters:

Among them, T, R and S represent the translation transformation matrix, rotation transformation matrix and scaling transformation matrix respectively, t represents the parameter of translation, s represents the scaling coefficient, θ represents the angle of rotation, and the subscripts x and y represent the coordinate position between the x axis and y. axis corresponding component.

Among them, translation is to move each point to (x+t, y+t); rotation is to rotate clockwise around the origin by θ angle, scaling is to enlarge or reduce the abscissa of each point by sx times, and the ordinate is enlarged (reduced) by sy times.

1-3) Multiply the translation transformation matrix T, rotation transformation matrix R and scaling transformation matrix S to obtain the transformation matrix tfm:

Formula (2) indicates that R transformation is performed first, then S transformation is performed, and then T transformation is performed. In other words, the transformation sequence is to do the later transformation in the formula first, and then do the previous transformation. Matrix multiplication is left multiplication. Matrix left multiplication can be understood as placing the matrix to the left of the multiplication sign and multiplying.

The transformation relationship between the facial key points of the front face reference picture and the facial key points of the portrait picture can be:

It can be seen that through alignment processing, the transformation matrix tfm that aligns the portrait image to the front face reference image can be calculated.

2) Perform decomposition operation on the transformation matrix tfm, modify the scaling coefficient to 1, and obtain the first affine transformation matrix tfm_stage1.

This step decomposes the transformation matrix tfm. The decomposition process is to decompose tfm into S·tfm_stage1. Theoretically, the transformation matrix tfm can be decomposed into any number of terms, such as tfm=tfm_n·tfm_n-1·....·tfm_2·tfm_1. In the technical solution of this application, it is decomposed into first performing tfm_stage1 transformation, and then performing S transformation, making it equivalent to the transformation matrix tfm. Because the S transformation is known, tfm_stage1 can be decomposed.

The decomposition operation of tfm is as follows:

Among them, the calculation formulas of the scaling coefficients sx and sy in the rotation matrix S can be as follows:

Then, change the scaling factor to 1, that is, change S to 1. tfm_stage1 is multiplied by 1, then the diagonal matrix is equivalent to not multiplying. In other words, if the scaling coefficient is 1, it means changing sx and sy in the rotation transformation matrix S to 1, which is equivalent to not performing S transformation, and the rest is tfm_stage1.

Therefore, the first affine transformation matrix required to transform to the intermediate image is obtained as follows:

3) Transform the portrait image into an intermediate image with unchanged resolution through the first affine transformation matrix tfm_stage1.

Since the scaling coefficient in the first affine transformation matrix tfm_stage1 is modified to 1, which is equivalent to no S transformation, the resolution of the transformed portrait image does not change. Finally, the portrait image is rotated and translated through the first affine transformation matrix tfm_stage1. Process and transform into an intermediate image with unchanged resolution.

After transforming the initial photo into an intermediate image with constant resolution through the first affine transformation matrix (compared with the original image of the initial photo, the resolution of the area corresponding to the same as the original image has not changed, the entire image involves rotation and Cropping, the size may change to a certain extent), if the image resolution of the photo after transformation (for example, after rotation and cropping) is greater than or equal to the set threshold, the first interpolation method is used for interpolation. If the image resolution is less than the set threshold, the second interpolation method is used. interpolation method. The first interpolation method may be, for example, the INTER_CUBIC interpolation method (cubic spline interpolation); the second interpolation method may be, for example, the INTER_AREA (area interpolation) interpolation method. For pictures with larger resolutions, the INTER_CUBIC interpolation method has better scaling effects. For compressing faces to pictures with smaller resolutions, the INTER_AREA interpolation method can avoid the graininess caused by INTER_CUBIC.

Using opencv (a cross-platform computer vision and machine learning software library released based on the Apache 2.0 license (open source)) to implement affine transformation generally involves the warpAffine function, which can implement some simple remapping. This application plan modifies the use of the warpAffine function. After first using the warpAffine function to rotate and translate the portrait image, the image is then scaled. For pictures with larger resolutions, for example, if the short side of the picture is greater than or equal to 260, the INTER_CUBIC interpolation method is used for interpolation. If the short side of the picture is less than 260, the INTER_AREA interpolation method is used for interpolation.

Since this application modifies the scaling coefficient of the first affine transformation matrix tfm_stage1 to 1, the modification to the use of the warpAffine function will not affect the use of the first affine transformation matrix tfm_stage1, but will improve the effect of the second affine transformation matrix tfm_stage2 after application. .

In S202, the portrait area of the middle picture is subjected to definition optimization processing according to the first optimization algorithm.

The human face is an area of greater concern in the picture, also known as the portrait area. The embodiment of the present application proposes to separately enhance the definition of the human face. In step S202, the portrait area of the middle picture is subjected to sharpness optimization processing according to a first optimization algorithm. The first optimization algorithm may be, for example, a face blindness restoration algorithm.

Blind face restoration refers to reconstructing face photos that have been degraded (low resolution, noise, blur, image compression, etc.), and finally obtain less degraded, clear face photos. Blind face restoration mainly uses geometric face prior information (key points, face segmentation, face heat map) to repair the face. Among them, GFPGAN is the most advanced model for blind face restoration algorithm. It uses generated face priors for blind face restoration. This application can use the GFPGAN model to optimize face clarity but is not limited to this.

In S203, the background area of the middle picture is subjected to definition optimization processing according to the second optimization algorithm.

In the embodiment of the present application, because the face area is processed and optimized separately, this step is mainly to enhance the clarity of the background except the face. This application can optimize the definition of the background area of the middle picture according to the second optimization algorithm. The second optimization algorithm may be, for example, a picture super-resolution algorithm or the like. This application uses a super-resolution algorithm model such as the RealESRGAN model to optimize background clarity. The RealESRGAN model mainly simulates various "degradation" processes in the process of high-resolution images becoming low-resolution, and then allows the model to see a blurry image and then deduce its high-definition image.

Step S203 and step S202 can be performed separately and simultaneously, and there is no sequential relationship between them.

It should be noted that by using the image super-resolution algorithm to transform the image from low resolution to high resolution, the information contained in the image changes from less to more. If the information contained in the original image is missing and the clarity is reduced, the super-resolution is finally performed. Resolution-processed models are also affected. For example, for example, if the original picture is 1024x1024, this application solution is used to directly perform super-resolution processing on the picture without scaling, and obtain photo A of 2048x2048; use relevant technology to first scale the original picture to a 512x512 picture, and then scale the The picture is then subjected to super-resolution processing to obtain photo B of 2048x2048; since photo B has been zoomed first, the clarity will decrease, so photo A obtained by using the solution of this application can have better quality than photo B obtained by using related technologies. of clarity. The same is true for face blindness restoration algorithms.

It should also be noted that if other portrait processing logic needs to be performed on the intermediate picture according to different business requirements, corresponding processing can also be performed on the intermediate picture, which is not limited by the embodiments of this application.

In S204, the portrait area and the background area that have been separately processed for sharpness optimization are fused using a Poisson fusion algorithm to obtain a fused picture.

This application applies the Poisson fusion algorithm to paste the sharpness-enhanced face back to the background sharpness-enhanced picture to complete the photo sharpness enhancement and obtain the processed fusion picture.

Image fusion is a key technology in image splicing technology. Its principle is to achieve smooth transition and seamless splicing between spliced images by redefining and calculating the pixels in the overlapping zones in the spliced images. The Poisson fusion algorithm based on Poisson's equation uses the gradient field of two images to guide the difference in the overlapping area, turning the image fusion problem into a minimization problem of solving the difference between the gradient field of the target image block and the background guidance gradient field. Good image fusion effect can be achieved.

In S205, the fused picture is transformed into a picture of a specified specification through the second affine transformation matrix.

This step determines the second affine transformation matrix tfm_stage2 of the intermediate picture aligned to the specified specification photo.

The principle of the determination process of the second affine transformation matrix tfm_stage2 and the first affine transformation matrix tfm_stage1 is the same (see Figure 4). According to the face key points of the intermediate picture and the face key points of the specified specification picture, Get the transformation matrix tfm. However, at this time, there is no need to decompose the transformation matrix tfm, and there is no need to modify the scaling coefficient to 1. The obtained transformation matrix tfm only needs to be directly used as the second affine transformation matrix tfm_stage2.

This step S205 uses the second affine transformation matrix to transform the fused picture into a picture of specified specifications, that is, applying affine transformation to the fused picture obtained after the fusion process to obtain the final photo of specified specifications.

In summary, the technical solution of the embodiment of the present application modifies the scaling coefficient of the affine transformation matrix to 1 so that the resolution of the image does not change after rotation and translation correction of the initial photo, and specifies the affine transformation process. The interpolation method used by scaling can reduce the loss of sharpness in the photo processing pipeline. Although the image after the second affine transformation was used in the end, the intermediate image after the first affine transformation was used before the last step. The intermediate image maintains the same resolution as the initial photo, thereby avoiding It eliminates the loss of clarity during processing and improves the clarity of the entire photo after processing.

Corresponding to the foregoing application function implementation method embodiments, embodiments of the present application also provide a photo processing device, electronic equipment and corresponding embodiments.

FIG. 5 is a schematic structural diagram of a photo processing device according to an embodiment of the present application.

Referring to FIG. 5 , a photo processing device 50 includes: a first transformation module 51 , an optimization processing module 52 , a fusion processing module 53 , and a second transformation module 54 .

The first transformation module 51 is used to transform the initial photo into an intermediate picture with unchanged resolution. The first transformation module 51 transforms the initial photo into an intermediate picture with unchanged resolution through a first affine transformation matrix. The first affine transformation matrix can be obtained in the following way: align the initial photo to the reference image to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, set the transformation matrix, and obtain the first affine transformation matrix. radial transformation matrix. Among them, the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameters; the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S are left multiplied to obtain the transformation matrix; the transformation matrix Perform a decomposition operation and modify the scaling coefficient to 1 to obtain the first affine transformation matrix.

The optimization processing module 52 is configured to perform definition optimization processing on the intermediate pictures transformed by the first transformation module 51 according to the portrait area and the background area respectively. The optimization processing module 52 can optimize the definition of the portrait area of the middle picture according to the first optimization algorithm; and perform the definition optimization processing of the background area of the middle picture according to the second optimization algorithm. Among them, the first optimization algorithm may be a blind face restoration algorithm; the second optimization algorithm may be an image super-resolution algorithm.

The fusion processing module 53 is used to fuse the portrait area and the background area that have been separately processed for sharpness optimization by the optimization processing module 52 to obtain a fused picture. The fusion processing module 53 can use the Poisson fusion algorithm to fuse the portrait area and the background area that have been separately processed for sharpness optimization to obtain a fused picture.

The second transformation module 54 is used to transform the fused picture obtained by the fusion processing module 53 into a picture of a specified specification. The second transformation module 54 may transform the fused picture into a picture of a specified specification through a second affine transformation matrix. The second affine transformation matrix can be obtained in the following manner: align the intermediate picture to the specified specification picture to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, and use the transformation matrix as the second affine transformation matrix. Among them, the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameters; by left-multiplying the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S, the transformation matrix is obtained, and the transformation matrix is as The second affine transformation matrix.

It can be seen from this embodiment that the related technology is to first scale the initial photo and then process it, and the scaling process will cause the loss of clarity in the photo processing process. However, the technical solution of this application transforms the initial photo into an intermediate picture. An intermediate image with the same resolution, which can reduce or avoid the loss of sharpness in the photo processing process. In addition, this application performs sharpness optimization processing on the portrait area and background area of the middle image respectively, and then fuses the portrait area and background area respectively after the sharpness optimization processing to obtain a fused image, which can be based on the image in the image. The image characteristics of different areas are optimized for different definitions, which further reduces or avoids the loss of definition during the photo processing process. Therefore, the technical solution of this application can improve the clarity of photos through optimization processing.

FIG. 6 is another schematic structural diagram of a photo processing device according to an embodiment of the present application.

Referring to FIG. 6 , a photo processing device 50 includes: a first transformation module 51 , an optimization processing module 52 , a fusion processing module 53 , and a second transformation module 54 .

The functions of the first transformation module 51, the optimization processing module 52, the fusion processing module 53, and the second transformation module 54 can be referred to the description in Figure 5.

Among them, the first transformation module 51 can transform the initial photo into an intermediate picture with unchanged resolution through the first affine transformation matrix; the second transformation module 54 can transform the fused picture into a specified specification picture through the second affine transformation matrix.

The optimization processing module 52 may include: a first optimization sub-module 521 and a second optimization sub-module 522 .

The first optimization sub-module 521 is used to optimize the definition of the portrait area of the middle picture according to the first optimization algorithm;

The second optimization sub-module 522 is used to optimize the definition of the background area of the middle picture according to the second optimization algorithm.

Among them, the first optimization algorithm may be a blind face restoration algorithm; the second optimization algorithm may be an image super-resolution algorithm. GFPGAN is the most advanced model of blind face restoration algorithm. It uses generated face priors for blind face restoration. This application can use GFPGAN to optimize face clarity but is not limited to this. This application can use a super-resolution algorithm model such as the RealESRGAN model to optimize background clarity.

Regarding the devices in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Referring to FIG. 7 , electronic device 1000 includes memory 1010 and processor 1020 .

The processor 1020 can be a central processing unit (CPU), other general-purpose processor, digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or field-processable processor. Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

Memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and persistent storage. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. Persistent storage may be readable and writable storage. Persistent storage may be a non-volatile storage device that does not lose stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (eg, magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (eg, floppy disk, optical drive). System memory can be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory can store some or all of the instructions and data the processor needs to run. In addition, memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (eg, DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks, and/or optical disks may also be used. In some embodiments, memory 1010 may include a readable and/or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (eg, DVD-ROM, dual-layer DVD-ROM), Read-only Blu-ray discs, ultra-density discs, flash memory cards (such as SD card, min SD card, Micro-SD card, etc.), magnetic floppy disks, etc. Computer-readable storage media do not contain carrier waves and transient electronic signals that are transmitted wirelessly or wired.

The memory 1010 stores executable code. When the executable code is processed by the processor 1020, the processor 1020 can be caused to execute part or all of the above-mentioned methods.

In addition, the method according to the present application can also be implemented as a computer program or computer program product, which computer program or computer program product includes computer program code instructions for executing part or all of the steps in the above method of the present application.

Alternatively, the application may also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) with executable code (or computer program or computer instruction code) stored thereon, When the executable code (or computer program or computer instruction code) is executed by the processor of the electronic device (or server, etc.), the processor is caused to execute part or all of the respective steps of the above method according to the present application.

The embodiments of the present application have been described above. The above description is illustrative, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (10)

1. A photo processing method, characterized in that it includes: Transform the initial photo into an intermediate image with the same resolution; Perform definition optimization processing on the intermediate image according to the portrait area and background area respectively; Fusion of the portrait area and background area that have been separately processed for sharpness optimization to obtain a fused image; Transform the fused image into a specified specification image. 2. The method according to claim 1, characterized in that said converting the initial photo into an intermediate picture with unchanged resolution includes: The initial photo is transformed into an intermediate image with constant resolution through the first affine transformation matrix. 3. The method according to claim 2, characterized in that the first affine transformation matrix is obtained in the following manner: Align the initial photo to the reference image to obtain the transformation parameters of the position coordinates; A transformation matrix is constructed according to the transformation parameters, and the transformation matrix is set to obtain a first affine transformation matrix. 4. The method according to claim 2, characterized in that: After the initial photo is transformed into an intermediate picture with constant resolution through the first affine transformation matrix, if the picture resolution is greater than or equal to the set threshold, the first interpolation method is used for interpolation. If the picture resolution is less than the set threshold, Set the threshold and use the second interpolation method for interpolation. 5. The method according to claim 1, wherein the step of optimizing the definition of the intermediate picture according to the portrait area and the background area includes: Perform sharpness optimization processing on the portrait area of the intermediate picture according to the first optimization algorithm; The background area of the intermediate picture is subjected to definition optimization processing according to the second optimization algorithm. 6. The method according to claim 1, characterized in that said converting the fused picture into a specified specification picture includes: Transform the fused image into a specified specification image through a second affine transformation matrix; The second affine transformation matrix is obtained in the following manner: align the intermediate picture to the specified specification picture to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, and use the transformation matrix as the second Affine transformation matrix. 7. A photo processing device, characterized by comprising: The first transformation module is used to transform the initial photo into an intermediate image with unchanged resolution; An optimization processing module, configured to optimize the definition of the intermediate picture transformed by the first transformation module according to the portrait area and the background area respectively; A fusion processing module, used to fuse the portrait area and the background area after the definition optimization processing by the optimization processing module, respectively, to obtain a fused picture; The second transformation module is used to transform the fused picture obtained by the fusion processing module into a picture of specified specifications. 8. The device according to claim 7, characterized in that: The first transformation module transforms the initial photo into an intermediate picture with constant resolution through a first affine transformation matrix; or, The second transformation module transforms the fused picture into a picture of a specified specification through a second affine transformation matrix. 9. An electronic device, characterized in that it includes: processor; and A memory having executable code stored thereon, which when executed by the processor causes the processor to perform the method of any one of claims 1-6. 10. A computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute as claimed in any one of claims 1-6. method described.