CN116934637A - Photo processing method, device and equipment - Google Patents

Photo processing method, device and equipment Download PDF

Info

Publication number
CN116934637A
CN116934637A CN202210355149.6A CN202210355149A CN116934637A CN 116934637 A CN116934637 A CN 116934637A CN 202210355149 A CN202210355149 A CN 202210355149A CN 116934637 A CN116934637 A CN 116934637A
Authority
CN
China
Prior art keywords
picture
transformation matrix
image
photo
resolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210355149.6A
Other languages
Chinese (zh)
Inventor
苗锋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing New Oxygen World Wide Technology Consulting Co ltd
Original Assignee
Soyoung Technology Beijing Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Soyoung Technology Beijing Co Ltd filed Critical Soyoung Technology Beijing Co Ltd
Priority to CN202210355149.6A priority Critical patent/CN116934637A/en
Publication of CN116934637A publication Critical patent/CN116934637A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20221Image fusion; Image merging
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30196Human being; Person
    • G06T2207/30201Face

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

The application relates to a photo processing method, a photo processing device and photo processing equipment. The photo processing method comprises the following steps: transforming the initial picture into an intermediate picture with unchanged resolution; respectively carrying out definition optimization processing on the intermediate picture according to the portrait area and the background area; fusing the portrait area and the background area which are subjected to definition optimization respectively to obtain a fused picture; and transforming the fusion picture into a picture with specified specification. The scheme provided by the application can improve the definition of the photo.

Description

照片处理方法、装置及设备Photo processing methods, devices and equipment

技术领域Technical field

本申请涉及图像处理技术领域,尤其涉及一种照片处理方法、装置及设备。The present application relates to the field of image processing technology, and in particular to a photo processing method, device and equipment.

背景技术Background technique

在日常生活中,人们经常会使用证件照。证件照是用来证明身份的照片,一般对照片的清晰度有较高的要求。相关技术中有各种不同的证件照生成和处理方法,一般是将照片缩放变换到指定的规格,然后进行相关优化处理。In daily life, people often use ID photos. ID photos are photos used to prove identity, and generally have high requirements for photo clarity. There are various methods for generating and processing ID photos in related technologies. Generally, the photos are scaled and transformed to specified specifications, and then related optimization processing is performed.

但是,相关技术中的照片处理方法,由于将照片缩放变换到指定规格再优化处理,不可避免的造成图片清晰度下降,从而影响了照片的清晰度。However, the photo processing method in the related art inevitably causes a decrease in image clarity due to scaling and transforming the photo to a specified specification and then optimizing the process, thus affecting the clarity of the photo.

发明内容Contents of the invention

为解决或部分解决相关技术中存在的问题,本申请提供一种照片处理方法、装置及设备,该照片处理方法、装置及设备,能够提高照片的清晰度。In order to solve or partially solve the problems existing in related technologies, this application provides a photo processing method, device and equipment. The photo processing method, device and equipment can improve the clarity of photos.

本申请第一方面提供一种照片处理方法,包括:The first aspect of this application provides a photo processing method, including:

将初始照片变换为分辨率不变的中间图片;Transform the initial photo into an intermediate image with the same resolution;

将所述中间图片按人像区域和背景区域分别进行清晰度优化处理;Perform definition optimization processing on the intermediate image according to the portrait area and background area respectively;

将分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片;Fusion of the portrait area and background area that have been separately processed for sharpness optimization to obtain a fused image;

将所述融合图片变换为指定规格图片。Transform the fused image into a specified specification image.

在一实施方式中,所述将初始照片变换为分辨率不变的中间图片,包括:In one embodiment, transforming the initial photo into an intermediate image with unchanged resolution includes:

通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片。The initial photo is transformed into an intermediate image with constant resolution through the first affine transformation matrix.

在一实施方式中,所述第一仿射变换矩阵按以下方式得到:In one implementation, the first affine transformation matrix is obtained in the following manner:

将所述初始照片对齐到参考图得到位置坐标的变换参数;Align the initial photo to the reference image to obtain the transformation parameters of the position coordinates;

根据所述变换参数构建变换矩阵,将所述变换矩阵进行设定处理,得到第一仿射变换矩阵。A transformation matrix is constructed according to the transformation parameters, and the transformation matrix is set to obtain a first affine transformation matrix.

在一实施方式中,所述通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片后,若所述图片分辨率大于或等于设定阈值,采用第一插值法进行插值,若图片分辨率小于所述设定阈值,采用第二插值法进行插值。In one embodiment, after the initial photo is transformed into an intermediate picture with constant resolution through the first affine transformation matrix, if the resolution of the picture is greater than or equal to the set threshold, the first interpolation method is used for interpolation, If the image resolution is smaller than the set threshold, the second interpolation method is used for interpolation.

在一实施方式中,所述将所述中间图片按人像区域和背景区域分别进行清晰度优化处理,包括:In one embodiment, the definition optimization processing of the intermediate picture according to the portrait area and the background area includes:

将所述中间图片的人像区域按第一优化算法进行清晰度优化处理;Perform sharpness optimization processing on the portrait area of the intermediate picture according to the first optimization algorithm;

将所述中间图片的背景区域按第二优化算法进行清晰度优化处理。The background area of the intermediate picture is subjected to definition optimization processing according to the second optimization algorithm.

在一实施方式中,所述将所述融合图片变换为指定规格图片,包括:In one embodiment, converting the fused picture into a picture of a specified specification includes:

通过第二仿射变换矩阵将所述融合图片变换为指定规格图片;Transform the fused image into a specified specification image through a second affine transformation matrix;

其中所述第二仿射变换矩阵按以下方式得到:将所述中间图片对齐到所述指定规格图片得到位置坐标的变换参数;根据所述变换参数构建变换矩阵,将所述变换矩阵作为第二仿射变换矩阵。The second affine transformation matrix is obtained in the following manner: align the intermediate picture to the specified specification picture to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, and use the transformation matrix as the second Affine transformation matrix.

本申请第二方面提供一种照片处理装置,包括:A second aspect of this application provides a photo processing device, including:

第一变换模块,用于将初始照片变换为分辨率不变的中间图片;The first transformation module is used to transform the initial photo into an intermediate image with unchanged resolution;

优化处理模块,用于将所述第一变换模块变换的中间图片按人像区域和背景区域分别进行清晰度优化处理;An optimization processing module, configured to optimize the definition of the intermediate picture transformed by the first transformation module according to the portrait area and the background area respectively;

融合处理模块,用于将所述优化处理模块分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片;A fusion processing module, used to fuse the portrait area and the background area after the definition optimization processing by the optimization processing module, respectively, to obtain a fused picture;

第二变换模块,用于将所述融合处理模块得到的融合图片变换为指定规格图片。The second transformation module is used to transform the fused picture obtained by the fusion processing module into a picture of specified specifications.

在一实施方式中,所述第一变换模块通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片;或,In one embodiment, the first transformation module transforms the initial photo into an intermediate picture with constant resolution through a first affine transformation matrix; or,

所述第二变换模块通过第二仿射变换矩阵将所述融合图片变换为指定规格图片。The second transformation module transforms the fused picture into a picture of a specified specification through a second affine transformation matrix.

本申请第三方面提供一种电子设备,包括:The third aspect of this application provides an electronic device, including:

处理器;以及processor; and

存储器,其上存储有可执行代码,当所述可执行代码被所述处理器执行时,使所述处理器执行如上所述的方法。A memory has executable code stored thereon, and when the executable code is executed by the processor, causes the processor to perform the method as described above.

本申请第四方面提供一种计算机可读存储介质,其上存储有可执行代码,当所述可执行代码被电子设备的处理器执行时,使所述处理器执行如上所述的方法。A fourth aspect of the present application provides a computer-readable storage medium on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

本申请提供的技术方案可以包括以下有益效果:The technical solution provided by this application can include the following beneficial effects:

相关技术是先将初始照片进行缩放后处理,而缩放处理会导致照片处理流程中的清晰度流失,而本申请的技术方案将初始照片变换为中间图片时是变换为分辨率不变的中间图片,这样可以减少或避免照片处理流程中的清晰度流失。另外,本申请是将所述中间图片按人像区域和背景区域分别进行清晰度优化处理,然后再将分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片,这样可以根据图片中的不同区域的图像特点进行不同的清晰度优化处理,也进一步减少或避免照片处理流程中的清晰度流失。因此,本申请的技术方案,能够通过优化处理提高照片的清晰度。The related technology is to first scale the initial photo and then process it, and the scaling process will cause the loss of clarity in the photo processing process. However, the technical solution of this application transforms the initial photo into an intermediate image with the same resolution. , which can reduce or avoid sharpness loss during photo processing. In addition, this application performs sharpness optimization processing on the portrait area and background area of the intermediate picture respectively, and then fuses the portrait area and background area respectively after sharpness optimization processing to obtain a fused picture, which can be based on the picture. Different sharpness optimization processes are performed on the image characteristics of different areas in the photo, which further reduces or avoids the loss of sharpness in the photo processing process. Therefore, the technical solution of this application can improve the clarity of photos through optimization processing.

进一步的,本申请的技术方案,可以通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片,通过第二仿射变换矩阵将所述融合图片变换为指定规格图片。通过使用两次仿射变换,也可以减少照片处理流程中的清晰度流失。Furthermore, according to the technical solution of the present application, the initial photo can be transformed into an intermediate picture with unchanged resolution through a first affine transformation matrix, and the fused picture can be transformed into a picture of specified specifications through a second affine transformation matrix. By using two affine transformations, it is also possible to reduce the loss of sharpness in the photo processing pipeline.

进一步的,本申请的技术方案,可以将所述中间图片的人像区域按第一优化算法进行清晰度优化处理;将所述中间图片的背景区域按第二优化算法进行清晰度优化处理。由于人像区域进行了单独优化处理,人像区域的清晰度可以得到提高;而除人像区域外的其他背景区域也采用其他更匹配的算法单独优化处理,背景区域的清晰度也可以得到提高。这样,整个照片的清晰度相比于采用相关技术处理得到的清晰度都会更高。Furthermore, according to the technical solution of this application, the portrait area of the middle picture can be processed for sharpness optimization according to the first optimization algorithm; and the background area of the middle picture can be processed for sharpness optimization according to the second optimization algorithm. Since the portrait area is individually optimized, the clarity of the portrait area can be improved; while other background areas other than the portrait area are also individually optimized using other more matching algorithms, and the clarity of the background area can also be improved. In this way, the clarity of the entire photo will be higher than that obtained by processing it using related technologies.

应当理解的是,以上的一般描述和后文的细节描述仅是示例性和解释性的,并不能限制本申请。It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present application.

附图说明Description of the drawings

通过结合附图对本申请示例性实施方式进行更详细地描述,本申请的上述以及其它目的、特征和优势将变得更加明显,其中,在本申请示例性实施方式中,相同的参考标号通常代表相同部件。The above and other objects, features and advantages of the present application will become more apparent by describing the exemplary embodiments of the present application in more detail with reference to the accompanying drawings, in which the same reference numerals generally refer to the exemplary embodiments of the present application. Same parts.

图1是本申请实施例示出的照片处理方法的流程示意图;Figure 1 is a schematic flowchart of a photo processing method according to an embodiment of the present application;

图2是本申请实施例示出的照片处理方法的另一流程示意图;Figure 2 is another schematic flowchart of a photo processing method according to an embodiment of the present application;

图3是本申请实施例示出的与图2对应的方法应用示意图;Figure 3 is a schematic diagram of the method application corresponding to Figure 2 shown in the embodiment of the present application;

图4是本申请实施例示出的确定仿射变换矩阵的过程示意图;Figure 4 is a schematic diagram of the process of determining an affine transformation matrix according to an embodiment of the present application;

图5是本申请实施例示出的照片处理装置的结构示意图;Figure 5 is a schematic structural diagram of a photo processing device according to an embodiment of the present application;

图6是本申请实施例示出的照片处理装置的另一结构示意图;Figure 6 is another structural schematic diagram of a photo processing device according to an embodiment of the present application;

图7是本申请实施例示出的电子设备的结构示意图。FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

具体实施方式Detailed ways

下面将参照附图更详细地描述本申请的实施方式。虽然附图中显示了本申请的实施方式,然而应该理解,可以以各种形式实现本申请而不应被这里阐述的实施方式所限制。相反,提供这些实施方式是为了使本申请更加透彻和完整,并且能够将本申请的范围完整地传达给本领域的技术人员。Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

在本申请使用的术语是仅仅出于描述特定实施例的目的,而非旨在限制本申请。在本申请和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式,除非上下文清楚地表示其他含义。还应当理解,本文中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a," "the" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

应当理解,尽管在本申请可能采用术语“第一”、“第二”、“第三”等来描述各种信息,但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如,在不脱离本申请范围的情况下,第一信息也可以被称为第二信息,类似地,第二信息也可以被称为第一信息。由此,限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本申请的描述中,“多个”的含义是两个或两个以上,除非另有明确具体的限定。It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present application, the first information may also be called second information, and similarly, the second information may also be called first information. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of this application, "plurality" means two or more than two, unless otherwise explicitly and specifically limited.

相关技术中的照片处理方法会造成图片清晰度下降,从而影响了照片的清晰度。针对上述问题,本申请实施例提供一种照片处理方法,能够提高照片的清晰度。The photo processing methods in the related art can cause the image clarity to decrease, thereby affecting the clarity of the photo. To address the above problems, embodiments of the present application provide a photo processing method that can improve the clarity of photos.

以下结合附图详细描述本申请实施例的技术方案。The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.

图1是本申请实施例示出的照片处理方法的流程示意图。FIG. 1 is a schematic flowchart of a photo processing method according to an embodiment of the present application.

参见图1,该方法包括:Referring to Figure 1, the method includes:

在S101中,将初始照片变换为分辨率不变的中间图片。In S101, the initial photo is transformed into an intermediate image with unchanged resolution.

该步骤S101可以通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片。初始照片例如可以是证件照但不局限于此。其中第一仿射变换矩阵可以按以下方式得到:将初始照片对齐到参考图得到位置坐标的变换参数;根据所述变换参数构建变换矩阵,将所述变换矩阵进行设定处理,得到第一仿射变换矩阵。其中,可以根据变换参数构建平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S;将平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S进行左乘运算,则得到变换矩阵;将变换矩阵进行分解运算,将缩放系数修改为1,得到第一仿射变换矩阵。由于第一仿射变换矩阵已经修改缩放系数为1,因此该步变换后照片的分辨率不变,可以减少图片的清晰度流失。In this step S101, the initial photo can be transformed into an intermediate picture with unchanged resolution through the first affine transformation matrix. The initial photo may be an ID photo, for example, but is not limited to this. The first affine transformation matrix can be obtained in the following manner: align the initial photo to the reference image to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, set the transformation matrix, and obtain the first affine transformation matrix. radial transformation matrix. Among them, the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameters; the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S are left multiplied to obtain the transformation matrix; the transformation matrix Perform a decomposition operation and modify the scaling coefficient to 1 to obtain the first affine transformation matrix. Since the first affine transformation matrix has modified the scaling coefficient to 1, the resolution of the photo remains unchanged after this step of transformation, which can reduce the loss of image clarity.

在S102中,将中间图片按人像区域和背景区域分别进行清晰度优化处理。In S102, the intermediate picture is subjected to sharpness optimization processing according to the portrait area and the background area respectively.

该步骤S102可以将中间图片的人像区域按第一优化算法进行清晰度优化处理;将中间图片的背景区域按第二优化算法进行清晰度优化处理。其中,第一优化算法可以是人脸盲复原算法;第二优化算法可以是图片超分辨率算法。根据图片不同区域的不同特点分别采用不同的优化处理方式,可以进一步提升图片的清晰度。In step S102, the portrait area of the middle picture may be subjected to sharpness optimization processing according to the first optimization algorithm; and the background area of the middle picture may be subjected to sharpness optimization processing according to the second optimization algorithm. Among them, the first optimization algorithm may be a blind face restoration algorithm; the second optimization algorithm may be an image super-resolution algorithm. Using different optimization processing methods according to the different characteristics of different areas of the image can further improve the clarity of the image.

在S103中,将分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片。In S103, the portrait area and the background area that have been separately processed for sharpness optimization are fused to obtain a fused picture.

该步骤S103可以将分别进行清晰度优化处理后的人像区域和背景区域采用泊松融合算法进行融合,得到融合图片。In step S103, the portrait area and the background area that have been separately processed for sharpness optimization can be fused using a Poisson fusion algorithm to obtain a fused picture.

在S104中,将融合图片变换为指定规格图片。In S104, the fused image is transformed into an image of a specified specification.

该步骤S104可以通过第二仿射变换矩阵将融合图片变换为指定规格图片。其中第二仿射变换矩阵可以按以下方式得到:将中间图片对齐到指定规格图片得到位置坐标的变换参数;根据所述变换参数构建变换矩阵,将所述变换矩阵作为第二仿射变换矩阵。其中可以根据变换参数构建平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S;将平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S进行左乘运算,则得到变换矩阵,将变换矩阵作为第二仿射变换矩阵。In this step S104, the fused picture can be transformed into a picture of a specified specification through a second affine transformation matrix. The second affine transformation matrix can be obtained in the following manner: align the intermediate picture to the specified specification picture to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, and use the transformation matrix as the second affine transformation matrix. Among them, the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameters; by left-multiplying the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S, the transformation matrix is obtained, and the transformation matrix is as The second affine transformation matrix.

从该实施例可以看出,相关技术是先将初始照片进行缩放后处理,而缩放处理会导致照片处理流程中的清晰度流失,而本申请的技术方案将初始照片变换为中间图片时是变换为分辨率不变的中间图片,这样可以减少或避免照片处理流程中的清晰度流失。另外,本申请是将中间图片按人像区域和背景区域分别进行清晰度优化处理,然后再将分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片,这样可以根据图片中的不同区域的图像特点进行不同的清晰度优化处理,也进一步减少或避免照片处理流程中的清晰度流失。因此,本申请实施例的技术方案,能够通过优化处理提高照片的清晰度。It can be seen from this embodiment that the related technology is to first scale the initial photo and then process it, and the scaling process will cause the loss of clarity in the photo processing process. However, the technical solution of this application transforms the initial photo into an intermediate picture. An intermediate image with the same resolution, which can reduce or avoid the loss of sharpness in the photo processing process. In addition, this application performs sharpness optimization processing on the portrait area and background area of the middle image respectively, and then fuses the portrait area and background area respectively after the sharpness optimization processing to obtain a fused image, which can be based on the image in the image. The image characteristics of different areas are optimized for different definitions, which further reduces or avoids the loss of definition during the photo processing process. Therefore, the technical solutions of the embodiments of the present application can improve the clarity of photos through optimization processing.

图2是本申请实施例示出的照片处理方法的另一流程示意图。FIG. 2 is another schematic flowchart of a photo processing method according to an embodiment of the present application.

本申请实施例的技术方案,先将初始照片通过第一仿射变换矩阵变换到一个中间图片。由于第一仿射变换矩阵已经修改缩放系数为1,因此该步变换后照片的分辨率没有改变(和初始照片的原图相比,与原图对应相同的区域得到分辨率没有改变,整张图片涉及旋转和裁剪,尺寸可能发生一定变化),只是进行了旋转和平移的校正;接着将中间图片按人像区域和背景区域分别进行清晰度优化处理并得到融合图片,最后再将融合图片通过第二仿射变换矩阵变换到指定规格大小。通过上述处理,减少了处理过程中的清晰度流失问题,提高了优化清晰度的效果,使得照片的整体清晰度得到更好的优化。The technical solution of the embodiment of the present application is to first transform the initial photo into an intermediate picture through the first affine transformation matrix. Since the first affine transformation matrix has modified the scaling coefficient to 1, the resolution of the photo after this step of transformation has not changed (compared with the original image of the initial photo, the resolution obtained in the same area corresponding to the original image has not changed, and the entire image has not changed. The image involves rotation and cropping, and the size may change to a certain extent), but the rotation and translation are corrected; then the middle image is processed for sharpness optimization according to the portrait area and background area respectively to obtain the fused image, and finally the fused image is passed through the third Two affine transformation matrices are transformed to the specified size. Through the above processing, the problem of sharpness loss during the processing is reduced, the effect of optimizing the sharpness is improved, and the overall sharpness of the photo is better optimized.

参见图2和图3,该方法包括:Referring to Figures 2 and 3, the method includes:

在S201中,通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片。In S201, the initial photo is transformed into an intermediate picture with unchanged resolution through the first affine transformation matrix.

以下以初始照片为人像图片、参考图为正脸参考图为例,介绍第一仿射变换矩阵的确定过程,该过程(可参见图4)包括:The following takes the initial photo as a portrait picture and the reference picture as a front face reference picture as an example to introduce the determination process of the first affine transformation matrix. This process (see Figure 4) includes:

1)根据人像图片的人脸关键点和正脸参考图的人脸关键点,求得变换矩阵tfm1) Obtain the transformation matrix tfm based on the key points of the face in the portrait picture and the key points of the front face reference picture.

1-1)将初始照片对齐到参考图得到位置坐标的变换参数。1-1) Align the initial photo to the reference image to obtain the transformation parameters of the position coordinates.

假设人像图片的人脸关键点的位置坐标为(x,y),正脸参考图的人脸关键点的位置坐标为(x’,y’),将人像图片的人脸关键点对齐正脸参考图的人脸关键点后,可以计算出将人像图片的人脸关键点的位置坐标变换为正脸参考图的人脸关键点的位置坐标的变换参数。所说的对齐,包括将人像图片进行平移、旋转和缩放处理等。其中变换参数包括进行平移处理的平移参数tx和ty、进行旋转处理的旋转参数θ、进行缩放处理的缩放系数sx和sy。Assume that the position coordinates of the key points of the face in the portrait picture are (x, y), and the position coordinates of the key points of the face in the front face reference picture are (x', y'). Align the key points of the face in the portrait picture to the front face. After referring to the facial key points of the reference picture, the transformation parameters that transform the position coordinates of the facial key points of the portrait picture into the position coordinates of the facial key points of the front face reference picture can be calculated. The so-called alignment includes translation, rotation and scaling of portrait images. The transformation parameters include translation parameters tx and ty for translation processing, rotation parameters θ for rotation processing, and scaling coefficients sx and sy for scaling processing.

其中,人脸关键点可以选人脸的设定数量关键特征点,例如3个或以上关键特征点等,例如可以选左眼、右眼、鼻子、左嘴角、右嘴角5个关键特征点但不局限于此。Among them, the key points of the face can be selected from a set number of key feature points of the face, such as 3 or more key feature points, etc. For example, you can choose 5 key feature points of the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth. Not limited to this.

1-2)根据变换参数构建平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S:1-2) Construct the translation transformation matrix T, rotation transformation matrix R and scaling transformation matrix S according to the transformation parameters:

其中T、R、S分别表示平移变换矩阵、旋转变换矩阵和缩放变换矩阵,t表示平移的参数,s表示缩放系数,θ表示旋转的角度,下标x、y表示坐标位置在x轴和y轴对应分量。Among them, T, R and S represent the translation transformation matrix, rotation transformation matrix and scaling transformation matrix respectively, t represents the parameter of translation, s represents the scaling coefficient, θ represents the angle of rotation, and the subscripts x and y represent the coordinate position between the x axis and y. axis corresponding component.

其中,平移是将每一点移到(x+t,y+t);旋转是围绕原点顺时针旋转θ角度,缩放是将每一点的横坐标放大或缩小sx倍,纵坐标放大(缩小)sy倍。Among them, translation is to move each point to (x+t, y+t); rotation is to rotate clockwise around the origin by θ angle, scaling is to enlarge or reduce the abscissa of each point by sx times, and the ordinate is enlarged (reduced) by sy times.

1-3)将平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S进行相乘运算,则得到变换矩阵tfm:1-3) Multiply the translation transformation matrix T, rotation transformation matrix R and scaling transformation matrix S to obtain the transformation matrix tfm:

公式(2)表示先进行R变换,再进行S变换,然后进行T变换。也就是说,变换顺序是先做公式中后面的变换,再做前面的变换,矩阵乘法是左乘。矩阵左乘,可以理解为矩阵放到乘号左边相乘。Formula (2) indicates that R transformation is performed first, then S transformation is performed, and then T transformation is performed. In other words, the transformation sequence is to do the later transformation in the formula first, and then do the previous transformation. Matrix multiplication is left multiplication. Matrix left multiplication can be understood as placing the matrix to the left of the multiplication sign and multiplying.

正脸参考图的人脸关键点和人像图片的人脸关键点的变换关系可以为:The transformation relationship between the facial key points of the front face reference picture and the facial key points of the portrait picture can be:

可见,通过对齐处理,就能计算出人像图片对齐到正脸参考图的变换矩阵tfm。It can be seen that through alignment processing, the transformation matrix tfm that aligns the portrait image to the front face reference image can be calculated.

2)将变换矩阵tfm进行分解运算,将缩放系数修改为1,得到第一仿射变换矩阵tfm_stage1。2) Perform decomposition operation on the transformation matrix tfm, modify the scaling coefficient to 1, and obtain the first affine transformation matrix tfm_stage1.

该步骤将变换矩阵tfm进行分解,分解的过程就是将tfm分解为S·tfm_stage1。理论上变换矩阵tfm可以分解为任意多项,例如tfm=tfm_n·tfm_n-1·....·tfm_2·tfm_1。本申请技术方案中,分解为先进行tfm_stage1变换,再进行S变换,让其和变换矩阵tfm等价。因为S变换是已知,所以tfm_stage1就可以分解出来。This step decomposes the transformation matrix tfm. The decomposition process is to decompose tfm into S·tfm_stage1. Theoretically, the transformation matrix tfm can be decomposed into any number of terms, such as tfm=tfm_n·tfm_n-1·....·tfm_2·tfm_1. In the technical solution of this application, it is decomposed into first performing tfm_stage1 transformation, and then performing S transformation, making it equivalent to the transformation matrix tfm. Because the S transformation is known, tfm_stage1 can be decomposed.

tfm的分解运算如下:The decomposition operation of tfm is as follows:

其中,旋转矩阵S中的缩放系数sx和sy的计算公式可以如下:Among them, the calculation formulas of the scaling coefficients sx and sy in the rotation matrix S can be as follows:

然后,将缩放系数修改为1,也即将S改成1。tfm_stage1乘1,那么对角线矩阵相当于不乘。也就是说,缩放系数为1,就是将旋转变换矩阵S中的sx和sy改成1,相当于没进行S变换,剩下就是tfm_stage1。Then, change the scaling factor to 1, that is, change S to 1. tfm_stage1 is multiplied by 1, then the diagonal matrix is equivalent to not multiplying. In other words, if the scaling coefficient is 1, it means changing sx and sy in the rotation transformation matrix S to 1, which is equivalent to not performing S transformation, and the rest is tfm_stage1.

因此,得到变换到中间图片所需要的第一仿射变换矩阵如下所示:Therefore, the first affine transformation matrix required to transform to the intermediate image is obtained as follows:

3)通过第一仿射变换矩阵tfm_stage1将人像图片变换为分辨率不变的中间图片。3) Transform the portrait image into an intermediate image with unchanged resolution through the first affine transformation matrix tfm_stage1.

由于第一仿射变换矩阵tfm_stage1中缩放系数修改为1,相当于没进行S变换,因此变换后的人像图片的分辨率没有改变,最终将人像图片通过第一仿射变换矩阵tfm_stage1进行旋转和平移处理,变换为分辨率不变的中间图片。Since the scaling coefficient in the first affine transformation matrix tfm_stage1 is modified to 1, which is equivalent to no S transformation, the resolution of the transformed portrait image does not change. Finally, the portrait image is rotated and translated through the first affine transformation matrix tfm_stage1. Process and transform into an intermediate image with unchanged resolution.

在通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片后(和初始照片的原图相比,与原图对应相同的区域得到分辨率没有改变,整张图片涉及旋转和裁剪,尺寸可能发生一定变化),若变换后(例如旋转裁剪后)照片的图片分辨率大于或等于设定阈值,采用第一插值法进行插值,若图片分辨率小于设定阈值,采用第二插值法进行插值。其中第一插值法例如可以是INTER_CUBIC插值法(三次样条插值);第二插值法例如可以是INTER_AREA(区域插值)插值法。对于分辨率较大的图片,INTER_CUBIC插值法有更好的缩放效果,对于将人脸压缩到分辨率较小的图片,采用INTER_AREA插值法可以避免INTER_CUBIC造成的颗粒感。After transforming the initial photo into an intermediate image with constant resolution through the first affine transformation matrix (compared with the original image of the initial photo, the resolution of the area corresponding to the same as the original image has not changed, the entire image involves rotation and Cropping, the size may change to a certain extent), if the image resolution of the photo after transformation (for example, after rotation and cropping) is greater than or equal to the set threshold, the first interpolation method is used for interpolation. If the image resolution is less than the set threshold, the second interpolation method is used. interpolation method. The first interpolation method may be, for example, the INTER_CUBIC interpolation method (cubic spline interpolation); the second interpolation method may be, for example, the INTER_AREA (area interpolation) interpolation method. For pictures with larger resolutions, the INTER_CUBIC interpolation method has better scaling effects. For compressing faces to pictures with smaller resolutions, the INTER_AREA interpolation method can avoid the graininess caused by INTER_CUBIC.

利用opencv(一个基于Apache2.0许可(开源)发行的跨平台计算机视觉和机器学习软件库)实现仿射变换一般会涉及到warpAffine函数,warpAffine函数可以实现一些简单的重映射。本申请方案,修改了warpAffine函数的使用方式,在先用warpAffine函数对人像图片进行旋转和平移的变换后,然后对图片进行缩放。对于分辨率较大的图片,例如若图片短边大于或等于260,则使用INTER_CUBIC插值法进行插值,若图片短边小于260,则使用INTER_AREA插值法进行插值。Using opencv (a cross-platform computer vision and machine learning software library released based on the Apache 2.0 license (open source)) to implement affine transformation generally involves the warpAffine function, which can implement some simple remapping. This application plan modifies the use of the warpAffine function. After first using the warpAffine function to rotate and translate the portrait image, the image is then scaled. For pictures with larger resolutions, for example, if the short side of the picture is greater than or equal to 260, the INTER_CUBIC interpolation method is used for interpolation. If the short side of the picture is less than 260, the INTER_AREA interpolation method is used for interpolation.

由于本申请将第一仿射变换矩阵tfm_stage1的缩放系数修改为1,对warpAffine函数使用的修改不影响第一仿射变换矩阵tfm_stage1的使用,但会改善第二仿射变换矩阵tfm_stage2应用后的效果。Since this application modifies the scaling coefficient of the first affine transformation matrix tfm_stage1 to 1, the modification to the use of the warpAffine function will not affect the use of the first affine transformation matrix tfm_stage1, but will improve the effect of the second affine transformation matrix tfm_stage2 after application. .

在S202中,将中间图片的人像区域按第一优化算法进行清晰度优化处理。In S202, the portrait area of the middle picture is subjected to definition optimization processing according to the first optimization algorithm.

人脸是图片中比较关注的区域,也称为人像区域,本申请实施例方案单独进行人脸清晰度增强。该步骤S202将中间图片的人像区域按第一优化算法进行清晰度优化处理,第一优化算法例如可以是人脸盲复原算法。The human face is an area of greater concern in the picture, also known as the portrait area. The embodiment of the present application proposes to separately enhance the definition of the human face. In step S202, the portrait area of the middle picture is subjected to sharpness optimization processing according to a first optimization algorithm. The first optimization algorithm may be, for example, a face blindness restoration algorithm.

人脸盲复原指将退化(低分辨率、噪声、模糊、图片压缩等)的人脸照片进行重建,最终得到退化较少、清晰的人脸照片。人脸盲复原主要是借助几何人脸先验信息(关键点,人脸分割,人脸热图)修复人脸。其中,GFPGAN是人脸盲复原算法最先进的模型,它使用了生成人脸先验进行人脸盲复原。本申请可以使用GFPGAN模型进行人脸清晰度优化但不局限于此。Blind face restoration refers to reconstructing face photos that have been degraded (low resolution, noise, blur, image compression, etc.), and finally obtain less degraded, clear face photos. Blind face restoration mainly uses geometric face prior information (key points, face segmentation, face heat map) to repair the face. Among them, GFPGAN is the most advanced model for blind face restoration algorithm. It uses generated face priors for blind face restoration. This application can use the GFPGAN model to optimize face clarity but is not limited to this.

在S203中,将中间图片的背景区域按第二优化算法进行清晰度优化处理。In S203, the background area of the middle picture is subjected to definition optimization processing according to the second optimization algorithm.

本申请实施例方案中,因为人脸区域单独处理优化,因此该步骤主要是对除人脸外的背景清晰度进行增强。本申请可以将中间图片的背景区域按第二优化算法进行清晰度优化处理。第二优化算法例如可以是图片超分辨率算法等。本申请使用超分辨率算法模型例如RealESRGAN模型进行背景清晰度优化。RealESRGAN模型主要通过模拟高分辨率图像变低分辩率过程中的各种“退化”过程,然后让模型看到一张糊图后倒推出来它的高清图。In the embodiment of the present application, because the face area is processed and optimized separately, this step is mainly to enhance the clarity of the background except the face. This application can optimize the definition of the background area of the middle picture according to the second optimization algorithm. The second optimization algorithm may be, for example, a picture super-resolution algorithm or the like. This application uses a super-resolution algorithm model such as the RealESRGAN model to optimize background clarity. The RealESRGAN model mainly simulates various "degradation" processes in the process of high-resolution images becoming low-resolution, and then allows the model to see a blurry image and then deduce its high-definition image.

该步骤S203与步骤S202之间可以分开同时进行,两者之间没有顺序关系。Step S203 and step S202 can be performed separately and simultaneously, and there is no sequential relationship between them.

需说明的是,利用图片超分辨率算法,从低分辨率到高分辨率变换图片,图片包含的信息是由少变多的,如果原图片包含的信息缺失,清晰度降低后,最终进行超分辨率处理后的模型也会受到影响。举例说明,例如原图片为1024x1024,利用本申请方案将图片不经过缩放处理而是直接进行超分辨率处理,得到2048x2048的照片A;采用相关技术将原图片先缩放为512x512的图片,将缩放后的图片再进行超分辨率处理,得到2048x2048的照片B;由于照片B先进行过缩放处理,清晰度会下降,那么采用本申请方案得到的照片A比采用相关技术得到的照片B可以具有更好的清晰度。对于人脸盲复原算法而言,也是同样的道理。It should be noted that by using the image super-resolution algorithm to transform the image from low resolution to high resolution, the information contained in the image changes from less to more. If the information contained in the original image is missing and the clarity is reduced, the super-resolution is finally performed. Resolution-processed models are also affected. For example, for example, if the original picture is 1024x1024, this application solution is used to directly perform super-resolution processing on the picture without scaling, and obtain photo A of 2048x2048; use relevant technology to first scale the original picture to a 512x512 picture, and then scale the The picture is then subjected to super-resolution processing to obtain photo B of 2048x2048; since photo B has been zoomed first, the clarity will decrease, so photo A obtained by using the solution of this application can have better quality than photo B obtained by using related technologies. of clarity. The same is true for face blindness restoration algorithms.

还需说明的是,如果根据不同业务需求需要对中间图片进行其他人像处理逻辑,也可以对中间图片进行相应的处理,本申请实施例不加以限定。It should also be noted that if other portrait processing logic needs to be performed on the intermediate picture according to different business requirements, corresponding processing can also be performed on the intermediate picture, which is not limited by the embodiments of this application.

在S204中,将分别进行清晰度优化处理后的人像区域和背景区域采用泊松融合算法进行融合,得到融合图片。In S204, the portrait area and the background area that have been separately processed for sharpness optimization are fused using a Poisson fusion algorithm to obtain a fused picture.

本申请应用泊松融合算法,将清晰度增强后的人脸贴回到背景清晰度增强后的图片,完成照片清晰度增强,得到处理后的融合图片。This application applies the Poisson fusion algorithm to paste the sharpness-enhanced face back to the background sharpness-enhanced picture to complete the photo sharpness enhancement and obtain the processed fusion picture.

图像融合是图像拼接技术中的关键技术,其原理是通过对拼接图像中重合带的像素进行重新定义计算,实现拼接图像之间的平滑过渡和无缝拼接。基于泊松方程的泊松融合算法,利用两幅图像的梯度场,对重合区域进行引导差值,将图像融合问题变成求解目标图像块梯度场与背景指导梯度场差值的最小化问题,可以取得良好的图像融合效果。Image fusion is a key technology in image splicing technology. Its principle is to achieve smooth transition and seamless splicing between spliced images by redefining and calculating the pixels in the overlapping zones in the spliced images. The Poisson fusion algorithm based on Poisson's equation uses the gradient field of two images to guide the difference in the overlapping area, turning the image fusion problem into a minimization problem of solving the difference between the gradient field of the target image block and the background guidance gradient field. Good image fusion effect can be achieved.

在S205中,通过第二仿射变换矩阵将融合图片变换为指定规格图片。In S205, the fused picture is transformed into a picture of a specified specification through the second affine transformation matrix.

该步骤确定中间图片对齐到指定规格照片的第二仿射变换矩阵tfm_stage2。This step determines the second affine transformation matrix tfm_stage2 of the intermediate picture aligned to the specified specification photo.

第二仿射变换矩阵tfm_stage2与第一仿射变换矩阵tfm_stage1的确定过程的原理是相同的(可参见图4),可以根据中间图片的人脸关键点和指定规格图片的人脸关键点,求得变换矩阵tfm。但是,此时不需要将变换矩阵tfm进行分解运算,也不需要将缩放系数修改为1,只需将得到的变换矩阵tfm直接作为第二仿射变换矩阵tfm_stage2。The principle of the determination process of the second affine transformation matrix tfm_stage2 and the first affine transformation matrix tfm_stage1 is the same (see Figure 4). According to the face key points of the intermediate picture and the face key points of the specified specification picture, Get the transformation matrix tfm. However, at this time, there is no need to decompose the transformation matrix tfm, and there is no need to modify the scaling coefficient to 1. The obtained transformation matrix tfm only needs to be directly used as the second affine transformation matrix tfm_stage2.

该步骤S205通过第二仿射变换矩阵将融合图片变换为指定规格图片,也即对融合处理后得到的融合图片应用仿射变换,得到最终的指定规格的照片。This step S205 uses the second affine transformation matrix to transform the fused picture into a picture of specified specifications, that is, applying affine transformation to the fused picture obtained after the fusion process to obtain the final photo of specified specifications.

综上所描述,本申请实施例的技术方案,通过将仿射变换矩阵的缩放系数修改为1使得初始照片经过旋转和平移校正后,图片的分辨率没有改变,及指定了仿射变换过程中缩放使用的插值方法,可以减少照片处理流程中的清晰度流失。虽然最后采用的是第二次仿射变换后的图片,但在最后一步之前都是使用的第一次仿射变换后的中间图片,该中间图片和初始照片保持了一样的分辨率,从而避免了处理过程中清晰度的流失,提高了整个照片处理后的清晰度。In summary, the technical solution of the embodiment of the present application modifies the scaling coefficient of the affine transformation matrix to 1 so that the resolution of the image does not change after rotation and translation correction of the initial photo, and specifies the affine transformation process. The interpolation method used by scaling can reduce the loss of sharpness in the photo processing pipeline. Although the image after the second affine transformation was used in the end, the intermediate image after the first affine transformation was used before the last step. The intermediate image maintains the same resolution as the initial photo, thereby avoiding It eliminates the loss of clarity during processing and improves the clarity of the entire photo after processing.

与前述应用功能实现方法实施例相对应,本申请实施例还提供了一种照片处理装置、电子设备及相应的实施例。Corresponding to the foregoing application function implementation method embodiments, embodiments of the present application also provide a photo processing device, electronic equipment and corresponding embodiments.

图5是本申请实施例示出的照片处理装置的结构示意图。FIG. 5 is a schematic structural diagram of a photo processing device according to an embodiment of the present application.

参见图5,一种照片处理装置50,包括:第一变换模块51、优化处理模块52、融合处理模块53、第二变换模块54。Referring to FIG. 5 , a photo processing device 50 includes: a first transformation module 51 , an optimization processing module 52 , a fusion processing module 53 , and a second transformation module 54 .

第一变换模块51,用于将初始照片变换为分辨率不变的中间图片。第一变换模块51通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片。其中第一仿射变换矩阵可以按以下方式得到:将初始照片对齐到参考图得到位置坐标的变换参数;根据所述变换参数构建变换矩阵,将所述变换矩阵进行设定处理,得到第一仿射变换矩阵。其中,可以根据变换参数构建平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S;将平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S进行左乘运算,则得到变换矩阵;将变换矩阵进行分解运算,将缩放系数修改为1,得到第一仿射变换矩阵。The first transformation module 51 is used to transform the initial photo into an intermediate picture with unchanged resolution. The first transformation module 51 transforms the initial photo into an intermediate picture with unchanged resolution through a first affine transformation matrix. The first affine transformation matrix can be obtained in the following way: align the initial photo to the reference image to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, set the transformation matrix, and obtain the first affine transformation matrix. radial transformation matrix. Among them, the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameters; the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S are left multiplied to obtain the transformation matrix; the transformation matrix Perform a decomposition operation and modify the scaling coefficient to 1 to obtain the first affine transformation matrix.

优化处理模块52,用于将第一变换模块51变换的中间图片按人像区域和背景区域分别进行清晰度优化处理。优化处理模块52可以将中间图片的人像区域按第一优化算法进行清晰度优化处理;将中间图片的背景区域按第二优化算法进行清晰度优化处理。其中,第一优化算法可以是人脸盲复原算法;第二优化算法可以是图片超分辨率算法。The optimization processing module 52 is configured to perform definition optimization processing on the intermediate pictures transformed by the first transformation module 51 according to the portrait area and the background area respectively. The optimization processing module 52 can optimize the definition of the portrait area of the middle picture according to the first optimization algorithm; and perform the definition optimization processing of the background area of the middle picture according to the second optimization algorithm. Among them, the first optimization algorithm may be a blind face restoration algorithm; the second optimization algorithm may be an image super-resolution algorithm.

融合处理模块53,用于将优化处理模块52分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片。融合处理模块53可以将分别进行清晰度优化处理后的人像区域和背景区域采用泊松融合算法进行融合,得到融合图片。The fusion processing module 53 is used to fuse the portrait area and the background area that have been separately processed for sharpness optimization by the optimization processing module 52 to obtain a fused picture. The fusion processing module 53 can use the Poisson fusion algorithm to fuse the portrait area and the background area that have been separately processed for sharpness optimization to obtain a fused picture.

第二变换模块54,用于将融合处理模块53得到的融合图片变换为指定规格图片。第二变换模块54可以通过第二仿射变换矩阵将融合图片变换为指定规格图片。其中第二仿射变换矩阵可以按以下方式得到:将中间图片对齐到指定规格图片得到位置坐标的变换参数;根据所述变换参数构建变换矩阵,将所述变换矩阵作为第二仿射变换矩阵。其中可以根据变换参数构建平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S;将平移变换矩阵T、旋转变换矩阵R和缩放变换矩阵S进行左乘运算,则得到变换矩阵,将变换矩阵作为第二仿射变换矩阵。The second transformation module 54 is used to transform the fused picture obtained by the fusion processing module 53 into a picture of a specified specification. The second transformation module 54 may transform the fused picture into a picture of a specified specification through a second affine transformation matrix. The second affine transformation matrix can be obtained in the following manner: align the intermediate picture to the specified specification picture to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, and use the transformation matrix as the second affine transformation matrix. Among them, the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S can be constructed according to the transformation parameters; by left-multiplying the translation transformation matrix T, the rotation transformation matrix R and the scaling transformation matrix S, the transformation matrix is obtained, and the transformation matrix is as The second affine transformation matrix.

从该实施例可以看出,相关技术是先将初始照片进行缩放后处理,而缩放处理会导致照片处理流程中的清晰度流失,而本申请的技术方案将初始照片变换为中间图片时是变换为分辨率不变的中间图片,这样可以减少或避免照片处理流程中的清晰度流失。另外,本申请是将中间图片按人像区域和背景区域分别进行清晰度优化处理,然后再将分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片,这样可以根据图片中的不同区域的图像特点进行不同的清晰度优化处理,也进一步减少或避免照片处理流程中的清晰度流失。因此,本申请的技术方案,能够通过优化处理提高照片的清晰度。It can be seen from this embodiment that the related technology is to first scale the initial photo and then process it, and the scaling process will cause the loss of clarity in the photo processing process. However, the technical solution of this application transforms the initial photo into an intermediate picture. An intermediate image with the same resolution, which can reduce or avoid the loss of sharpness in the photo processing process. In addition, this application performs sharpness optimization processing on the portrait area and background area of the middle image respectively, and then fuses the portrait area and background area respectively after the sharpness optimization processing to obtain a fused image, which can be based on the image in the image. The image characteristics of different areas are optimized for different definitions, which further reduces or avoids the loss of definition during the photo processing process. Therefore, the technical solution of this application can improve the clarity of photos through optimization processing.

图6是本申请实施例示出的照片处理装置的另一结构示意图。FIG. 6 is another schematic structural diagram of a photo processing device according to an embodiment of the present application.

参见图6,一种照片处理装置50,包括:第一变换模块51、优化处理模块52、融合处理模块53、第二变换模块54。Referring to FIG. 6 , a photo processing device 50 includes: a first transformation module 51 , an optimization processing module 52 , a fusion processing module 53 , and a second transformation module 54 .

第一变换模块51、优化处理模块52、融合处理模块53、第二变换模块54的功能可以参见图5中的描述。The functions of the first transformation module 51, the optimization processing module 52, the fusion processing module 53, and the second transformation module 54 can be referred to the description in Figure 5.

其中,第一变换模块51可以通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片;第二变换模块54可以通过第二仿射变换矩阵将融合图片变换为指定规格图片。Among them, the first transformation module 51 can transform the initial photo into an intermediate picture with unchanged resolution through the first affine transformation matrix; the second transformation module 54 can transform the fused picture into a specified specification picture through the second affine transformation matrix.

优化处理模块52可以包括:第一优化子模块521、第二优化子模块522。The optimization processing module 52 may include: a first optimization sub-module 521 and a second optimization sub-module 522 .

第一优化子模块521,用于将中间图片的人像区域按第一优化算法进行清晰度优化处理;The first optimization sub-module 521 is used to optimize the definition of the portrait area of the middle picture according to the first optimization algorithm;

第二优化子模块522,用于将中间图片的背景区域按第二优化算法进行清晰度优化处理。The second optimization sub-module 522 is used to optimize the definition of the background area of the middle picture according to the second optimization algorithm.

其中,第一优化算法可以是人脸盲复原算法;第二优化算法可以是图片超分辨率算法。GFPGAN是人脸盲复原算法最先进的模型,它使用了生成人脸先验进行人脸盲复原。本申请可以使用GFPGAN进行人脸清晰度优化但不局限于此。本申请可以使用超分辨率算法模型例如RealESRGAN模型进行背景清晰度优化。Among them, the first optimization algorithm may be a blind face restoration algorithm; the second optimization algorithm may be an image super-resolution algorithm. GFPGAN is the most advanced model of blind face restoration algorithm. It uses generated face priors for blind face restoration. This application can use GFPGAN to optimize face clarity but is not limited to this. This application can use a super-resolution algorithm model such as the RealESRGAN model to optimize background clarity.

关于上述实施例中的装置,其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述,此处将不再做详细阐述说明。Regarding the devices in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

图7是本申请实施例示出的电子设备的结构示意图。FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

参见图7,电子设备1000包括存储器1010和处理器1020。Referring to FIG. 7 , electronic device 1000 includes memory 1010 and processor 1020 .

处理器1020可以是中央处理单元(Central Processing Unit,CPU),还可以是其他通用处理器、数字信号处理器(Digital Signal Processor,DSP)、专用集成电路(Application Specific Integrated Circuit,ASIC)、现场可编程门阵列(Field-Programmable Gate Array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。The processor 1020 can be a central processing unit (CPU), other general-purpose processor, digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or field-processable processor. Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

存储器1010可以包括各种类型的存储单元,例如系统内存、只读存储器(ROM)和永久存储装置。其中,ROM可以存储处理器1020或者计算机的其他模块需要的静态数据或者指令。永久存储装置可以是可读写的存储装置。永久存储装置可以是即使计算机断电后也不会失去存储的指令和数据的非易失性存储设备。在一些实施方式中,永久性存储装置采用大容量存储装置(例如磁或光盘、闪存)作为永久存储装置。另外一些实施方式中,永久性存储装置可以是可移除的存储设备(例如软盘、光驱)。系统内存可以是可读写存储设备或者易失性可读写存储设备,例如动态随机访问内存。系统内存可以存储一些或者所有处理器在运行时需要的指令和数据。此外,存储器1010可以包括任意计算机可读存储媒介的组合,包括各种类型的半导体存储芯片(例如DRAM,SRAM,SDRAM,闪存,可编程只读存储器),磁盘和/或光盘也可以采用。在一些实施方式中,存储器1010可以包括可读和/或写的可移除的存储设备,例如激光唱片(CD)、只读数字多功能光盘(例如DVD-ROM,双层DVD-ROM)、只读蓝光光盘、超密度光盘、闪存卡(例如SD卡、min SD卡、Micro-SD卡等)、磁性软盘等。计算机可读存储媒介不包含载波和通过无线或有线传输的瞬间电子信号。Memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and persistent storage. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. Persistent storage may be readable and writable storage. Persistent storage may be a non-volatile storage device that does not lose stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (eg, magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (eg, floppy disk, optical drive). System memory can be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory can store some or all of the instructions and data the processor needs to run. In addition, memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (eg, DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks, and/or optical disks may also be used. In some embodiments, memory 1010 may include a readable and/or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (eg, DVD-ROM, dual-layer DVD-ROM), Read-only Blu-ray discs, ultra-density discs, flash memory cards (such as SD card, min SD card, Micro-SD card, etc.), magnetic floppy disks, etc. Computer-readable storage media do not contain carrier waves and transient electronic signals that are transmitted wirelessly or wired.

存储器1010上存储有可执行代码,当可执行代码被处理器1020处理时,可以使处理器1020执行上文述及的方法中的部分或全部。The memory 1010 stores executable code. When the executable code is processed by the processor 1020, the processor 1020 can be caused to execute part or all of the above-mentioned methods.

此外,根据本申请的方法还可以实现为一种计算机程序或计算机程序产品,该计算机程序或计算机程序产品包括用于执行本申请的上述方法中部分或全部步骤的计算机程序代码指令。In addition, the method according to the present application can also be implemented as a computer program or computer program product, which computer program or computer program product includes computer program code instructions for executing part or all of the steps in the above method of the present application.

或者,本申请还可以实施为一种计算机可读存储介质(或非暂时性机器可读存储介质或机器可读存储介质),其上存储有可执行代码(或计算机程序或计算机指令代码),当可执行代码(或计算机程序或计算机指令代码)被电子设备(或服务器等)的处理器执行时,使处理器执行根据本申请的上述方法的各个步骤的部分或全部。Alternatively, the application may also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) with executable code (or computer program or computer instruction code) stored thereon, When the executable code (or computer program or computer instruction code) is executed by the processor of the electronic device (or server, etc.), the processor is caused to execute part or all of the respective steps of the above method according to the present application.

以上已经描述了本申请的各实施例,上述说明是示例性的,并非穷尽性的,并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下,对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择,旨在最好地解释各实施例的原理、实际应用或对市场中的技术的改进,或者使本技术领域的其他普通技术人员能理解本文披露的各实施例。The embodiments of the present application have been described above. The above description is illustrative, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (10)

1.一种照片处理方法,其特征在于,包括:1. A photo processing method, characterized in that it includes: 将初始照片变换为分辨率不变的中间图片;Transform the initial photo into an intermediate image with the same resolution; 将所述中间图片按人像区域和背景区域分别进行清晰度优化处理;Perform definition optimization processing on the intermediate image according to the portrait area and background area respectively; 将分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片;Fusion of the portrait area and background area that have been separately processed for sharpness optimization to obtain a fused image; 将所述融合图片变换为指定规格图片。Transform the fused image into a specified specification image. 2.根据权利要求1所述的方法,其特征在于,所述将初始照片变换为分辨率不变的中间图片,包括:2. The method according to claim 1, characterized in that said converting the initial photo into an intermediate picture with unchanged resolution includes: 通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片。The initial photo is transformed into an intermediate image with constant resolution through the first affine transformation matrix. 3.根据权利要求2所述的方法,其特征在于,所述第一仿射变换矩阵按以下方式得到:3. The method according to claim 2, characterized in that the first affine transformation matrix is obtained in the following manner: 将所述初始照片对齐到参考图得到位置坐标的变换参数;Align the initial photo to the reference image to obtain the transformation parameters of the position coordinates; 根据所述变换参数构建变换矩阵,将所述变换矩阵进行设定处理,得到第一仿射变换矩阵。A transformation matrix is constructed according to the transformation parameters, and the transformation matrix is set to obtain a first affine transformation matrix. 4.根据权利要求2所述的方法,其特征在于:4. The method according to claim 2, characterized in that: 所述通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片后,若图片分辨率大于或等于设定阈值,采用第一插值法进行插值,若图片分辨率小于所述设定阈值,采用第二插值法进行插值。After the initial photo is transformed into an intermediate picture with constant resolution through the first affine transformation matrix, if the picture resolution is greater than or equal to the set threshold, the first interpolation method is used for interpolation. If the picture resolution is less than the set threshold, Set the threshold and use the second interpolation method for interpolation. 5.根据权利要求1所述的方法,其特征在于,所述将所述中间图片按人像区域和背景区域分别进行清晰度优化处理,包括:5. The method according to claim 1, wherein the step of optimizing the definition of the intermediate picture according to the portrait area and the background area includes: 将所述中间图片的人像区域按第一优化算法进行清晰度优化处理;Perform sharpness optimization processing on the portrait area of the intermediate picture according to the first optimization algorithm; 将所述中间图片的背景区域按第二优化算法进行清晰度优化处理。The background area of the intermediate picture is subjected to definition optimization processing according to the second optimization algorithm. 6.根据权利要求1所述的方法,其特征在于,所述将所述融合图片变换为指定规格图片,包括:6. The method according to claim 1, characterized in that said converting the fused picture into a specified specification picture includes: 通过第二仿射变换矩阵将所述融合图片变换为指定规格图片;Transform the fused image into a specified specification image through a second affine transformation matrix; 其中所述第二仿射变换矩阵按以下方式得到:将所述中间图片对齐到所述指定规格图片得到位置坐标的变换参数;根据所述变换参数构建变换矩阵,将所述变换矩阵作为第二仿射变换矩阵。The second affine transformation matrix is obtained in the following manner: align the intermediate picture to the specified specification picture to obtain the transformation parameters of the position coordinates; construct a transformation matrix according to the transformation parameters, and use the transformation matrix as the second Affine transformation matrix. 7.一种照片处理装置,其特征在于,包括:7. A photo processing device, characterized by comprising: 第一变换模块,用于将初始照片变换为分辨率不变的中间图片;The first transformation module is used to transform the initial photo into an intermediate image with unchanged resolution; 优化处理模块,用于将所述第一变换模块变换的中间图片按人像区域和背景区域分别进行清晰度优化处理;An optimization processing module, configured to optimize the definition of the intermediate picture transformed by the first transformation module according to the portrait area and the background area respectively; 融合处理模块,用于将所述优化处理模块分别进行清晰度优化处理后的人像区域和背景区域进行融合,得到融合图片;A fusion processing module, used to fuse the portrait area and the background area after the definition optimization processing by the optimization processing module, respectively, to obtain a fused picture; 第二变换模块,用于将所述融合处理模块得到的融合图片变换为指定规格图片。The second transformation module is used to transform the fused picture obtained by the fusion processing module into a picture of specified specifications. 8.根据权利要求7所述的装置,其特征在于:8. The device according to claim 7, characterized in that: 所述第一变换模块通过第一仿射变换矩阵将初始照片变换为分辨率不变的中间图片;或,The first transformation module transforms the initial photo into an intermediate picture with constant resolution through a first affine transformation matrix; or, 所述第二变换模块通过第二仿射变换矩阵将所述融合图片变换为指定规格图片。The second transformation module transforms the fused picture into a picture of a specified specification through a second affine transformation matrix. 9.一种电子设备,其特征在于,包括:9. An electronic device, characterized in that it includes: 处理器;以及processor; and 存储器,其上存储有可执行代码,当所述可执行代码被所述处理器执行时,使所述处理器执行如权利要求1-6中任一项所述的方法。A memory having executable code stored thereon, which when executed by the processor causes the processor to perform the method of any one of claims 1-6. 10.一种计算机可读存储介质,其上存储有可执行代码,当所述可执行代码被电子设备的处理器执行时,使所述处理器执行如权利要求1-6中任一项所述的方法。10. A computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute as claimed in any one of claims 1-6. method described.
CN202210355149.6A 2022-03-31 2022-03-31 Photo processing method, device and equipment Pending CN116934637A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210355149.6A CN116934637A (en) 2022-03-31 2022-03-31 Photo processing method, device and equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210355149.6A CN116934637A (en) 2022-03-31 2022-03-31 Photo processing method, device and equipment

Publications (1)

Publication Number Publication Date
CN116934637A true CN116934637A (en) 2023-10-24

Family

ID=88374305

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210355149.6A Pending CN116934637A (en) 2022-03-31 2022-03-31 Photo processing method, device and equipment

Country Status (1)

Country Link
CN (1) CN116934637A (en)

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102496158A (en) * 2011-11-24 2012-06-13 中兴通讯股份有限公司 Method and device for image information processing
CN107330854A (en) * 2017-06-15 2017-11-07 武汉大学 A kind of image super-resolution Enhancement Method based on new type formwork
CN109711364A (en) * 2018-12-29 2019-05-03 成都视观天下科技有限公司 A kind of facial image super-resolution reconstruction method, device and computer equipment
WO2019104705A1 (en) * 2017-12-01 2019-06-06 华为技术有限公司 Image processing method and device
CN110490796A (en) * 2019-04-11 2019-11-22 福建师范大学 A kind of human face super-resolution processing method and system of the fusion of low-and high-frequency ingredient
CN110992268A (en) * 2019-12-06 2020-04-10 广州酷狗计算机科技有限公司 Background setting method, device, terminal and storage medium
CN111062868A (en) * 2019-12-03 2020-04-24 广州极泽科技有限公司 Image processing method, device, machine readable medium and equipment
CN111915484A (en) * 2020-07-06 2020-11-10 天津大学 Reference image guiding super-resolution method based on dense matching and self-adaptive fusion
WO2021000841A1 (en) * 2019-06-30 2021-01-07 华为技术有限公司 Method for generating user profile photo, and electronic device
CN112836545A (en) * 2019-11-22 2021-05-25 北京新氧科技有限公司 A 3D face information processing method, device and terminal
CN113673474A (en) * 2021-08-31 2021-11-19 Oppo广东移动通信有限公司 Image processing method, image processing device, electronic equipment and computer readable storage medium
CN113947792A (en) * 2021-10-15 2022-01-18 广州华多网络科技有限公司 Target face image matching method and its device, equipment, medium and product

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102496158A (en) * 2011-11-24 2012-06-13 中兴通讯股份有限公司 Method and device for image information processing
CN107330854A (en) * 2017-06-15 2017-11-07 武汉大学 A kind of image super-resolution Enhancement Method based on new type formwork
WO2019104705A1 (en) * 2017-12-01 2019-06-06 华为技术有限公司 Image processing method and device
CN109711364A (en) * 2018-12-29 2019-05-03 成都视观天下科技有限公司 A kind of facial image super-resolution reconstruction method, device and computer equipment
CN110490796A (en) * 2019-04-11 2019-11-22 福建师范大学 A kind of human face super-resolution processing method and system of the fusion of low-and high-frequency ingredient
WO2021000841A1 (en) * 2019-06-30 2021-01-07 华为技术有限公司 Method for generating user profile photo, and electronic device
CN112836545A (en) * 2019-11-22 2021-05-25 北京新氧科技有限公司 A 3D face information processing method, device and terminal
CN111062868A (en) * 2019-12-03 2020-04-24 广州极泽科技有限公司 Image processing method, device, machine readable medium and equipment
CN110992268A (en) * 2019-12-06 2020-04-10 广州酷狗计算机科技有限公司 Background setting method, device, terminal and storage medium
CN111915484A (en) * 2020-07-06 2020-11-10 天津大学 Reference image guiding super-resolution method based on dense matching and self-adaptive fusion
CN113673474A (en) * 2021-08-31 2021-11-19 Oppo广东移动通信有限公司 Image processing method, image processing device, electronic equipment and computer readable storage medium
CN113947792A (en) * 2021-10-15 2022-01-18 广州华多网络科技有限公司 Target face image matching method and its device, equipment, medium and product

Similar Documents

Publication Publication Date Title
CN109859097B (en) Face image processing method, device, image processing device, and medium
CN109753971B (en) Correction method and device for distorted text lines, character recognition method and device
CN107403415B (en) Method and device for quality enhancement of compressed depth map based on fully convolutional neural network
CN111724313B (en) Shadow map generation method and device
CN108961170A (en) Image processing method, device and system
US20180322669A1 (en) Virtual reality image compositing
CN113160079A (en) Portrait restoration model training method, portrait restoration method and device
US10275863B2 (en) Homography rectification
CN118096601A (en) Image restoration method and system based on wavelet transformation and multi-scale residual error network
CN119048356A (en) Video resolution enhancement system and method based on pre-training video generation model
WO2021051948A1 (en) Training method and device for document recognition based on neural network
CN116071279A (en) Image processing method, device, computer equipment and storage medium
CN115423697A (en) Image restoration method, terminal and computer storage medium
CN116934637A (en) Photo processing method, device and equipment
CN110660013B (en) Semiconductor device, image recognition system and image processing method
CN118736087A (en) A method and system for high-precision dynamic face reconstruction and high-resolution texture generation
CN114897732B (en) Image defogging method and device based on association of physical model and feature density
Zheng et al. Joint residual pyramid for joint image super-resolution
CN113784101B (en) Purple fringing correction method and purple fringing correction device
CN114612291A (en) Face image processing method and device
CN116934615A (en) Image restoration method, device, electronic equipment and storage medium
CN116503263A (en) Rendering image noise reduction method, model rendering method, device, equipment and storage medium
CN113362351B (en) Image processing method, device, electronic device and storage medium
CN115375539A (en) Image resolution enhancement, multi-frame image super-resolution system and method
US9019309B2 (en) Adaptive image manipulation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20240509

Address after: 100102 room 1201, 12 / F, building 8, yard 34, Chuangyuan Road, Chaoyang District, Beijing

Applicant after: Beijing new oxygen world wide Technology Consulting Co.,Ltd.

Country or region after: China

Address before: 100102 room 901, 9 / F, room 1001, 10 / F, building 8, yard 34, Chuangyuan Road, Chaoyang District, Beijing

Applicant before: Beijing New Oxygen Technology Co.,Ltd.

Country or region before: China

TA01 Transfer of patent application right