TWI723123B - Image fusion method, device and equipment - Google Patents

Abstract

The present invention provides a method, device and equipment for image fusion, wherein the method includes: determining a fusion area in an image to obtain a first template; down-sampling the first template to obtain a second template; Normalize the pixel value of each pixel in to obtain a third template; perform up-sampling on the third template to obtain a fourth template, the number of pixels in the fourth template is equal to the number of pixels in the first template Number; the pixel value of each pixel in the fourth template is used as the weight of the corresponding pixel in the image, and each pixel in the fusion area of the image and the fusion material are weighted and fused. In addition, before the normalization, it may also include the steps of using a predefined smoothing template to smooth the edges of the second template and adjusting the brightness of the second template based on the brightness of the fusion area in the image. The invention can reduce the calculation amount of image fusion, and reduce time cost and resource consumption.

Description

Image fusion method, device and equipment

The present invention relates to the technical field of computer image processing, in particular to an image fusion method, device and equipment.

With the continuous popularization of smart terminals, people's demand for using smart terminals for image processing is increasing, and various beauty apps are widely favored by beauty lovers. In this type of APP, image fusion processing is often involved. The existing image fusion processing is more complicated, so that when the pixel area involved in image fusion is large, the time cost caused by the amount of calculation is very large. The immediacy is difficult to guarantee, and it consumes and occupies a lot of system resources.

In view of this, the present invention provides a method, device and equipment for image fusion, so as to reduce the calculation amount of image fusion, and reduce time cost and resource consumption.

The specific technical solutions are as follows: the present invention provides a method for image fusion, which includes: determining a fusion area in an image to obtain a first template; down-sampling the first template to obtain a second template; The pixel value of each pixel in the template is normalized to obtain a third template; the third template is up-sampled to obtain a fourth template, the number of pixels of the fourth template is equal to the number of pixels of the first template Number; the pixel value of each pixel in the fourth template is used as the weight of the corresponding pixel in the image, and each pixel in the fusion area in the image and the fusion material are weighted and fused.

According to a preferred embodiment of the present invention, the determining the fusion area in the image to obtain the first template includes: locating feature points of the fusion target in the image, and the feature points include contour points; using the located feature points to remove In the image except for the fusion target area, the first template is obtained.

倍，該N為2以上的正整數；對該第三模板進行升採樣，得到第四模板包括：採用逆仿射變換的方式，對該第三模板進行升採樣，使得到的第四模板的像素點個數為該第三模板的N倍。 According to a preferred embodiment of the present invention, down-sampling the first template to obtain the second template includes: using an affine transformation method to down-sample the first template so that the number of pixels of the second template is obtained For the first template

Times, the N is a positive integer greater than 2; up-sampling the third template to obtain the fourth template includes: using inverse affine transformation, up-sampling the third template, so that the obtained fourth template The number of pixels is N times of the third template.

According to a preferred embodiment of the present invention, before normalizing the pixel value of each pixel in the second template, the method further includes: smoothing the edge of the second template.

According to a preferred embodiment of the present invention, the smoothing of the edge of the second template includes: extending the contour points of the fusion area in the second template outward and/or inward by M pixels, where M is a preset A positive integer of, the area surrounded by the extended pixels is taken as the area to be smoothed; the predefined smoothing template is affine to the area to be smoothed to obtain a smoothed second template.

According to a preferred embodiment of the present invention, the affine the predefined smoothing template to the area to be smoothed includes: affine the pixel value of the pixel in the smoothing template to the pixel value of the pixel at the corresponding position in the area to be smoothed .

According to a preferred embodiment of the present invention, the affine the predefined smoothing template to the area to be smoothed includes: the smoothing area on the smoothing template and the area to be smoothed on the second template are respectively triangulated in the same way And obtain the same number of triangular areas; each triangular area in the smooth template is affine respectively to the triangular area at the corresponding position in the second template.

According to a preferred embodiment of the present invention, before normalizing the pixel value of each pixel in the second template, the method further includes: performing brightness statistics on the fusion area in the image; and according to the brightness statistics result, Adjust the brightness of the smoothed second template.

According to a preferred embodiment of the present invention, the brightness adjustment of the smoothed second template based on the brightness statistics includes: determining the average brightness of the fusion area in the image and the average brightness of the smoothed second template. The difference between the two; the brightness value of each pixel in the second template after the smoothing process is respectively added to the difference.

According to a preferred embodiment of the present invention, performing weighted fusion of each pixel in the fusion area and the fusion material in the image includes: using Image _{i_new} =weight_mask _i * Image _{i_old} +(1-weight_mask _i )*Color _i to determine the result after fusion The pixel value of each pixel; where Image _{i_new} is the pixel value of the i-th pixel obtained after the fusion area in the image is fused, weight_mask _{i is} the pixel value of the i-th pixel in the fourth template, Image _{i_old} Is the pixel value of the i-th pixel in the fusion area of the image, and Color _i is the pixel value of the i-th pixel provided by the fusion material.

According to a preferred embodiment of the present invention, the method is applied to a beauty APP; the fusion area is a face area; and the fusion material is a foundation color.

The present invention also provides an image fusion device, which includes: a template determining unit for determining a fusion area in the image to obtain a first template; a down-sampling unit for down-sampling the first template, Obtain the second template; a normalization unit for normalizing the pixel value of each pixel in the second template to obtain the third template; up-sampling unit for up-sampling the third template to obtain The fourth template, the number of pixels in the fourth template is equal to the number of pixels in the first template; a weighted fusion unit is used for the pixel value of each pixel in the fourth template as the corresponding pixel in the image The weight of the pixel points, and each pixel point and the fusion material in the fusion area in the image are weighted and fused.

倍，該N為2以上的正整數；該升採樣單元，具體用於採用逆仿射變換的方式，對該第三模板進行升採樣，使得到的第四模板的像素點個數為該第三模板的N倍。 According to a preferred embodiment of the present invention, the template determining unit is specifically used to: perform feature point location on the fusion target in the image, the feature point includes contour points; use the located feature points to remove the fusion target from the image Outside the area, get the first template. According to a preferred embodiment of the present invention, the down-sampling unit is specifically used to down-sample the first template by means of affine transformation, so that the number of pixels of the second template obtained is that of the first template.

Times, the N is a positive integer greater than 2; the up-sampling unit is specifically used to up-sample the third template by means of inverse affine transformation, so that the number of pixels of the fourth template obtained is the first N times the three templates.

According to a preferred embodiment of the present invention, the device further includes: an edge smoothing unit for smoothing the edges of the second template, and outputting the smoothed second template to the normalization unit.

According to a preferred embodiment of the present invention, the edge smoothing unit is specifically configured to extend the contour points of the fusion area in the second template outward and/or inward respectively by M pixels, where M is a preset positive integer , Taking the area surrounded by the expanded pixels as the area to be smoothed; affine a predefined smoothing template to the area to be smoothed to obtain a smoothed second template.

According to a preferred embodiment of the present invention, when the edge smoothing unit affines the predefined smoothing template to the area to be smoothed, it specifically executes: affine the pixel values of the pixels in the smoothing template to be in the area to be smoothed The pixel value of the pixel at the corresponding position.

According to a preferred embodiment of the present invention, when the edge smoothing unit affines the predefined smoothing template to the area to be smoothed, it specifically executes: the smoothing area on the smoothing template and the area to be smoothed on the second template respectively Triangulation is performed in the same way to obtain the same number of triangular areas; each triangular area in the smooth template is affineed to the triangular area at the corresponding position in the second template.

According to a preferred embodiment of the present invention, the device further includes: a brightness adjustment unit, configured to obtain a second template output by the edge smoothing unit, perform brightness statistics on the fusion region in the image, and perform brightness statistics on the obtained The second template performs brightness adjustment, and the second template after the brightness adjustment is output to the normalization unit.

According to a preferred embodiment of the present invention, when the brightness adjustment unit adjusts the brightness of the smoothed second template according to the brightness statistics, it specifically executes: determining the average brightness of the fusion area in the image and the smoothed The difference between the mean brightness values of the second template; the brightness value of each pixel in the second template after the smoothing process is respectively added to the difference.

According to a preferred embodiment of the present invention, the weighted fusion unit is specifically used to: use Image _{i_new} = weight_mask _i * Image _{i_old} +(1-weight_mask _i )*Color _i to determine the pixel value of each pixel point obtained after fusion; where Image _{i_new} is the pixel value of the i-th pixel obtained after the fusion area in the image is fused, weight_mask _{i is} the pixel value of the i-th pixel in the fourth template, Image _{i_old} is the i-th pixel in the fusion area in the image The pixel value of each pixel, Color _i is the pixel value of the i-th pixel provided by the fusion material.

According to a preferred embodiment of the present invention, the device is applied to a beauty APP; the fusion area is a face area; and the fusion material is a foundation color.

The present invention also provides a device, including one or more processors; memory; one or more programs, the one or more programs are stored in the memory and executed by the one or more processors to achieve the following Operation: Determine the fusion area in the image to obtain the first template; down-sample the first template to obtain the second template; normalize the pixel value of each pixel in the second template to obtain the third template ; Up-sampling the third template to obtain a fourth template, the number of pixels in the fourth template is equal to the number of pixels in the first template; the pixel value of each pixel in the fourth template is used as the The weight of the corresponding pixel in the image is weighted and fused with each pixel in the fusion area of the image and the fusion material.

It can be seen from the above technical solutions that the present invention adopts the method of down-sampling the image to calculate the weight of the down-sampled fusion area, and then up-sample it back to the original image size to obtain the pixels of the fusion area in the original image. The weights corresponding to the points during fusion greatly reduce the amount of calculation caused by the weight calculation, and reduce the time cost and resource consumption.

01‧‧‧Template determination unit

02‧‧‧Downsampling unit

03‧‧‧Normalization unit

04‧‧‧Upsampling unit

05‧‧‧Weighted Fusion Unit

06‧‧‧Edge smoothing unit

07‧‧‧Brightness adjustment unit

Fig. 1 is a flowchart of the main method provided by an embodiment of the present invention; Fig. 2 is a flowchart of a detailed method provided by an embodiment of the present invention; Fig. 3a is a schematic diagram of a face image provided by an embodiment of the present invention; Fig. 3a is a schematic diagram of feature point positioning; Fig. 3c is a schematic diagram of the first template area obtained based on Fig. 3b; Fig. 3d is the area to be smoothed generated based on the face contour in Fig. 3c; Fig. 3e is the use of triangulation combined with a smoothing template A schematic diagram of smoothing; FIG. 4 is a structure diagram of an apparatus provided by an embodiment of the present invention; FIG. 5 is a structure diagram of an apparatus provided by an embodiment of the present invention.

In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following describes the present invention in detail with reference to the drawings and specific embodiments.

The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms of "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

It should be understood that the term "and/or" used in this text is only an association relationship describing related objects, which means that there can be three relationships, for example, A and/or B can mean that A alone exists, and A and B exist at the same time. There are three cases of B alone. In addition, the character "/" in this text generally indicates that the associated objects before and after are in an "or" relationship.

Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determination" or "in response to detection". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined" or "response to determination" or "when detected (stated condition or event)" Event)" or "response to detection (statement or event)".

Fig. 1 is a flowchart of the main method provided by an embodiment of the present invention. As shown in Fig. 1, the method mainly includes the following steps:

In 101, the fusion area in the image is determined, and the first template is obtained.

The image involved in this step is the image that needs to be fused, and the fusion area in the image refers to the area that needs to be fused. The fusion area may be a designated destination area, or a destination area determined through feature point positioning, which will be detailed in subsequent embodiments. This step is actually to cut out the fusion area in the image to obtain the first template, which can be obtained by removing the area outside the fusion target in the image.

In 102, the first template is down-sampled to obtain the second template.

In order to reduce the amount of calculation generated by the fusion processing on the fusion region, in this step, the first template may be down-sampling processing, that is, the number of pixels in the first template can be reduced.

倍，該N為2以上的正整數。 There are many ways to down-sampling an image, such as nearest neighbor down-sampling, B-spline down-sampling and so on. In the embodiment of the present invention, the method of affine transformation may be adopted, for example, the method of scaling and transforming the first template is adopted, and the affine parameters are set so that the number of pixels of the second template obtained is that of the first template.

Times, the N is a positive integer of 2 or more.

Since whether the image fusion is natural or not is usually reflected at the edge of the fusion area, preferably, in order to make the image fusion more natural, the edge of the second template may be further smoothed. There are also many ways to smooth the edges of an image, such as simple blur, Gaussian blur, median filter, and Gaussian filter. In the embodiment of the present invention, a predefined smoothing template may be used to smooth the edge of the second template, which will be described in detail in subsequent embodiments.

In addition, in order to reduce the impact of the difference between the brightness of the original image and the brightness of the smoothed second template, brightness statistics can be performed on the fusion area in the original image, and the smoothed second template can be calculated according to the brightness statistics results. The template performs brightness adjustment, and the specific adjustment method will be described in detail in subsequent embodiments.

In 103, the pixel value of each pixel in the second template is normalized to obtain the third template.

Determining the third template is actually determining the fusion weight used by each pixel in subsequent image fusion. During the fusion, in order to reflect the characteristics of each pixel of the image, the weight coefficient is embodied by the pixel value of each pixel in the second template. In this step, the pixel value of each pixel is normalized.

In 104, the third template is up-sampled to obtain the fourth template, and the number of pixels of the fourth template is equal to the number of pixels of the first template.

In the image fusion process, the amount of calculation is mainly reflected in the process of determining the fusion weight. After the weight is obtained by downsampling, the weight needs to be up-sampled to obtain the weight of each pixel corresponding to the fusion area in the original image. Therefore, in this step, the third template containing weight information is up-sampled to obtain the fourth template.

There are also many ways of upsampling, such as bilateral filtering, guided filtering, and bidirectional interpolation. In the embodiment of the present invention, inverse affine transformation can be used, that is, the third template is inversely affine transformed by using the affine parameters used in the affine transformation in step 102 to obtain the number of pixels and the first template. The same fourth template.

In 105, the pixel value of each pixel in the fourth template is used as the weight of the corresponding pixel in the image, and each pixel in the fusion area in the image and the fusion material are weighted and fused.

In the embodiment of the present invention, a weighted fusion method is adopted. The following formula can be used to determine the pixel value of each pixel obtained after fusion: Image _{i_new} =weight_mask _i * Image _{i_old} +(1-weight_mask _i )*Color _i (1 )

Where Image _{i_new} is the pixel value of the i-th pixel obtained after the fusion area in the image is fused, weight_mask _i is the pixel value of the i-th pixel in the fourth template, Image _{i_old} is the i-th pixel in the fusion area in the image The pixel value of the point, Color _i is the pixel value of the i-th pixel provided by the fusion material. In the embodiment of the present invention, the fusion material may be an image, one or more of a set of colors, and so on.

The method provided by the present invention can be applied to the fusion processing of static images. Since the calculation amount is greatly reduced and the immediacy can be ensured, it can also be applied to the fusion processing of film images. In addition, the execution subject of the above-mentioned method provided by the present invention can be an application in a user terminal, a plug-in program or a software development kit (SDK) and other functional units in a user terminal application, or it can be It is located on the server side, which is not particularly limited in the embodiment of the present invention. The above-mentioned applications may be, for example, image processing applications, beauty applications, and so on. The above method will be described in detail below with reference to FIG. 2, taking the foundation test on a face in a beauty application as an example.

FIG. 2 is a detailed flow chart of a method provided by an embodiment of the present invention. In this embodiment, a foundation test on the face in the image is implemented, that is, the face area in the image and the foundation color are merged. As shown in Figure 2, the process may specifically include the following steps:

In 201, feature point positioning is performed on the face area in the image to obtain contour points of the face and preset contour points of the organs.

In the embodiment of the present invention, the specific method of feature point positioning is not limited. Any feature point positioning method such as SDM (Supervised Descent Method) model-based positioning, id-exp model positioning, etc. can be used. Obtain the position information of the above-mentioned feature points.

Assuming that the feature points of the face region in Figure 3a are located, the feature points shown in Figure 3b can be obtained, namely the contour points of the face and the contour points of the eyes, eyebrows, and mouth. It should be noted that FIG. 3b is only schematic. In order to facilitate viewing the effect of exaggerating the feature points, in actual feature point positioning, the number and fineness of the located feature points may be inconsistent with those in FIG. 3b.

In 202, using the located feature points, remove the area except the face area in the image to obtain the first template.

In this embodiment, the area outside the contour points of the human face is actually removed, as well as the area surrounded by the contour points of the eyes, eyebrows, and mouth. The obtained first template area is schematically shown as the white area in FIG. 3c, and the actual pixel value of each pixel is not shown in FIG. 3c.

倍。 In 203, the first template is down-sampled by means of affine transformation, so that the number of pixels of the second template obtained is that of the first template.

Times.

倍，例如縮小至原來的

倍，在縮小的過程中，採樣點即 像素點的個數也減少至原來的

倍，也就是通常所說的圖像像素變少了。 The so-called affine transformation is a linear transformation from two-dimensional coordinates to two-dimensional coordinates. Affine transformation can be achieved through a series of atomic transformations, including: Translation, Scale, Flip, Rotation and Shear. Among them, the affine transformation involved in the embodiment of the present invention is the scaling transformation, and an appropriate scaling parameter (ie, affine parameter) is set, and the first template is reduced to the original

Times, for example, reduced to the original

In the process of shrinking, the number of sampling points, that is, the number of pixels, is also reduced to the original

Times, that is, the pixels of the image as it is commonly referred to are reduced.

Among them, the larger the value of N, the smaller the amount of calculation generated, and the smaller the value of N, the higher the quality of image processing, which can be measured and selected according to actual needs.

In 204, the contour points in the second template are respectively expanded outward and inward by M pixel points to obtain the area to be smoothed.

This step actually prepares for the smoothing of the edges of the image and determines the area to be smoothed. In this step, the contour points of the face, the contour points of the eyes, the contour points of the eyebrows, and the contour points of the mouth in the second template can be expanded inward and/or outward respectively by M pixels, where M is a preset A positive integer, such as 3, respectively obtains the strip-shaped area to be smoothed. The so-called inward and outward expansion can be the expansion in two directions along the normal line of the line connecting the contour points.

As shown schematically in Fig. 3d, Fig. 3d only shows the area to be smoothed generated by the contour of the face, and the contours of the eye, eyebrows, and mouth have similar areas to be smoothed.

In 205, a triangulation method is used to affine a predefined face smoothing template to the area to be smoothed to obtain a smoothed second template.

The purpose of smoothing the edges of the face is to smoothly change the brightness of the face in the image, reduce the sudden change gradient, and make the edges of the face softer and more natural, thereby improving the image quality.

In order to speed up the smoothing process, a smoothing template method is adopted in this embodiment. Since the general shape of the human face is basically the same, the shape of the human face can be used in advance to form a template with smoothed edges. In this template, the smoothed edge area can be appropriately larger. When the second template is smoothed, the predefined smoothing template can be affine to the area to be smoothed, so as to complete the smoothing of the area to be smoothed. In this way, edge smoothing can be quickly achieved, and the amount of calculation and time overhead caused by instant blur smoothing is avoided.

In the triangulation method used in this step, the smooth area on the smoothing template and the area to be smoothed on the second template are triangulated in the same way to obtain the same number of triangular areas. Taking the area to be smoothed as an example, the inner and outer edges are equally divided into m points, and then the area to be smoothed is divided into 2m triangles, where m is a preset positive integer, as shown in Figure 3e. The smooth area on the smooth template adopts the same division method, and each triangle area in the smooth template is respectively affine to the triangle area at the corresponding position in the second template.

Among them, the affine involved in this step refers to the affine of the pixel value of the pixel at the corresponding position, that is, the pixel value of the pixel on the smoothing template is affine to the pixel at the corresponding position in the area to be smoothed on the second template, for example Affine the pixel point A of the smooth area on the smoothing template to the pixel point a of the area to be smoothed on the second template, then the pixel point a takes the pixel value of the pixel point A, if the pixel point A is the vertex of a certain triangle obtained by splitting Angle, then the pixel point a is the apex angle of the triangle at the corresponding position on the second template.

In 206, brightness statistics are performed on the face area in the image, and brightness adjustment is performed on the smoothed second template according to the brightness statistics result.

The purpose of this step is to make the brightness of the smoothed second template match the actual brightness of the face in the original image as much as possible. For example, you can count the average value of the brightness of the face area in the original image and the average value of the brightness of the smoothed second template to determine the difference; then add the brightness value of each pixel in each smoothed second template Above the difference. Of course, other specific methods of brightness adjustment can also be used, which will not be listed here.

In 207, the pixel values in the second template after the brightness adjustment are normalized to obtain the third template.

In order to reflect the characteristics of each pixel in the image during image fusion, the weight coefficient is reflected by the pixel value of each pixel in the second template. In this step, the pixel value of each pixel is normalized. The pixel value of each pixel in the third template obtained can be used as the weight of the pixel in the subsequent fusion.

The calculation of weights is one of the important processes that bring about a large amount of calculation in image fusion. In the embodiment of the present invention, firstly, by down-sampling, the weight is calculated based on a template with fewer pixels, and then the up-sampling is performed. , Get the weights of all pixels, compared to directly calculate the weights based on the original image size, which greatly reduces the computational consumption and time cost.

In 208, the third template is subjected to inverse affine transformation using the affine parameters used in 203 to obtain the fourth template.

The scaling transformation is adopted in 203, and the scaling transformation is also adopted in this step. However, in this step, the corresponding affine parameters need to be set according to the affine parameters set in 203, so as to realize the inverse transformation. The number of pixels of the fourth template obtained after the inverse affine transformation is increased to the same as that of the first template. In other words, this step is actually to upsample the number of pixels back to the first template to improve the resolution. In the process of inverse affine transformation, as the number of pixels increases, the pixel value of the increased pixel can be obtained by interpolation.

In 209, the pixel value of each pixel in the fourth template is used as the weight of the corresponding pixel in the image, and each pixel in the face area in the image and the foundation color are weighted and merged.

When performing weighted fusion in this step, the above formula (1) can be used, where Image _{i_new} is the pixel value of the i-th pixel obtained after the fusion of the face area in the image, and Image _{i_old} is the face area in the image ( The area that does not include eyebrows, eyes, and mouth) is the pixel value of the i-th pixel, and Color _i is the pixel value of the foundation color. In this embodiment, the pixel value of the foundation color of each pixel can take the same value.

The pixel value of the pixel involved in the embodiment of the present invention involves the values of the three channels of R, G, and B. When performing the above-mentioned affine and fusion processing, it is necessary to separately analyze the values of the three channels of R, G, and B of each pixel. The values are processed separately, which is a relatively well-known content and is only briefly described here.

The above is a detailed description of the method provided by the present invention, and the device provided by the present invention will be described in detail below with reference to FIG. 4. As shown in FIG. 4, the device may include: a template determination unit 01, a down-sampling unit 02, a normalization unit 03, an up-sampling unit 04, and a weighted fusion unit 05, and may further include an edge smoothing unit 06 and a brightness adjustment unit 07 . The main functions of each component are as follows:

The template determining unit 01 is responsible for determining the fusion area in the image to obtain the first template. Specifically, the template determination unit 01 may first locate the feature points of the fusion target in the image, and the feature points include contour points; then use the located feature points to remove the area except the fusion target in the image to obtain the first template.

倍，N為2以上的正整數。 The down-sampling unit 02 is responsible for down-sampling the first template to obtain the second template. There are many ways to down-sampling an image, such as nearest neighbor down-sampling, B-spline down-sampling and so on. In the embodiment of the present invention, the down-sampling unit 02 uses an affine transformation method to down-sample the first template so that the number of pixels of the second template obtained is that of the first template.

Times, N is a positive integer of 2 or more.

The normalization unit 03 is responsible for normalizing the pixel value of each pixel in the second template to obtain the third template.

The up-sampling unit 04 is responsible for up-sampling the third template to obtain the fourth template. The number of pixels in the fourth template is equal to the number of pixels in the first template. There are also many ways of upsampling, such as bilateral filtering, guided filtering, and bidirectional interpolation. In the embodiment of the present invention, the up-sampling unit 04 may use inverse affine transformation to up-sample the third template, so that the number of pixels of the fourth template obtained is N times that of the third template.

The weighted fusion unit 05 is responsible for taking the pixel value of each pixel in the fourth template as the weight of the corresponding pixel in the image, and performing weighted fusion on each pixel of the fusion area in the image and the fusion material.

_{Specifically, Image i_new} =weight_mask _i * Image _{i_old} +(1-weight_mask _i )*Color _i can be used to determine the pixel value of each pixel obtained after fusion; where Image _{i_new} is the i-th one obtained after fusion of the fused area in the image The pixel value of the pixel, weight_mask _i is the pixel value of the i-th pixel in the fourth template, Image _{i_old} is the pixel value of the i-th pixel in the fusion area of the image, and Color _i is the i-th pixel provided by the fusion material The pixel value of the point.

In order to make the edges of the fusion region gradually gradually change, reduce the abrupt gradient, and be more integrated and natural, the edge smoothing unit 06 is responsible for smoothing the edges of the second template, and outputting the smoothed second template to the normalization unit 03.

Specifically, the edge smoothing unit 06 may extend the contour points of the fusion area in the second template outward and/or inward respectively by M pixels, where M is a preset positive integer, and the area surrounded by the extended pixels is taken as Area to be smoothed: Affine the predefined smoothing template to the area to be smoothed to obtain a smoothed second template. When performing affine, the pixel value of the pixel in the smoothing template can be affineed as the pixel value of the pixel at the corresponding position in the area to be smoothed.

The shape of the fusion area can be used in advance to form a template whose edges have been smoothed, and the template is a smooth template. This way of affine a pre-defined smoothing template to the area to be smoothed, thereby completing the smoothing of the area to be smoothed, can quickly realize edge smoothing, and avoid the amount of calculation and time overhead caused by instant blur smoothing.

Furthermore, when the edge smoothing unit 06 affines the predefined smoothing template to the area to be smoothed, it can adopt a triangulation method, that is, the smoothing area on the smoothing template and the area to be smoothed on the second template are respectively used Triangulation is performed in the same way to obtain the same number of triangle regions; then, each triangle region in the smooth template is affineed to the triangle region at the corresponding position in the second template.

The brightness adjustment unit 07 is responsible for obtaining the second template output by the edge smoothing unit 06, performs brightness statistics on the fusion area in the image, and performs brightness adjustment on the obtained second template according to the brightness statistics results, and adjusts the brightness of the second template after the brightness adjustment. Output to the normalization unit 03. The brightness adjustment unit 07 can make the brightness of the smoothed second template match the actual brightness of the human face in the original image as much as possible. When adjusting the brightness, the difference between the average brightness of the fusion area in the image and the average brightness of the smoothed second template can be determined; then the brightness value of each pixel in the smoothed second template Add the difference separately.

The device can be applied to image processing apps, and can also be applied to beauty apps, etc. The device can be embodied in the form of an application, which can be an application (nativeAPP) running locally on the device, or a webpage application (webApp) on the device browser. In addition, it can also be embodied in the form of a plug-in or SDK in the application.

In addition to being applied to the above-mentioned beauty APPs such as foundation test makeup scenes, the present invention can also be applied to other image fusion scenes, such as combining a red apple in one image with a yellow one in another image. Apple performs fusion.

The above-mentioned method and device provided by the embodiment of the present invention may be embodied by a computer program set up and running in the device. The device may include one or more processors, as well as memory and one or more programs, as shown in FIG. 5. The one or more programs are stored in the memory and executed by the one or more processors to implement the method flow and/or the device operation shown in the above embodiments of the present invention. For example, the process of the method executed by the one or more processors may include: determining the fusion area in the image to obtain the first template; down-sampling the first template to obtain the second template; The pixel value of each pixel is normalized to obtain the third template; the third template is up-sampled to obtain the fourth template, and the number of pixels in the fourth template is equal to the number of pixels in the first template; The pixel value of each pixel in the template is used as the weight of the corresponding pixel in the image, and each pixel in the fusion area in the image and the fusion material are weighted and fused.

It can be seen from the above description that the method and device provided by the present invention can have the following advantages:

1) The present invention adopts the method of down-sampling the image to calculate the weight of the down-sampled fusion area, and then up-sample it back to the original image size to obtain the corresponding pixel points of the fusion area in the original image during fusion The weight greatly reduces the amount of calculation caused by the weight calculation, and reduces the time cost and resource consumption.

2) In the weight calculation process, edge smoothing and/or brightness adjustment are performed on the down-sampled fusion area, which further reduces the amount of calculation and reduces time cost and resource consumption.

3) When performing edge smoothing, a predefined smoothing template is used, and the triangulation method can be further combined to quickly achieve edge smoothing, avoiding the amount of calculation and time overhead caused by instant blur smoothing.

4) It can still maintain real-time performance under high-resolution image quality. It can be applied not only to static images, but also to movie images.

5) When applied to foundation makeup, since only the edges of the face area are smoothed, the texture information of the face itself can be effectively preserved. There is no need for manual operation by the user, and the fusion weight can be automatically adjusted according to the brightness of the skin color of the face, and a more realistic makeup trial experience can be obtained.

In the several embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiment described above is only illustrative. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation.

The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized either in the form of hardware, or in the form of hardware plus software functional units.

The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium. The above-mentioned software functional unit is stored in a storage medium, and includes a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute the method of the various embodiments of the present invention Part of the steps. The aforementioned storage media include: flash drives, mobile hard drives, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disks or optical disks, etc., which can store code Medium.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection of the present invention. Within the range.

Claims (23)

An image fusion method, characterized in that the method comprises: determining a fusion area in an image, removing areas in the image except the fusion area to obtain a first template; down-sampling the first template , Obtain the second template; normalize the pixel value of each pixel in the second template to obtain the third template; perform up-sampling on the third template to obtain the fourth template, the pixels of the fourth template The number is equal to the number of pixels in the first template; the pixel value of each pixel in the fourth template is used as the weight of the corresponding pixel in the image, and each pixel in the fusion area of the image and the fusion material Perform weighted fusion. The method according to claim 1, wherein determining the fusion area in the image, removing the area other than the fusion area in the image, and obtaining the first template includes: the fusion target in the image Perform feature point positioning, where the feature points include contour points; use the located feature points to remove areas in the image other than the fusion target to obtain the first template. The method according to claim 1, wherein down-sampling the first template to obtain the second template includes: down-sampling the first template by means of affine transformation, so that the obtained second template The number of pixels is that of the first template

Times, the N is a positive integer greater than 2; up-sampling the third template to obtain the fourth template includes: using inverse affine transformation, up-sampling the third template so that the obtained fourth template The number of pixels in is N times that of the third template. The method according to claim 1, wherein, before normalizing the pixel value of each pixel in the second template, the method further comprises: smoothing the edge of the second template. The method according to claim 4, wherein the smoothing processing on the edge of the second template includes: extending the contour points of the fusion area in the second template outward and/or inward respectively by M pixels, and the M It is a preset positive integer, and the area surrounded by the expanded pixels is taken as the area to be smoothed; the predefined smoothing template is affine to the area to be smoothed to obtain the smoothed second template. The method according to claim 5, wherein the affine the predefined smoothing template to the area to be smoothed comprises: affine the pixel value of the pixel in the smoothing template to the pixel at the corresponding position in the area to be smoothed The pixel value of the point. The method according to claim 5, wherein the affine the predefined smoothing template to the area to be smoothed includes: the smoothing area on the smoothing template and the area to be smoothed on the second template are respectively the same Triangulate by way of triangulation to obtain the same number of triangular areas; affine each triangular area in the smooth template to the second template The triangular area corresponding to the location. The method according to claim 4, wherein, before normalizing the pixel value of each pixel in the second template, the method further includes: performing brightness statistics on the fusion region in the image; according to Based on the brightness statistics result, brightness adjustment is performed on the smoothed second template. The method according to claim 8, wherein the performing brightness adjustment on the smoothed second template according to the brightness statistics includes: determining the average brightness of the fusion area in the image and the smoothed second template The difference between the average brightness values of the two templates; the brightness value of each pixel in the second template after the smoothing process is added to the difference. The method according to claim 1, wherein the weighted fusion of each pixel in the fusion area and the fusion material in the image includes: using Image _{i_new} = weight_mask _i * Image _{i_old} + (1-weight_mask _i ) * Color _i to determine The pixel value of each pixel obtained after fusion; where Image _{i_new} is the pixel value of the i-th pixel obtained after fusion of the fused area in the image, and weight_mask _{i is} the pixel value of the i-th pixel in the fourth template , Image _{i_old} is the pixel value of the i-th pixel in the fusion area of the image, and Color _i is the pixel value of the i-th pixel provided by the fusion material. The method according to any one of claims 1 to 10, wherein: This method is applied to beauty apps; the fusion area is a face area; the fusion material is a foundation color. An image fusion device, characterized in that the device includes: a template determining unit for determining a fusion area in an image, removing the area except the fusion area in the image to obtain a first template; downsampling A unit for down-sampling the first template to obtain a second template; a normalization unit for normalizing the pixel value of each pixel in the second template to obtain a third template; up-sampling Unit for up-sampling the third template to obtain a fourth template, the number of pixels of the fourth template is equal to the number of pixels of the first template; the weighted fusion unit is used for the fourth template The pixel value of each pixel is used as the weight of the corresponding pixel in the image, and each pixel and the fusion material in the fusion area in the image are weighted and fused. The device according to claim 12, wherein the template determining unit is configured to: locate a feature point of the fusion target in the image, the feature point includes a contour point; use the located feature point to remove the image In the area other than the fusion target, the first template is obtained. The apparatus according to claim 12, wherein the down-sampling unit is configured to down-sample the first template by means of affine transformation, so that the number of pixels of the second template obtained is the first Template

Times, the N is a positive integer greater than 2; the up-sampling unit is used to up-sample the third template by inverse affine transformation, so that the number of pixels of the fourth template obtained is the first N times the three templates. The device according to claim 12, wherein the device further comprises: an edge smoothing unit, configured to smooth the edges of the second template, and output the smoothed second template to the normalization unit. The device according to claim 15, wherein the edge smoothing unit is configured to extend the contour points of the fusion region in the second template outward and/or inward respectively by M pixels, where M is a preset A positive integer, the area surrounded by the expanded pixels is regarded as the area to be smoothed; the predefined smoothing template is affine to the area to be smoothed to obtain the smoothed second template. The device according to claim 16, wherein, when the edge smoothing unit affines the predefined smoothing template to the area to be smoothed, executes: affine the pixel value of the pixel in the smoothing template as the to-be-smoothed area. The pixel value of the pixel at the corresponding position in the smooth area. The device according to claim 16, wherein, when the edge smoothing unit affines the predefined smoothing template to the area to be smoothed, execute: The smooth area on the smoothing template and the area to be smoothed on the second template are respectively triangulated in the same way to obtain the same number of triangular areas; each triangular area in the smoothing template is affineed to the first The triangular area of the corresponding position in the second template. The device according to claim 15, wherein the device further comprises: a brightness adjustment unit, configured to obtain the second template output by the edge smoothing unit, and perform brightness statistics on the fusion area in the image, according to brightness statistics As a result, brightness adjustment is performed on the acquired second template, and the brightness adjusted second template is output to the normalization unit. The device according to claim 19, wherein the brightness adjustment unit performs brightness adjustment on the smoothed second template according to the brightness statistics result: determining the sum of the brightness averages of the fusion area in the image The difference between the average brightness values of the second template after the smoothing process; the brightness values of the pixels in the second template after the smoothing process are respectively added to the difference. The device according to claim 12, wherein the weighted fusion unit is used to: use Image _{i_new} = weight_mask _i * Image _{i_old} + (1-weight_mask _i ) * Color _i to determine the pixel value of each pixel after fusion ; Where Image _{i_new} is the pixel value of the i-th pixel obtained after the fusion area in the image is fused, weight_mask _{i is} the pixel value of the i-th pixel in the fourth template, Image _{i_old} is the fusion area in the image The pixel value of the i-th pixel, Color _i is the pixel value of the i-th pixel provided by the fusion material. The device according to any one of claims 12 to 21, wherein the device is applied to a beauty APP; the fusion area is a face area; and the fusion material is a foundation color. A device that includes one or more processors; memory; one or more programs, the one or more programs are stored in the memory, and executed by the one or more processors to achieve the following operations: determine the image Remove the fusion area in the image except the fusion area to obtain the first template; down-sample the first template to obtain the second template; the pixels of each pixel in the second template The value is normalized to obtain the third template; the third template is up-sampled to obtain the fourth template, the number of pixels of the fourth template is equal to the number of pixels of the first template; the fourth template The pixel value of each pixel in the image is respectively used as the weight of the corresponding pixel in the image, and each pixel and the fusion material in the fusion area in the image are weighted and fused.

Images