CN112785533B - Image fusion method, image fusion device, electronic equipment and storage medium - Google Patents
Image fusion method, image fusion device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112785533B CN112785533B CN201911082008.6A CN201911082008A CN112785533B CN 112785533 B CN112785533 B CN 112785533B CN 201911082008 A CN201911082008 A CN 201911082008A CN 112785533 B CN112785533 B CN 112785533B
- Authority
- CN
- China
- Prior art keywords
- image
- region
- type
- sub
- face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 21
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 25
- 239000003086 colorant Substances 0.000 claims description 5
- 238000001514 detection method Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 230000001815 facial effect Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 6
- 210000004209 hair Anatomy 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000001965 increasing effect Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 101150013335 img1 gene Proteins 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 241001270131 Agaricus moelleri Species 0.000 description 1
- 101100248200 Arabidopsis thaliana RGGB gene Proteins 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000002500 effect on skin Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 210000000720 eyelash Anatomy 0.000 description 1
- 210000003746 feather Anatomy 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 101150071665 img2 gene Proteins 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 210000000697 sensory organ Anatomy 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The disclosure provides an image fusion method, an image fusion device, electronic equipment and a computer readable storage medium, and belongs to the technical field of image processing. The method is applied to the terminal equipment with the image sensor, and comprises the following steps: acquiring a first image and a second image acquired by the image sensor, wherein the first image and the second image are images of the same area, and the number of pixels of the first image is higher than that of the second image; dividing the region into a first type of sub-region and a second type of sub-region; and fusing the first image and the second image based on the first type subarea in the first image and the second type subarea in the second image to obtain a target image. The method and the device can solve the problem of more noise of the high-pixel image, and improve the quality of image shooting.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image fusion method, an image fusion apparatus, an electronic device, and a computer readable storage medium.
Background
With the development of computer technology and multimedia technology, many image capturing devices and applications have emerged, thereby generating a large amount of image data. In order to meet the shooting requirements of people, the improvement of pixels of an image sensor is a popular development direction in the industry, for example, an image sensor with millions or even tens of millions of pixels is commonly adopted on a mobile phone, and can support shooting of ultra-high-definition pictures. However, while a high pixel count image can retain more detail of the subject, the noise is large, and a low pixel count image may not be sufficiently sharp.
Therefore, how to improve the noise while increasing the pixels is a problem to be solved.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The disclosure provides an image fusion method, an image fusion device, electronic equipment and a computer readable storage medium, so as to at least improve the problem of high noise in the existing high-pixel picture to a certain extent.
Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.
According to one aspect of the present disclosure, there is provided an image fusion method applied to a terminal device having an image sensor, including: acquiring a first image and a second image acquired by the image sensor, wherein the first image and the second image are images of the same area, and the number of pixels of the first image is higher than that of the second image; dividing the region into a first type of sub-region and a second type of sub-region; and fusing the first image and the second image based on the first type subarea in the first image and the second type subarea in the second image to obtain a target image.
According to an aspect of the present disclosure, there is provided an image fusion apparatus applied to a terminal device provided with an image sensor, including: the image acquisition module is used for acquiring a first image and a second image acquired by the image sensor, wherein the first image and the second image are images of the same area, and the pixel number of the first image is higher than that of the second image; the region dividing module is used for dividing the region into a first type of sub-region and a second type of sub-region; and the image fusion module is used for fusing the first image and the second image based on the first type subarea in the first image and the second type subarea in the second image to obtain a target image.
According to one aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of any of the above via execution of the executable instructions.
According to one aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of the above.
Exemplary embodiments of the present disclosure have the following advantageous effects:
acquiring a first image and a second image acquired by the image sensor, wherein the first image and the second image are images of the same area, the pixel number of the first image is higher than that of the second image, the area is divided into a first type subarea and a second type subarea, and the first image and the second image are fused based on the first type subarea in the first image and the second type subarea in the second image to obtain a target image. On the one hand, as the details of the images in the first image are rich, the light incoming amount in the second image is large, the noise is small, and the advantages in the two images can be integrated by fusing the first image and the second image, so that a target image with high quality can be obtained; on the other hand, based on the first type subarea in the first image and the second type subarea in the second image, the target image is obtained through fusion, so that the target image can keep the fine sense of the first type subarea and the soft sense of the second subarea, the pertinence of the target image to the representation of the local area is improved, and the target image has wider application scenes.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort.
Fig. 1 schematically shows a flowchart of an image fusion method in the present exemplary embodiment;
fig. 2 schematically shows a schematic view of a color filter array in the present exemplary embodiment;
fig. 3 schematically shows a sub-flowchart of an image fusion method in the present exemplary embodiment;
fig. 4 schematically illustrates a schematic view of obtaining a first image in the present exemplary embodiment;
fig. 5 schematically illustrates a schematic diagram of obtaining a second image in the present exemplary embodiment;
fig. 6 schematically shows a schematic view of a face image in the present exemplary embodiment;
fig. 7 schematically shows a sub-flowchart of another image fusion method in the present exemplary embodiment;
fig. 8 schematically shows a block diagram of an image fusion apparatus in the present exemplary embodiment;
fig. 9 schematically shows an electronic device for implementing the above method in the present exemplary embodiment;
fig. 10 schematically illustrates a computer-readable storage medium for implementing the above-described method in the present exemplary embodiment.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
The exemplary embodiment of the present disclosure provides an image fusion method, and the exemplary embodiment of the present disclosure provides an image fusion method, which may be applied to a terminal device such as a mobile phone, a tablet computer, a digital camera, and the like. The terminal device is provided with an image sensor, which can be used for acquiring images. The image sensor can realize shooting 6400 ten thousand and 1600 ten thousand original image data successively by setting different hardware registers.
The following describes the present exemplary embodiment with reference to fig. 1, and as shown in fig. 1, the image fusion method may include the following steps S110 to S140:
step S110, a first image and a second image acquired by an image sensor are acquired, wherein the first image and the second image are images of the same area, and the pixel number of the first image is higher than that of the second image.
The areas shot by the first image and the second image are the same areas, the shot objects are the same objects, namely the shot images have the same content, and the difference is that the pixel number is higher than that of the second image, for example, the first image can be 6400 ten thousand pixels, and the second image can be 1600 ten thousand pixels.
In an exemplary embodiment, the image sensor may be a four Bayer (Quad Bayer) image sensor, which refers to an image sensor employing a four Bayer color filter array. Referring to fig. 2, the left diagram shows a standard bayer color filter array, in which the unit array of the filter is arranged as GRBG (or BGGR, GBRG, RGGB), and most of the image sensors employ the standard bayer color filter array; the right diagram in fig. 2 shows a four bayer color filter array, in which four adjacent units in a unit array of a filter are the same color, and currently, a part of high-pixel image sensors adopts the four bayer color filter array. Based on this, as shown with reference to fig. 3, step S110 may be specifically implemented by the following steps S310 to S330:
step S310, acquiring a raw bayer image based on a four bayer color filter array by a four bayer image sensor.
The bayer image is an image in a RAW format, and is image data obtained by converting an acquired optical signal into a digital signal by an image sensor, and in the bayer image, each pixel has only one color of RGB. In the present exemplary embodiment, after an image is acquired by using a four bayer image sensor, the raw image data obtained is the above raw bayer image, in which the color arrangement of pixels is as shown in the right diagram in fig. 2, and four adjacent pixels are of the same color.
Step S320, performing demosaicing processing and demosaicing processing on the original bayer image to obtain a first image.
Wherein, demosaic processing (Remosaic) refers to fusing a raw bayer image based on a four bayer color filter array into a bayer image based on a standard bayer color filter array; demosaicing (Demosaic) refers to fusing bayer images into complete RGB images. As shown in fig. 4, the original bayer image P may be demosaiced to obtain a bayer image Q1 based on a standard bayer color filter array; and then demosaicing is carried out on the Bayer image Q1 based on the standard Bayer color filter array, so as to obtain a first image IMG1 in RGB format. The demosaicing and demosaicing may be implemented by different interpolation algorithms, or may be implemented by other related algorithms such as neural networks, which are not limited in this disclosure. An ISP (Image Signal Processing ) unit, which is matched with the image sensor, is generally configured in the terminal device to perform the above-described demosaicing and demosaicing processes. Each pixel of the first image IMG1 has pixel values of three channels of RGB, denoted C. In addition, the processing procedures of demosaicing and demosaicing may be combined into a single interpolation procedure, that is, each pixel point is directly interpolated based on the pixel data in the original bayer image, so as to obtain the pixel value of the missing color channel, for example, the processing procedures may be implemented by adopting algorithms such as linear interpolation, mean interpolation, and the like, so as to obtain the first image.
Step S330, combining four adjacent pixels with the same color in the original Bayer image into one pixel, and performing demosaicing processing on the Bayer image after the pixels are combined to obtain a second image.
As shown in fig. 5, the original bayer image P is first subjected to a pixel "four-in-one" process, that is, pixels of the same color in 2 x 2 units are combined into one pixel, and the bayer image Q2 after the pixels are combined is also based on the arrangement of a standard bayer color filter array, and compared with the Q1 and Q2 in fig. 4, the pixels are reduced to 1/4, and meanwhile, the area of each pixel is increased to 4 times, so that the light inlet amount of each pixel is increased; and demosaicing is carried out on the Q2 to obtain a second image IMG2 in an RGB format. It can be seen that the number of pixels of the first image is four times that of the second image.
In another embodiment, the terminal device may be configured with two image sensors of different pixels, for example many cell phones are currently configured with dual cameras. Wherein the image sensor with higher pixels is used for shooting a first image, and the image sensor with lower pixels is used for shooting a second image. Since the image acquisition process of the two image sensors is completed in one shooting, the exposure degree is similar, so that the resolution of the first image is higher, but the noise of the first image may be more due to the influence of the light sensing amount, and the second image is opposite.
Step S120, dividing the region into a first type of sub-region and a second type of sub-region.
In general, the flatness of a screen captured by an image sensor varies according to the captured content. Specifically, the area in the captured image may be divided into a flat area and a non-flat area, where the flat area and the non-flat area are a set of relative concepts that may be used to represent the degree of density or sparseness of the content or texture in the image, the denser the image content, the higher the degree of variation in pixel values within the image, which represents the richer the details in the image, the higher the non-flatness, and the lower the flatness. For example, when a person is photographed, the hair area of the person may be divided into non-flat areas, the image of the hair area may show a finer state, such as frizziness, suppleness, or bright filaments, and the face area may be divided into flat areas, and the face image may show less details than the hair, but may show a better skin effect. As shown in fig. 6, the left side of fig. 6 is a lower pixel number image, the right side of fig. 6 is a higher pixel number image, and the higher pixel number image can be seen compared with the lower pixel number image, the hair area S detail is better, the lower pixel number image is softer than the higher pixel number image, and the face area P is softer and better skin quality is shown. Based on this, a non-flat region with richer detail can be determined as a first type of sub-region, and a flat region with less detail can be determined as a second type of sub-region. In the present exemplary embodiment, the region may be divided into the first-type sub-region and the second-type sub-region in various ways, for example, by extracting image features such as brightness, color, etc., and performing region division according to the image features; or in a specific application scene, such as a face image, the first type subarea and the second type subarea in the face can be identified by combining a face recognition method; the first type sub-region and the second type sub-region can also be divided by calculating the non-flatness of the image, and the disclosure is not limited in detail. Wherein, the image non-flatness can be calculated and characterized in a variety of ways, the present disclosure provides the following several embodiments, but the following should not limit the scope of the present disclosure:
(1) And calculating the pixel value variance in each region in the first image or the second image, and taking the pixel value variance as the image non-flatness. For example, the pixel value variance in S1 (IMG 1) or S1 (IMG 2) may be counted as the image non-flatness of the region S1. The pixel value can be converted into the gray value to calculate the variance during calculation, the variance can be calculated in three RGB channels respectively, and the variance mean value of the three channels can be taken. Additionally, the pixel value variance may be replaced with the pixel value standard deviation.
(2) A difference between the center pixel value and the edge pixel value in each region in the first image or the second image is calculated, and the difference is taken as the image non-flatness. For example, when calculating the pixel difference value of the region S1, S1 (IMG 1) or S1 (IMG 2) may be first divided into a center portion and an edge portion, and the pixel values of the center portion and the edge portion may be counted respectively to calculate the difference value; or from the central point of S1, calculating the pixel difference value between the central point and the edges of a plurality of layers, and then merging. The pixel difference between the center and the edge can also reflect the content density of the region and thus can also be a measure of the flatness of the image.
(3) And calculating the information entropy in each region in the first image or the second image, and taking the information entropy as the image non-flatness. The information entropy of an image, also called image entropy, is a measure that characterizes the change in information within an image. For example, when calculating the information entropy of the region S1, the pixels in S1 (IMG 1) or S1 (IMG 2) may be converted into gray scales, and the occurrence probability of each gray scale value is counted, for example, m gray scale values appear in total in S1 (IMG 1), where the occurrence probabilities are p1, p2, …, pm, respectively, then the information entropy
Of course, other approximate calculations may be used.
It should be noted that, because the pixels of the first image are higher, the image information is richer, and the data of the first image is adopted to calculate the non-flatness of the image, so that the result is more accurate; however, the data volume of the second image is 1/4 of that of the first image, and the calculation amount can be reduced by adopting the second image, so that the processing speed is improved; in practical application, the selection can be performed by combining hardware conditions, user requirements and the like. In addition, the above modes can be combined arbitrarily to synthesize a plurality of indexes to represent the image non-flatness, so that the robustness of the algorithm can be improved, and the disclosure is not limited to this.
In an exemplary embodiment, the region may be divided by a method of extracting image features, and in particular, the step S120 may include the steps of:
extracting image features from at least one of the first image and the second image;
dividing the region into a high-frequency region and a non-high-frequency region according to the image characteristics, wherein the high-frequency region is a first type of sub-region, and the non-high-frequency region is a second type of sub-region.
The high-frequency region refers to a non-flat region with a large number of details, such as human hair, more complicated clothing, and animal feathers, and the non-high-frequency region refers to a flat region with a small number of details, such as human cheeks and skin of limbs. Image features are data that can reflect the characteristics of various regions in an image, such as color features, texture features, shape features, or spatial relationship features, among others. In the present exemplary embodiment, since the same content is contained in the first image and the second image, the extraction of the image features can be performed from the first image or the second image. Specifically, there may be various extraction manners of image features, for example, a plurality of Haar (Haar) feature templates are used to traverse the first image or the second image, and feature values are determined to extract corresponding image features. The present disclosure does not specifically limit the manner of extracting the image features. Further, by the extracted image features, it can be determined which part of the region is the region with more details and which part is the region with less details, that is, the division of the region into the first type sub-region and the second type sub-region is completed.
Step S130, fusing the first image and the second image based on the first type subarea in the first image and the second type subarea in the second image to obtain a target image.
From the above, the content of the first image and the second image is the same, the number of pixels of the first image is higher, the details in the image are clearer and richer, but the noise is usually higher, the number of pixels of the second image is lower, but the light incoming amount on a single pixel is larger, and the noise is lower. Therefore, the first image and the second image can be fused to integrate advantages in the two images, and a target image with higher quality is obtained. The target image may be a final output image, for example, when the user photographs with the mobile phone, the image sensor outputs the target picture to be displayed on the screen of the mobile phone by performing the above steps S110 to S130 after initially collecting the image data.
In practical applications, the target image may be stored as images with different pixel numbers, for example, it may be stored as images with higher pixel numbers of 6400 ten thousand, or may be stored as images with lower pixel numbers of 1600 ten thousand. In the present exemplary embodiment, the fusion scheme may be determined based on the requirement of the final number of pixels that the target image needs to hold.
In an exemplary embodiment, the step S130 may specifically include the following steps:
up-sampling a second type of sub-region in a second image based on the pixel number of the first image, and fusing the second type of sub-region with the first type of sub-region in the first image to obtain a target image; the number of pixels of the target image and the first image are the same.
When the number of pixels of the target image is the same as that of the first image, that is, the target image is an image with a higher number of pixels, because the number of pixels of the second image is lower than that of the first image, in order to enable the second image to be fused with the first image better, up-sampling processing is required to be performed on the second type of sub-area in the second image, that is, the second type of sub-area is subjected to image amplification, so that the second type of sub-area can be displayed on a display device with higher resolution. Specifically, new pixels can be inserted between pixels by adopting a proper interpolation algorithm (such as bilinear interpolation algorithm) on the basis of the image pixels of the second type of sub-region, so as to amplify the second type of sub-region. In addition, the method of upsampling may also include other various methods, such as interpolation methods, such as nearest neighbor method, cubic interpolation method, etc., transposed convolution, upsampling, or pooling, etc., which are not specifically limited in this disclosure. After the second type subarea is converted into the image with the same pixel number as the first image, the fusion of the second type subarea and the first type subarea of the first image can be completed, and the target image with higher pixel number is obtained.
In an exemplary embodiment, the step S130 may include the steps of:
downsampling the first type subareas in the first image based on the pixel number of the second image, and fusing the first type subareas with the second type subareas in the second image to obtain a target image; the number of pixels of the target image and the second image are the same.
When the number of pixels of the target image is the same as that of the second image, that is, the target image is an image with a lower number of pixels, since the number of pixels of the first image is higher than that of the second image, in order to enable the first image to be fused with the second image better, downsampling processing is required to be performed on the first type of sub-area in the first image, that is, the first type of sub-area is subjected to image reduction. Specifically, a window with a preset size may be set, and the image pixels included in the window are converted into a pixel point on the basis of the image pixels of the first type of sub-region, where the value of the pixel point may be an average value of all the pixel points in the window, so as to implement the reduction of the first type of sub-region. After the first type subarea is converted into the image with the same pixel number as the second image, the fusion of the first type subarea and the second type subarea of the second image can be realized, and the target image with lower pixel number is obtained.
In the present exemplary embodiment, when the first image and the second image are fused, the first image and the second image may be color images, gray images, or the like, which is not particularly limited in this disclosure.
Based on the above description, in the present exemplary embodiment, the first image and the second image acquired by the image sensor are acquired, the first image and the second image are images of the same area, the number of pixels of the first image is higher than that of the second image, the area is divided into the first type of sub-area and the second type of sub-area, and the first image and the second image are fused based on the first type of sub-area in the first image and the second type of sub-area in the second image, so as to obtain the target image. On the one hand, as the details of the images in the first image are rich, the light incoming amount in the second image is large, the noise is small, and the advantages in the two images can be integrated by fusing the first image and the second image, so that a target image with high quality can be obtained; on the other hand, based on the first type subarea in the first image and the second type subarea in the second image, the target image is obtained through fusion, so that the target image can keep the fine sense of the first type subarea and the soft sense of the second subarea, the pertinence of the target image to the representation of the local area is improved, and the target image has wider application scenes.
In an exemplary embodiment, the first image and the second image may be face images, the first type of sub-region is a non-face region, and the second type of sub-region is a face region.
In particular, the present exemplary embodiment may be applied to a scene photographed by a face, where the first image and the second image are face images. In the face region of the face image, since the difference of the facial complexion of the person is small, the similarity of the parts except the five sense organs is high, the face image is in a flat state, and in order to present a good skin state, the face image which is usually expected to be shot can have the characteristics of softness and low noise; and for non-facial areas such as hairs, eyelashes and the like, the characteristics of the structure, the position, the size and the like of the non-facial areas have larger flexibility, and the finer the images of the areas are, the more the images of the areas can reflect the reality and the stereoscopic impression of the face images, so that the liveliness of the images of the faces can be reflected. Thus, a facial region may be considered a second type of sub-region and a non-facial region may be considered a first type of sub-region.
In an exemplary embodiment, the dividing the region into the first type of sub-region and the second type of sub-region may include the following steps:
step S710, extracting color distribution of the first image or the second image, and determining colors within a preset range as face colors;
step S720, based on the area with the face color in the first image or the second image, further determining the face area by adopting face pattern detection;
in step S730, an area other than the face area is determined as a non-face area.
Considering that in the face image, the skin color of the face has a large difference from the facial color, but the color difference of the face is small in the part except the facial region, and the skin color is usually in a stable interval. Accordingly, the face region of the face can be determined by extracting the color distribution of the first image or the second image. Specifically, a preset range may be set, and when the extracted color distribution is in the preset range, the color in the preset range in the face image may be considered as a face color, and the area corresponding to the face color is further considered as a face area.
In some special cases, errors may occur when using color distribution to determine the facial region, such as when the lips are lighter in the figure, or when the lips are similar to skin colors, the facial region may not be accurately distinguished from the lip region. Based on this, the present exemplary embodiment can further detect the position of the face region in the face image using the face pattern after determining the face region by the face color. The face pattern may be a template pattern for confirming a face area, which may be customized in advance as needed. It is contemplated that different faces may differ, for example, the facial shape and characteristics of a male versus a female may differ somewhat, or the facial image of the same female may differ. Therefore, the present exemplary embodiment can provide a plurality of face patterns, detect by a plurality of face patterns at the time of detection, or detect a plurality of times using the same face pattern, or the like, to ensure the accuracy of the determined face area. When the face region is determined, the corresponding region outside the face region will be determined as a non-face region.
The exemplary embodiment of the disclosure also provides an image fusion device. Referring to fig. 8, the apparatus 800 may include an image acquisition module 810 for acquiring a first image and a second image acquired by an image sensor, the first image and the second image being images of a same area, the first image having a higher number of pixels than the second image; a region dividing module 820 for dividing the region into a first type of sub-region and a second type of sub-region; the image fusion module 830 is configured to fuse the first image and the second image based on the first type of sub-region in the first image and the second type of sub-region in the second image, so as to obtain a target image.
In an exemplary embodiment, the image sensor includes a four bayer image sensor; the image acquisition module may include: an image acquisition unit for acquiring a raw bayer image based on a four bayer color filter array by a four bayer image sensor; the first image processing unit is used for performing demosaicing processing and demosaicing processing on the original Bayer image to obtain a first image; the second image processing unit is used for merging four adjacent pixels with the same color in the original Bayer image into one pixel, and demosaicing the Bayer image after the pixels are merged to obtain a second image; wherein the number of pixels of the first image is four times that of the second image.
In an exemplary embodiment, the region dividing module may include: a feature extraction unit for extracting image features from at least one of the first image and the second image; the region dividing unit is used for dividing the region into a high-frequency region and a non-high-frequency region according to the image characteristics, wherein the high-frequency region is a first type of sub-region, and the non-high-frequency region is a second type of sub-region.
In an exemplary embodiment, the first image and the second image are face images, the first type of sub-region is a non-face region, and the second type of sub-region is a face region.
In an exemplary embodiment, the region dividing module may include: a color extraction unit configured to extract a color distribution of the first image or the second image, and determine a color within a preset range therein as a face color; a region detection unit for further determining a face region using face pattern detection based on a region of a face color in the first image or the second image; an area other than the face area is determined as a non-face area.
In an exemplary embodiment, the image fusion module may include: the first image fusion unit is used for up-sampling the second type of sub-regions in the second image based on the pixel number of the first image and fusing the second type of sub-regions with the first type of sub-regions in the first image to obtain a target image; the number of pixels of the target image and the first image are the same.
In an exemplary embodiment, the image fusion module may include: the first image fusion unit is used for downsampling the first type of subareas in the first image based on the pixel number of the second image and fusing the first type of subareas with the second type of subareas in the second image to obtain a target image; the number of pixels of the target image and the second image are the same.
The specific details of each module/unit in the above apparatus are already described in the embodiments of the method section, and the details not disclosed can be found in the embodiments of the method section, so that they will not be described here again.
The exemplary embodiments of the present disclosure also provide an electronic device capable of implementing the above method.
Those skilled in the art will appreciate that the various aspects of the present disclosure may be implemented as a system, method, or program product. Accordingly, various aspects of the disclosure may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.
An electronic device 900 according to such an exemplary embodiment of the present disclosure is described below with reference to fig. 9. The electronic device 900 shown in fig. 9 is merely an example and should not be construed to limit the functionality and scope of use of embodiments of the present disclosure in any way.
As shown in fig. 9, the electronic device 900 is embodied in the form of a general purpose computing device. Components of electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting the different system components (including the storage unit 920 and the processing unit 910), a display unit 940, and an image sensor 970, the image sensor 970 for capturing images.
Wherein the storage unit stores program code that is executable by the processing unit 910 such that the processing unit 910 performs steps according to various exemplary embodiments of the present disclosure described in the above-described "exemplary methods" section of the present specification. For example, the processing unit 910 may execute steps S110 to S130 shown in fig. 1, may execute steps S310 to S330 shown in fig. 3, and the like.
The storage unit 920 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 921 and/or cache memory 922, and may further include Read Only Memory (ROM) 923.
The storage unit 920 may also include a program/utility 924 having a set (at least one) of program modules 925, such program modules 925 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The bus 930 may be one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 900 may also communicate with one or more external devices 1100 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 900, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 900 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 950. Also, electronic device 900 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 over the bus 930. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 900, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solutions according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the exemplary embodiments of the present disclosure.
Exemplary embodiments of the present disclosure also provide a computer readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification. In some possible implementations, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the disclosure as described in the "exemplary methods" section of this specification, when the program product is run on the terminal device.
Referring to fig. 10, a program product 1000 for implementing the above-described method according to an exemplary embodiment of the present disclosure is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
Furthermore, the above-described figures are only schematic illustrations of processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with exemplary embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (9)
1. An image fusion method applied to a terminal device provided with an image sensor, comprising:
acquiring a first image and a second image acquired by the image sensor, wherein the first image and the second image are images of the same area, and the number of pixels of the first image is higher than that of the second image; the first image and the second image are face images;
dividing a human face in the region into a first type subarea and a second type subarea, wherein the first type subarea is a non-face region of the human face in the region, the second type subarea is a face region of the human face in the region, and the detail representation of the non-face region is more than that of the face region;
and fusing the first image and the second image based on the first type subarea in the first image and the second type subarea in the second image to obtain a target image.
2. The method of claim 1, wherein the image sensor comprises a four bayer image sensor;
the acquiring a first image and a second image acquired by the image sensor includes:
acquiring a raw bayer image based on a four bayer color filter array by the four bayer image sensor;
demosaicing processing and demosaicing processing are carried out on the original Bayer image, and the first image is obtained;
combining four adjacent pixels with the same color in the original Bayer image into one pixel, and demosaicing the Bayer image after the pixels are combined to obtain the second image;
wherein the number of pixels of the first image is four times that of the second image.
3. The method of claim 1, wherein the dividing the region into a first type of sub-region and a second type of sub-region comprises:
extracting image features from at least one of the first image and the second image;
the region is divided into a high frequency region and a non-high frequency region according to the image features.
4. The method of claim 1, wherein the dividing the portrait in the area into a first type of subarea and a second type of subarea comprises:
extracting color distribution of the first image or the second image, and determining colors within a preset range as face colors;
further determining the face region using face pattern detection based on the region in the first image or the second image that is the face color;
an area of the area other than the face area of the face is determined as the non-face area.
5. The method according to any one of claims 1 to 4, wherein the fusing the first image and the second image based on the first type of sub-region in the first image and the second type of sub-region in the second image to obtain the target image includes:
based on the pixel number of the first image, up-sampling a second type of sub-region in the second image, and fusing the second type of sub-region with a first type of sub-region in the first image to obtain a target image; the number of pixels of the target image and the first image are the same.
6. The method according to any one of claims 1 to 4, wherein the fusing the first image and the second image based on the first type of sub-region in the first image and the second type of sub-region in the second image to obtain the target image includes:
downsampling a first type of sub-region in the first image based on the pixel number of the second image, and fusing the first type of sub-region with a second type of sub-region in the second image to obtain a target image; the number of pixels of the target image and the second image are the same.
7. An image fusion apparatus applied to a terminal device provided with an image sensor, comprising:
the image acquisition module is used for acquiring a first image and a second image acquired by the image sensor, wherein the first image and the second image are images of the same area, and the pixel number of the first image is higher than that of the second image; the shooting objects of the first image and the second image are portraits; the first image and the second image are face images;
the region dividing module is used for dividing the human face in the region into a first type of sub-region and a second type of sub-region, wherein the first type of sub-region is a non-face region of the human face in the region, and the second type of sub-region is a face region of the human face in the region;
and the image fusion module is used for fusing the first image and the second image based on the first type subarea in the first image and the second type subarea in the second image to obtain a target image.
8. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the method of any of claims 1-6 via execution of the executable instructions.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911082008.6A CN112785533B (en) | 2019-11-07 | 2019-11-07 | Image fusion method, image fusion device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911082008.6A CN112785533B (en) | 2019-11-07 | 2019-11-07 | Image fusion method, image fusion device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112785533A CN112785533A (en) | 2021-05-11 |
| CN112785533B true CN112785533B (en) | 2023-06-16 |
Family
ID=75747848
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911082008.6A Active CN112785533B (en) | 2019-11-07 | 2019-11-07 | Image fusion method, image fusion device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112785533B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106412407A (en) * | 2016-11-29 | 2017-02-15 | 广东欧珀移动通信有限公司 | Control method, control device and electronic device |
| CN106815827A (en) * | 2017-01-18 | 2017-06-09 | 聚龙智瞳科技有限公司 | Image interfusion method and image fusion device based on Bayer format |
| CN107730448A (en) * | 2017-10-31 | 2018-02-23 | 北京小米移动软件有限公司 | U.S. face method and device based on image procossing |
| CN107808136A (en) * | 2017-10-31 | 2018-03-16 | 广东欧珀移动通信有限公司 | Image processing method, device, readable storage medium storing program for executing and computer equipment |
| CN109272459A (en) * | 2018-08-20 | 2019-01-25 | Oppo广东移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
| CN109639982A (en) * | 2019-01-04 | 2019-04-16 | Oppo广东移动通信有限公司 | A kind of image denoising method, device, storage medium and terminal |
| CN110278374A (en) * | 2019-06-27 | 2019-09-24 | Oppo广东移动通信有限公司 | Image processing method and device, electronic equipment, computer readable storage medium |
| CN110336942A (en) * | 2019-06-28 | 2019-10-15 | Oppo广东移动通信有限公司 | A kind of virtualization image acquiring method and terminal, computer readable storage medium |
| CN110378846A (en) * | 2019-06-28 | 2019-10-25 | 北京字节跳动网络技术有限公司 | A kind of method, apparatus, medium and the electronic equipment of processing image mill skin |
| CN110675404A (en) * | 2019-09-03 | 2020-01-10 | RealMe重庆移动通信有限公司 | Image processing method, image processing apparatus, storage medium, and terminal device |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7602942B2 (en) * | 2004-11-12 | 2009-10-13 | Honeywell International Inc. | Infrared and visible fusion face recognition system |
| KR100745981B1 (en) * | 2006-01-13 | 2007-08-06 | 삼성전자주식회사 | Method and apparatus scalable face recognition based on complementary features |
| US8620038B2 (en) * | 2006-05-05 | 2013-12-31 | Parham Aarabi | Method, system and computer program product for automatic and semi-automatic modification of digital images of faces |
| KR101590868B1 (en) * | 2009-07-17 | 2016-02-02 | 삼성전자주식회사 | A image processing method an image processing apparatus a digital photographing apparatus and a computer-readable storage medium for correcting skin color |
| CN103888679A (en) * | 2014-03-13 | 2014-06-25 | 北京智谷睿拓技术服务有限公司 | Image collection method and device |
| CN106357967B (en) * | 2016-11-29 | 2018-01-19 | 广东欧珀移动通信有限公司 | Control method, control device and electronic installation |
| CN106713790B (en) * | 2016-11-29 | 2019-05-10 | Oppo广东移动通信有限公司 | Control method, control device and electronic device |
| CN109325988B (en) * | 2017-07-31 | 2022-11-11 | 腾讯科技（深圳）有限公司 | Facial expression synthesis method and device and electronic equipment |
| CN107610077A (en) * | 2017-09-11 | 2018-01-19 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic installation and computer-readable recording medium |
| KR102466998B1 (en) * | 2018-02-09 | 2022-11-14 | 삼성전자주식회사 | Method and apparatus for image fusion |
| CN110324626B (en) * | 2019-07-10 | 2021-05-18 | 武汉大学苏州研究院 | Dual-code-stream face resolution fidelity video coding and decoding method for monitoring of Internet of things |
-
2019
- 2019-11-07 CN CN201911082008.6A patent/CN112785533B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106412407A (en) * | 2016-11-29 | 2017-02-15 | 广东欧珀移动通信有限公司 | Control method, control device and electronic device |
| CN106815827A (en) * | 2017-01-18 | 2017-06-09 | 聚龙智瞳科技有限公司 | Image interfusion method and image fusion device based on Bayer format |
| CN107730448A (en) * | 2017-10-31 | 2018-02-23 | 北京小米移动软件有限公司 | U.S. face method and device based on image procossing |
| CN107808136A (en) * | 2017-10-31 | 2018-03-16 | 广东欧珀移动通信有限公司 | Image processing method, device, readable storage medium storing program for executing and computer equipment |
| CN109272459A (en) * | 2018-08-20 | 2019-01-25 | Oppo广东移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
| CN109639982A (en) * | 2019-01-04 | 2019-04-16 | Oppo广东移动通信有限公司 | A kind of image denoising method, device, storage medium and terminal |
| CN110278374A (en) * | 2019-06-27 | 2019-09-24 | Oppo广东移动通信有限公司 | Image processing method and device, electronic equipment, computer readable storage medium |
| CN110336942A (en) * | 2019-06-28 | 2019-10-15 | Oppo广东移动通信有限公司 | A kind of virtualization image acquiring method and terminal, computer readable storage medium |
| CN110378846A (en) * | 2019-06-28 | 2019-10-25 | 北京字节跳动网络技术有限公司 | A kind of method, apparatus, medium and the electronic equipment of processing image mill skin |
| CN110675404A (en) * | 2019-09-03 | 2020-01-10 | RealMe重庆移动通信有限公司 | Image processing method, image processing apparatus, storage medium, and terminal device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112785533A (en) | 2021-05-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10540806B2 (en) | Systems and methods for depth-assisted perspective distortion correction | |
| US20210133926A1 (en) | Image super-resolution reconstruction method, mobile terminal, and computer-readable storage medium | |
| US10887519B2 (en) | Method, system and apparatus for stabilising frames of a captured video sequence | |
| JP2020144907A (en) | Configurable convolution engine for interleave channel data | |
| CN110637297B (en) | Convolution engine, data processing method and electronic equipment | |
| US8947501B2 (en) | Scene enhancements in off-center peripheral regions for nonlinear lens geometries | |
| EP3644599B1 (en) | Video processing method and apparatus, electronic device, and storage medium | |
| JPWO2018047687A1 (en) | Three-dimensional model generation device and three-dimensional model generation method | |
| US8903139B2 (en) | Method of reconstructing three-dimensional facial shape | |
| JP5949331B2 (en) | Image generating apparatus, image generating method, and program | |
| US20140340486A1 (en) | Image processing system, image processing method, and image processing program | |
| CN112767290B (en) | Image fusion method, image fusion device, storage medium and terminal device | |
| JP2000102040A (en) | Electronic stereo camera | |
| KR101598159B1 (en) | Image providing method and image providing device | |
| US20150356743A1 (en) | Photographic subject tracking device and camera | |
| WO2022160857A1 (en) | Image processing method and apparatus, and computer-readable storage medium and electronic device | |
| WO2023284401A1 (en) | Image beautification processing method and apparatus, storage medium, and electronic device | |
| WO2014002521A1 (en) | Image processing device and image processing method | |
| CN110929615A (en) | Image processing method, image processing apparatus, storage medium, and terminal device | |
| CN113610865A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| KR20160037115A (en) | Apparatus and method of processing images | |
| TWI559762B (en) | An image processing apparatus, an image processing method, and a recording medium | |
| CN111083359B (en) | Image processing method and apparatus, electronic device, and computer-readable storage medium | |
| CN112785533B (en) | Image fusion method, image fusion device, electronic equipment and storage medium | |
| CN113805824B (en) | Electronic device and method for displaying image on display apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |