WO2022021999A1 - Image processing method and image processing apparatus - Google Patents

Abstract

Disclosed are an image processing method and an image processing apparatus. The image processing method comprises: acquiring multiple frames of images under a single exposure according to a first exposure parameter, wherein the multiple frames of images under the single exposure are images that contain target regions and have the same exposure parameter; and synthesizing the multiple frames of images under the single exposure to obtain a first image. According to the image processing method, the present invention solves the technical problems of serious detail loss, a ghost image and an insufficient dynamic range in a photographed target image.

Description

Image processing method and image processing device

This application claims the priority of the Chinese patent application filed on July 27, 2020 with the application number 202010731900.9 and the application title "An Image Processing Method and Image Processing Device", the entire contents of which are incorporated herein by reference middle.

technical field

The present invention relates to computer vision technology, and in particular, to an image processing method and an image processing device.

Background technique

In recent years, with the improvement of the shooting performance of electronic devices and the popularization of mobile phone shooting, people have increased the requirements for mobile phone shooting. However, limited by the size and quality of portable electronic devices, for example, when shooting the moon at a distance, it is difficult for users to obtain a clear and bright photo of the moon subject. Usually, the captured image will appear blurred, noisy, detail loss, dynamic Insufficient range, etc.

In view of the above-mentioned problems, the existing technology usually adopts the shooting of the same scene in different exposure modes, and then obtains multiple frames of photos and fuses them into one frame of wide dynamic range image. In the process of synthesizing multi-frame images, due to the unaligned images, the existence of moving objects, etc., the generated images will suffer from serious loss of details and ghosting.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide an image processing method and an image processing device, so as to at least solve the technical problems of serious loss of details, ghost images and insufficient dynamic range in the captured target image.

According to an aspect of the embodiments of the present invention, an image processing method is provided, comprising: acquiring a single-exposed multi-frame image according to a first exposure parameter, wherein the single-exposed multi-frame image is an image containing a target area with the same exposure parameter image; a first image is obtained by synthesizing multiple frames of images of a single exposure.

Optionally, the first exposure parameter is determined by metering the target area in the preview image, or the first exposure parameter is selected within a preset range.

Optionally, image enhancement is performed on the target area of the first image according to the pre-trained enhancement model to obtain a second image including a clear target area; the first image and the second image are fused to obtain a third image.

Optionally, metering the target area in the preview image to determine the first exposure parameter includes: performing feature detection in the preview image to obtain the target area; metering the target area to obtain the average brightness of the target area; The average brightness of , determines the first exposure parameter.

Optionally, synthesizing the single-exposed multi-frame images to obtain the first image, further comprising performing multi-frame fusion on the single-exposed multi-frame images using a multi-frame image super-resolution technology to obtain a fourth image; The image is subjected to single-frame dynamic range expansion to obtain a first image.

Optionally, using the multi-frame image super-resolution technology to perform multi-frame fusion on the single-exposed multi-frame images to obtain a fourth image, comprising: selecting a frame with the highest definition as a reference frame among the single-exposed multi-frame images; The remaining frames of the single-exposed multi-frame image are aligned to the reference frame based on features to obtain an aligned multi-frame image; the aligned multi-frame images are weighted and fused based on spatial information to obtain a fourth image.

Optionally, performing multi-frame fusion on the single-exposed multi-frame images using the multi-frame image super-resolution technology to obtain a fourth image, further comprising:

Sort the multi-frame images of a single exposure according to the sharpness, and select the first frame image as the reference frame;

The reference frame is weighted and fused to the second frame image based on feature alignment and spatial information, and the fused image is updated to the reference frame;

Processes based on feature alignment, weighted fusion based on spatial information, and updating the reference frame are sequentially performed, until the last frame image completes the feature alignment based and the weighted fusion based on spatial information, and a fourth image is obtained.

Optionally, performing single-frame dynamic range expansion on the fourth image to obtain the first image includes: constructing a logarithmic brightness pyramid according to the fourth image to obtain ambient light brightness of different scales; The digital brightness pyramid performs layer-by-layer downsampling, reconstruction and mapping on the fourth image to obtain the logarithmic reflection of the object surface at each pixel position; using the local mean and mean square error information, the logarithmic reflection of the object surface is mapped to the image value range, Get the first image.

Optionally, performing image enhancement on the target area of the first image according to the pre-trained enhancement model, to obtain a second image containing a clear target area, includes: using the constructed sample training set training to obtain a pre-trained enhancement model; The trained enhancement model performs image enhancement on the target area of the first image to obtain a second image containing a clear target area.

Optionally, using the constructed sample training set to train to obtain a pre-trained enhanced model includes: obtaining a first quality sample set collected by the electronic device, and obtaining a second quality sample set at the same location, wherein the image quality of the second quality sample set is high. in the first quality sample set; group the first quality sample set and the second quality sample set according to different magnifications of the equipment, and align the grouped sample sets according to the features to obtain the first sample set; The set is fed into the AI training engine to obtain a pre-trained augmented model.

Optionally, the second quality sample set includes: high-quality image samples collected by a high-definition electronic device or high-quality image samples stored in a storage device.

Optionally, the grouped sample sets are aligned according to features, wherein the features include: features of the image groups in the first quality sample set and the second quality sample set or features of auxiliary calibration points around the image groups.

According to another aspect of the embodiments of the present invention, an image processing apparatus is further provided, including: an acquisition unit configured to acquire a single-exposure multi-frame image according to a first exposure parameter, where the single-exposure multi-frame image includes a target area images with the same exposure parameters; the first fusion unit is configured to perform synthesis processing on multiple frames of images with a single exposure to obtain a first image.

Optionally, the image processing apparatus further includes a detection unit configured to determine the first exposure parameter or select the first exposure parameter within a preset range by metering the target area in the preview image.

Optionally, the image processing apparatus further includes: an enhancement unit, configured to perform image enhancement on a target area of the first image according to a pre-trained enhancement model, to obtain a second image containing a clear target; a second fusion unit, configured to Fusion processing is performed on the first image and the second image to obtain a third image.

Optionally, the detection unit includes: a detection module configured to perform feature detection in the preview image to obtain a target area; a photometric module configured to perform photometry on the target area to obtain an average brightness of the target area; a determination module, is configured to determine the first exposure parameter based on the average brightness of the target area.

Optionally, the first fusion unit includes: a fusion module configured to perform multi-frame fusion on a single-exposed multi-frame image using a multi-frame image super-resolution technology to obtain a fourth image; an extension module configured to perform multi-frame fusion on the fourth image; The image is subjected to single-frame dynamic range expansion to obtain a first image.

Optionally, the fusion module further includes: a reference sub-module, configured to select a frame with the highest definition among the single-exposed multi-frame images as a reference frame; an alignment sub-module configured to select the rest of the single-exposed multi-frame images as a reference frame; The frame-to-reference frame is aligned based on features to obtain aligned multi-frame images; the fusion sub-module is configured to weight the aligned multi-frame images based on spatial information to obtain a fourth image.

Optionally, the fusion module further includes: a first processing submodule, configured to sort the multi-frame images of a single exposure according to the definition, and select the first frame image as a reference frame; a second processing submodule, configured In order to fuse the reference frame based on feature alignment and weighted fusion based on spatial information to the second frame image, and update the fused image to the reference frame; the third processing sub-module is configured to sequentially perform feature alignment based, based on spatial information weighted fusion And the process of updating the reference frame, until the last frame image completes the weighted fusion based on feature alignment and based on spatial information, and obtains the fourth image.

Optionally, the expansion module further includes: for constructing a logarithmic brightness pyramid according to the fourth image, to obtain ambient light brightness of different scales; combining the ambient light brightness of different scales, perform layer-by-layer downlinking on the fourth image according to the logarithmic brightness pyramid. Sampling reconstruction and mapping are performed to obtain the logarithmic reflection amount of the object surface at each pixel position; the logarithmic reflection amount of the object surface is mapped to the image value range by using the local mean and mean square error information to obtain a first image.

Optionally, the enhancement unit includes: a building module configured to use the constructed sample training set for training to obtain a pre-trained enhancement model; an enhancement module configured to image the target area of the first image according to the pre-trained enhancement model. Enhanced to obtain a second image containing clear target areas.

Optionally, the building module further includes: a fourth processing submodule, configured to obtain a first quality sample set collected by the electronic device, and obtain a second quality sample set at the same location, where the image quality of the second quality sample set is higher than that of the first quality sample set. quality sample set; the fifth processing submodule is configured to group the first quality sample set and the second quality sample set according to different magnifications of the equipment, and align the grouped sample sets according to the features to obtain the first sample set ; The sixth processing sub-module is configured to send the first sample set to the AI training engine to obtain a pre-trained enhanced model.

According to another aspect of the embodiments of the present invention, a storage medium is further provided, and the storage medium includes a stored program, wherein the program executes any one of the above image processing methods.

According to another aspect of the embodiments of the present invention, a processor is further provided, and the processor is used for running a program; wherein, any one of the above image processing methods is executed when the program is running.

In the embodiment of the present invention, the following steps are performed: obtaining a single exposure multi-frame image according to a first exposure parameter, and the single-exposure multi-frame image is an image with the same exposure parameter including the target area; A composite process is performed to obtain a first image. Multi-frame fusion and image enhancement processing can be performed on multi-frame images with a single exposure to optimize image quality and obtain a clearer and brighter image containing the target area, which solves the problem of serious loss of details, ghosting and motion in the captured target image. Technical issues with insufficient scope.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

1 is a flowchart of an optional image processing method according to an embodiment of the present invention;

FIG. 2 is a flowchart of an optional single-exposure multi-frame synthesis method according to an embodiment of the present invention;

3 is a flowchart of an optional image processing method according to an embodiment of the present invention;

4 is a flowchart of an optional image enhancement method according to an embodiment of the present invention;

FIG. 5 is a structural block diagram of an optional image processing apparatus according to an embodiment of the present invention.

detailed description

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the order so used may be interchanged under appropriate circumstances so that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

The embodiments of the present invention may be applied to electronic devices having a camera unit, and the electronic devices may include: smart phones, tablet computers, desktop computers, personal digital assistants (PDAs), portable multimedia players (PMPs), cameras, and the like.

The following describes a flowchart of an optional image processing method according to an embodiment of the present invention. It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system, such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowcharts, in some cases, Steps shown or described may be performed in an order different from that herein.

Referring to FIG. 1 , it is a flowchart of an optional image processing method according to an embodiment of the present invention. In this embodiment, the moon is taken as an example for description. As shown in Figure 1, the image processing method includes the following steps:

S12, obtaining a single exposure multi-frame image according to the first exposure parameter; wherein, the single-exposure multi-frame image is an image with the same exposure parameter including the target area;

As an example, after determining the first exposure parameter of the captured moon image, the electronic device may capture multiple images of the moon through its camera unit according to the first exposure parameter, and collect multiple frames of images with a single exposure, wherein the single exposure The multiple frames of images are moon images with the same exposure parameters. It should be noted that the multi-frame images are not required to be strictly continuous, and can be shot physically completely continuously, or images can be acquired at intervals of several frames, and the total shooting time can be controlled within a certain range.

S14, synthesizing the single-exposed multi-frame images to obtain a first image;

In the embodiment of the present invention, through the above steps, multi-frame fusion and image enhancement processing can be performed on a multi-frame moon image with a single exposure, so as to optimize the image quality, obtain a clearer and brighter moon image, and solve the problem of the appearance of the captured moon image. Technical issues with severe loss of detail, ghosting, and insufficient dynamic range.

In an optional embodiment, the above-mentioned image processing method may further include step S10 for determining the first exposure parameter, specifically, the first exposure parameter may be determined by metering the target area in the preview image or in a preset The first exposure parameter is selected within the range of .

Wherein, in combination with the process of photographing the moon, determining the first exposure parameter by metering the target area in the preview image includes: performing feature detection in the preview image to obtain the moon area; metering the moon area to obtain the average brightness of the moon area; The average brightness of the area determines the first exposure parameter. Alternatively, those skilled in the art may preset a range of exposure parameters according to factors such as characteristics of the shooting target and the background environment where the target is located, and select the first exposure parameter within the range.

In addition, in the embodiment of the present invention, feature detection is performed on the preview image of the electronic device to obtain the moon area in the preview image. The electronic device can detect the moon in the preview image through a preset detection algorithm. The algorithm of the present invention for detecting the moon area Not limited. The accurate selection of a single exposure is mainly based on the accurate identification of the moon area and the accurate detection of the brightness. Similarly, the electronic device can meter the moon area in the preview interface through the preset metering algorithm to obtain the average brightness of the moon area. The invention does not limit the metering algorithm. In addition, the present application also pre-establishes the mapping relationship between the brightness of the moon area and the exposure parameters. After obtaining the average brightness of the moon area in the preview image, the electronic device can determine the electronic device according to the measured average brightness of the moon area and the pre-established mapping relationship. Exposure parameters of the device in the current state, that is, the first exposure parameters.

Step S14 will be described in detail below. Referring to FIG. 2 , it is a flowchart of an optional single-exposure multi-frame synthesis method according to an embodiment of the present invention.

In the embodiment of the present invention, in step S14, performing synthesis processing on the single-exposed multi-frame images, and obtaining the first image may include:

S140: Perform multi-frame fusion on a single-exposed multi-frame image using a multi-frame image super-resolution technology to obtain a fourth image;

S142: Perform single-frame dynamic range expansion on the fourth image to obtain the first image.

Optionally, in an embodiment of the present invention, performing multi-frame fusion on a single-exposed multi-frame image using a multi-frame image super-resolution technology to obtain a fourth image includes: selecting clear images from the single-exposed multi-frame images. The frame with the highest degree is used as the reference frame; the remaining frames of the single-exposed multi-frame image are aligned to the reference frame based on features to obtain the aligned multi-frame image; the aligned multi-frame images are weighted and fused based on the spatial information to obtain the fourth image . Usually, the multi-frame color RGB image finally obtained by the user using the electronic device is obtained by the Bayer data collected by the sensor through the demosaicing algorithm, which is essentially up-sampling the undersampled RGB data to the full-size RGB data, which inevitably increases the false information, resulting in lower resolution and higher noise. The super-resolution technology multi-frame fusion technology is based on Bayer data, selects the frame with the highest definition in a single exposure multi-frame image as a reference frame, and aligns the remaining frames with it. When collecting multiple frames of images with a single exposure, there is random displacement between frames. Through the real channel information of the remaining frames with a small random displacement at the channel position of the reference frame, the corresponding channels missing from the under-sampled Bayer data can be supplemented. information. The aligned multi-frame images are weighted and fused based on the spatial information to obtain the fourth image, and the final full-size or even higher-sized RGB image is obtained, thereby improving the resolution of the multi-frame image fusion of a single exposure, and at the same time, there is a certain degree of denoising. Effect.

Optionally, in an embodiment of the present invention, multi-frame fusion is performed on a single-exposed multi-frame image by using a multi-frame image super-resolution technology to obtain a fourth image, which can be fused while aligning, specifically including: The single-exposed multi-frame images are sorted according to the sharpness, and the first frame image is selected as the reference frame; the reference frame is weighted and fused to the second frame image based on feature alignment and spatial information, and the fused image is updated as Reference frame; perform the process of feature alignment, weighted fusion based on spatial information, and update reference frame in sequence, until the last frame image completes the feature alignment and spatial information weighted fusion, and obtains a fourth image. During the fusion process of each input frame of a single exposure multi-frame image, the fusion weight may be calculated by each input frame and the first frame image, or the fusion weight may be calculated by each input frame and the updated reference frame. Since the multi-frame fusion method of aligning and merging adopts the method of not fixing the reference frame, the error of the previous level will be cleared in the process of replacing the reference frame fusion, which can avoid the transmission of errors and stage-by-stage amplification, and ensure the processing accuracy.

Optionally, in an embodiment of the present invention, performing single-frame dynamic range expansion on the fourth image to obtain the first image includes: constructing a logarithmic brightness pyramid according to the fourth image, and obtaining ambient light brightness of different scales; According to the scale of ambient light brightness, the fourth image is down-sampled, reconstructed and mapped layer by layer according to the logarithmic brightness pyramid, and the logarithmic reflection of the object surface at each pixel position is obtained; using the local mean and mean square error information, the logarithm of the object surface The reflection amount is mapped to the image value interval to obtain the first image. Based on the retina-cerebral cortex theory, the human eye perceives the actual image from the product of the ambient illumination and the surface reflection of the object. The ambient illumination component is completely removed in the image and the reflection component is retained to restore the original information of the object, which can achieve the image enhancement effect. When the image is based on the logarithmic domain, the above product relationship can be converted into an addition-subtraction relationship, so the ambient light component can be completely removed from the image while the object reflection component can be retained to enhance the visual effect of the image.

According to the input fourth image after multi-frame fusion, a logarithmic brightness pyramid is constructed, and the fourth image is convolved with Gaussian kernels at different scales to approximate the ambient light brightness, so as to obtain the ambient light brightness of different scales; The ambient light component is used to retain the reflection component to restore the original information of the object. Combined with the ambient light brightness of different scales, the fourth image is down-sampled, reconstructed and mapped layer by layer according to the logarithmic brightness pyramid, and the logarithm of the object surface at each pixel position is obtained. Reflection amount: Using the local mean and mean square error information, the logarithmic reflection amount on the surface of the object is mapped from the logarithmic domain to the normal image numerical range to obtain the final brightness adjustment image, that is, the first image. For another example, in another embodiment, for a very dark scene, before performing single-frame dynamic expansion on the fourth image, the fourth image is first linearly brightened to obtain a fifth image, and the fourth image and the fifth image are obtained by linearly brightening the fourth image. The images are fused by Laplacian pyramid, and then subjected to single-frame dynamic range expansion. Since the fourth image and the fifth image actually participating in the fusion are changed from the same frame of the fourth image, alignment is not required, and there is no ghost image. In this way, whether in a normal scene or a very dark environment, the fused single-frame image can be brightened, the local contrast can be improved, and the saturation can be adjusted, while avoiding ghosting or synthesis anomalies that may be introduced by the multi-frame dynamic range expansion. .

Referring to FIG. 3 , it is a flowchart of an optional image processing method according to an embodiment of the present invention. For the fused image after the fusion of single-exposed multi-frame images, more detailed information of the target image can be recovered by this method. Compared with Figure 1, the image processing method also includes the following steps:

S16, performing image enhancement on the target area of the first image according to the pre-trained enhancement model, to obtain a second image containing a clear target area;

S18: Perform fusion processing on the first image and the second image to obtain a third image.

Step S16 will be described in detail below. Referring to FIG. 4 , it is a flowchart of an optional image enhancement method according to an embodiment of the present invention.

In the embodiment of the present invention, step S16, that is, performing image enhancement on the target area of the first image according to the pre-trained enhancement model to obtain a second image including a clear target area, including:

S162, using the constructed sample training set for training to obtain a pre-trained enhanced model;

S164: Perform image enhancement on the target area of the first image according to the pre-trained enhancement model to obtain a second image including a clear target area.

Optionally, in this embodiment of the present invention, using the constructed sample training set to train the pre-trained enhanced model includes: acquiring a first quality sample set collected by an electronic device, and acquiring a second quality sample set at the same location, wherein , the image quality of the second quality sample set is higher than that of the first quality sample set; the first quality sample set and the second quality sample set are grouped according to the different magnifications of the equipment, and the grouped sample sets are aligned according to the features to obtain the first quality sample set. Sample set; send the first sample set to the AI training engine to obtain a pre-trained enhanced model.

Optionally, in this embodiment of the present invention, acquiring the second quality sample set at the same location includes: high-quality image samples collected by a high-definition electronic device or high-quality image samples stored in a storage device.

Optionally, in this embodiment of the present invention, the grouped sample sets are aligned according to features, wherein the features include features such as color, brightness, and angle of the image groups in the first quality sample set and the second quality sample set, or features around the image groups. Auxiliary calibration point feature.

Referring to FIG. 5 , it is a structural block diagram of an optional image processing apparatus according to an embodiment of the present invention. As shown in Figure 5, the image processing device 4 includes:

The acquiring unit 42 is configured to acquire a single exposure multi-frame image according to the first exposure parameter; wherein, the single-exposure multi-frame image is an image with the same exposure parameter including the target area;

The first fusion unit 44 is configured to perform synthesis processing on a single exposure of multiple frames of images to obtain a first image.

The image processing device 4 may also include:

The enhancement unit 46 is configured to perform image enhancement on the first image according to the pre-trained enhancement model to obtain a second image containing a clear target;

The second fusion unit 48 is configured to perform fusion processing on the first image and the second image to obtain a third image.

In an optional embodiment, the above-mentioned image processing device 4 may further include a detection unit 40 configured to determine the first exposure parameter, specifically, the first exposure parameter may be determined by metering the target area in the preview image or The first exposure parameter is selected within a preset range.

Optionally, in this embodiment of the present invention, the detection unit 40 may include: a detection module 400 configured to perform feature detection in the preview image to obtain a target area; a photometric module 402 configured to perform photometry on the target area , to obtain the average brightness of the target area; the determining module 404 is configured to determine the first exposure parameter according to the average brightness of the target area. Alternatively, those skilled in the art may preset a range of exposure parameters according to factors such as characteristics of the shooting target and the background environment where the target is located, and select the first exposure parameter within the range.

Optionally, in this embodiment of the present invention, the first fusion unit 44 may include:

The fusion module 442 is configured to perform multi-frame fusion on the single-exposed multi-frame images using the multi-frame image super-resolution technology to obtain a fourth image;

The expansion module 444 is configured to perform single-frame dynamic range expansion on the fourth image to obtain the first image.

Optionally, in an embodiment of the present invention, the fusion module 442 includes: a reference sub-module 4422, which is configured to select a frame with the highest definition among the multi-frame images of single exposure as a reference frame; an alignment sub-module 4424, which is configured by The remaining frames of the multi-frame images configured as single exposure are aligned to the reference frame based on features to obtain the aligned multi-frame images; the fusion sub-module 4426 is configured to perform weighted fusion of the aligned multi-frame images based on spatial information to obtain the fourth image. image. Usually, the multi-frame color RGB image finally obtained by the user using the electronic device is obtained by the Bayer data collected by the sensor through the demosaicing algorithm. In essence, the undersampled RGB data is up-sampled to the full-size RGB data, which inevitably increases the false value. information, resulting in lower resolution and higher noise. The super-resolution technology multi-frame fusion technology is based on Bayer data, selects the frame with the highest definition in a single exposure multi-frame image as a reference frame, and aligns the remaining frames with it. When collecting multiple frames of images with a single exposure, there is random displacement between frames. Through the real channel information of the remaining frames with a small random displacement at the channel position of the reference frame, the corresponding channels missing from the under-sampled Bayer data can be supplemented. information. The aligned multi-frame images are weighted and fused based on the spatial information to obtain the fourth image, and the final full-size or even higher-sized RGB image is obtained, thereby improving the resolution of the multi-frame image fusion of a single exposure, and at the same time, there is a certain degree of denoising. Effect.

Optionally, in an embodiment of the present invention, the fusion module 442 includes: a first processing submodule 4423, configured to sort the multi-frame images of a single exposure according to the sharpness, and select the first frame image as the reference frame; the second processing sub-module 4425 is configured to fuse the reference frame into the second frame image based on feature alignment and spatial information weighting, and update the fused image to the reference frame; the third processing sub-module 4427, is It is configured to perform the process of feature alignment, weighted fusion based on spatial information, and updating the reference frame in sequence, until the last frame image completes the feature alignment and spatial information weighted fusion to obtain a fourth image. During the fusion process of each input frame of a single exposure multi-frame image, the fusion weight may be calculated by each input frame and the first frame image, or the fusion weight may be calculated by each input frame and the updated reference frame. Since the multi-frame fusion device for aligning and merging adopts the method of not fixing the reference frame, the error of the previous level will be cleared in the process of updating the reference frame fusion, which can avoid the transmission of errors and stage-by-stage amplification, and ensure the processing accuracy.

Optionally, in an embodiment of the present invention, the expansion module 444 further includes: constructing a logarithmic brightness pyramid according to the fourth image, to obtain ambient light brightness of different scales; combining the ambient light brightness of different scales, according to the logarithmic brightness pyramid The fourth image is down-sampled, reconstructed and mapped layer by layer to obtain the logarithmic reflection of the object surface at each pixel position; using the local mean and mean square error information, the logarithmic reflection of the object surface is mapped to the image numerical range to obtain the first image. . Based on the retina-cerebral cortex theory, the human eye perceives the actual image from the product of the ambient illumination and the surface reflection of the object. The ambient illumination component is completely removed in the image and the reflection component is retained to restore the original information of the object, which can achieve the image enhancement effect. If the image is based on the logarithmic domain, the above product relationship can be converted into an addition-subtraction relationship. After obtaining the ambient illumination component, the reflection component of the object can be retained by completely subtracting the ambient illumination component from the image, thereby achieving the effect of enhancing the image vision.

According to the input fourth image after multi-frame fusion, a logarithmic brightness pyramid is constructed, and the fourth image is convolved with Gaussian kernels at different scales to approximate the ambient light brightness, so as to obtain the ambient light brightness of different scales; The ambient light component is retained and the reflection component is retained to restore the original information of the object. Combined with the ambient light brightness of different scales, the fourth image is down-sampled, reconstructed and mapped layer by layer according to the logarithmic brightness pyramid, and the logarithm of the object surface at each pixel position is obtained. Reflection amount: Using the local mean and mean square error information, the logarithmic reflection amount on the surface of the object is mapped from the logarithmic domain to the normal image numerical range to obtain the final brightness adjustment image, that is, the first image. For another example, in another embodiment, for a very dark scene, before performing single-frame dynamic expansion on the fourth image, the fourth image is first linearly brightened to obtain a fifth image, and the fourth image and the fifth image are obtained by linearly brightening the fourth image. The images are fused by Laplacian pyramid, and then subjected to single-frame dynamic range expansion. Since the fourth image and the fifth image that actually participate in the fusion are changed from the same frame, alignment is not required, and there is no ghost image. In this way, whether in normal scenes or extremely dark environments, the fused single-frame image can brighten dark parts, improve local contrast, and adjust saturation, while avoiding ghosting or synthesis that may be introduced due to multi-frame dynamic range expansion. abnormal.

Optionally, in this embodiment of the present invention, the enhancement unit 46 may include:

Building module 462, configured to use the constructed sample training set for training to obtain a pre-trained enhanced model;

The enhancement module 464 is configured to perform image enhancement on the first image according to the pre-trained enhancement model to obtain a second image including a clear target area.

Optionally, in this embodiment of the present invention, the building module 462 may include: acquiring a first quality sample set collected by the electronic device, and acquiring a second quality sample set at the same location, where the image quality of the second quality sample set is higher than the first quality Sample set; group the first quality sample set and the second quality sample set according to the different magnifications of the equipment, and align the grouped sample sets according to the features to obtain the first sample set; send the first sample set to AI Train the engine to obtain a pre-trained augmented model.

Optionally, in this embodiment of the present invention, acquiring the second quality sample set at the same location includes: high-quality image samples collected by a high-definition electronic device or high-quality image samples stored in a storage device.

Optionally, in this embodiment of the present invention, the grouped sample sets are aligned according to features, wherein the features include features such as color, brightness, and angle of the image groups in the first quality sample set and the second quality sample set, or the periphery of the image groups. The auxiliary calibration point feature of .

According to another aspect of the embodiments of the present invention, a storage medium is further provided, and the storage medium includes a stored program, wherein the program executes any one of the above image processing methods.

According to another aspect of the embodiments of the present invention, a processor is further provided, and the processor is used for running a program; wherein, any one of the above image processing methods is executed when the program is running.

It should be noted that the above-mentioned image processing method or image processing device is not limited to photographing the moon, but is also suitable for photographing distant objects with obvious contrast with their backgrounds, such as the sun, lighthouses, stars, lights on buildings, etc. Both can solve the technical problems of severe loss of detail, ghosting and insufficient dynamic range in captured images.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are only illustrative, for example, the division of the units may be a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (24)

1. An image processing method for an electronic device with a camera unit, the method comprising:

   Acquiring a single exposure multi-frame image according to the first exposure parameter, the single-exposure multi-frame image is an image with the same exposure parameter including the target area;

   A first image is obtained by synthesizing the single-exposed multi-frame images.
2. The method according to claim 1, wherein the first exposure parameter is determined by metering light on the target area in the preview image, or the first exposure parameter is selected within a preset range.
3. The method of claim 1, wherein the method further comprises:

   Perform image enhancement on the target area of the first image according to the pre-trained enhancement model to obtain a second image including a clear target area;

   Fusion processing is performed on the first image and the second image to obtain a third image.
4. The method according to claim 2, wherein the determining the first exposure parameter by metering light on the target area in the preview image comprises:

   Perform feature detection in the preview image to obtain the target area;

   Perform photometry on the target area to obtain the average brightness of the target area;

   The first exposure parameter is determined according to the average brightness of the target area.
5. The method according to claim 1, wherein, performing a composite process on the single-exposed multi-frame images to obtain a first image, further comprising:

   performing multi-frame fusion on the single-exposed multi-frame images using multi-frame image super-resolution technology to obtain a fourth image;

   Performing single-frame dynamic range expansion on the fourth image to obtain the first image.
6. The method according to claim 5, wherein, performing multi-frame fusion on the single-exposed multi-frame images using a multi-frame image super-resolution technique to obtain a fourth image, comprising:

   Selecting a frame with the highest definition in the single-exposed multi-frame images as a reference frame;

   The remaining frames of the single-exposed multi-frame image are aligned to the reference frame based on features to obtain an aligned multi-frame image;

   The aligned multi-frame images are weighted and fused based on spatial information to obtain the fourth image.
7. The method according to claim 5, wherein, performing multi-frame fusion on the single-exposed multi-frame images using multi-frame image super-resolution technology to obtain a fourth image, further comprising:

   Sorting the multi-frame images of the single exposure according to the sharpness, and selecting the first frame image as the reference frame;

   The reference frame is weighted and fused to the second frame image based on feature alignment and spatial information, and the fused image is updated to the reference frame;

   The process of the feature-based alignment, the weighted fusion based on the spatial information and the updating of the reference frame is performed in sequence, until the last frame image completes the feature-based alignment and the weighted fusion based on the spatial information, and the fourth image is obtained.
8. The image processing method according to claim 5, wherein the performing single-frame dynamic range expansion on the fourth image to obtain the first image comprises:

   constructing a logarithmic brightness pyramid according to the fourth image to obtain ambient brightness of different scales;

   Combined with the ambient light brightness of different scales, perform layer-by-layer downsampling reconstruction and mapping on the fourth image according to the logarithmic brightness pyramid, and obtain the logarithmic reflection amount of the object surface at each pixel position;

   Using the local mean value and mean square error information, the logarithmic reflection amount of the surface of the object is mapped to an image numerical value interval to obtain the first image.
9. The method according to claim 3, wherein the performing image enhancement on the target area of the first image according to the pre-trained enhancement model to obtain the second image including the clear target area comprises: using the constructed sample training set training to obtain the pre-trained enhancement model; image enhancement is performed on the target area of the first image according to the pre-trained enhancement model to obtain the second image including the clear target area.
10. The method according to claim 9, wherein the pre-trained enhanced model obtained by using the constructed sample training set training comprises:

    Obtaining a first quality sample set collected by an electronic device, and obtaining a second quality sample set at the same location, where the image quality of the second quality sample set is higher than that of the first quality sample set;

    Grouping the first quality sample set and the second quality sample set according to different magnifications of the equipment, and aligning the grouped sample sets according to features to obtain the first sample set;

    The first sample set is sent to the AI training engine to obtain the pre-trained enhanced model.
11. The method according to claim 10, wherein the second quality sample set comprises: high-quality image samples collected by a high-definition electronic device or high-quality image samples stored in a storage device.
12. The method according to claim 10, wherein the grouped sample sets are aligned according to features, wherein the features include: features or images of image groups in the first quality sample set and the second quality sample set Auxiliary calibration point feature around the group.
13. An image processing device, comprising:

    an acquisition unit, configured to acquire a single-exposure multi-frame image according to the first exposure parameter, where the single-exposure multi-frame image is an image with the same exposure parameter including a target area;

    The first fusion unit is configured to perform synthesis processing on the single-exposed multi-frame images to obtain a first image.
14. The device according to claim 13, wherein the image processing device further comprises a detection unit, configured to determine the first exposure parameter by metering the target area in the preview image or to select the first exposure parameter within a preset range. Describe the first exposure parameter.
15. The apparatus of claim 13, wherein the apparatus further comprises:

    an enhancement unit, configured to perform image enhancement on the target area of the first image according to the pre-trained enhancement model to obtain a second image including a clear target area;

    The second fusion unit is configured to perform fusion processing on the first image and the second image to obtain a third image.
16. The apparatus of claim 14, wherein the detection unit comprises:

    a detection module, configured to perform feature detection in the preview image to obtain the target area;

    a light metering module configured to perform light metering on the target area to obtain the average brightness of the target area;

    The determining module is configured to determine the first exposure parameter according to the average brightness of the target area.
17. The apparatus of claim 13, wherein the first fusion unit comprises:

    a fusion module, configured to perform multi-frame fusion on the single-exposed multi-frame images using multi-frame image super-resolution technology to obtain a fourth image;

    The expansion module is configured to perform single-frame dynamic range expansion on the fourth image to obtain the first image.
18. The apparatus of claim 17, wherein the fusion module further comprises:

    a reference sub-module, configured to select a frame with the highest definition in the single-exposed multi-frame images as a reference frame;

    an alignment sub-module, configured to align the remaining frames of the single-exposed multi-frame images to the reference frame based on features to obtain aligned multi-frame images;

    The fusion sub-module is configured to perform weighted fusion of the aligned multi-frame images based on spatial information to obtain the fourth image.
19. The apparatus of claim 17, wherein the fusion module further comprises:

    a first processing submodule, configured to sort the multi-frame images of the single exposure according to the sharpness, and select the first frame image as a reference frame;

    The second processing submodule is configured to fuse the reference frame to the second frame image based on feature alignment and spatial information weight, and update the fused image to the reference frame;

    The third processing sub-module is configured to perform the process of feature-based alignment, weighted fusion based on spatial information and update reference frame in sequence, until the last frame of image completes the feature-based alignment and the weighted fusion based on spatial information , to obtain the fourth image.
20. The device according to claim 17, wherein the expansion module further comprises: for constructing a logarithmic brightness pyramid according to the fourth image, to obtain ambient light brightness of different scales; combining the ambient light brightness of different scales, Perform layer-by-layer downsampling reconstruction and mapping on the fourth image according to the logarithmic brightness pyramid, to obtain the logarithmic reflection amount of the object surface at each pixel position; using the local mean and mean square error information, the logarithm The reflection amount is mapped to the image value interval to obtain the first image.
21. The apparatus of claim 15, wherein the enhancement unit comprises:

    a building module configured to train with the constructed sample training set to obtain the pre-trained enhanced model;

    The enhancement module is configured to perform image enhancement on the target area of the first image according to the pre-trained enhancement model to obtain the second image including the clear target area.
22. The apparatus of claim 21, wherein the building block further comprises:

    a fourth processing submodule, configured to obtain a first quality sample set collected by the electronic device, and obtain a second quality sample set at the same location, where the image quality of the second quality sample set is higher than that of the first quality sample set;

    a fifth processing sub-module, configured to group the first quality sample set and the second quality sample set according to different magnifications of the equipment, and align the grouped sample sets according to features to obtain a first sample set ;

    The sixth processing sub-module is configured to send the first sample set to the AI training engine to obtain the pre-trained enhanced model.
23. A storage medium, wherein the storage medium includes a stored program, wherein the program executes the image processing method according to any one of claims 1 to 12.
24. A processor, wherein the processor is used to run a program, wherein the image processing method according to any one of claims 1 to 12 is executed when the program is run.

Images