WO2017140182A1

WO2017140182A1 - Image synthesis method and apparatus, and storage medium

Info

Publication number: WO2017140182A1
Application number: PCT/CN2016/112498
Authority: WO
Inventors: 戴向东
Original assignee: 努比亚技术有限公司
Priority date: 2016-02-15
Filing date: 2016-12-27
Publication date: 2017-08-24
Also published as: CN105744159B; CN105744159A

Abstract

Disclosed are an image synthesis method and apparatus, and a storage medium. The method comprises: obtaining at least two initial images, and converting the at least two initial images into at least two to-be-processed images, respectively, wherein the initial images are images based on a first color space, and the to-be-processed images are images based on a second color space; determining a high-frequency image portion and a low-frequency image portion corresponding to each of the to-be-processed images on the basis of the at least two to-be-processed images; and synthesizing the at least two to-be-processed images on the basis of the high-frequency image portions and the low-frequency image portions of the to-be-processed images and feature weights of the to-be-processed images, to obtain the synthesized image corresponding to the at least two initial images.

Description

Image synthesis method and device, storage medium

Technical field

The present invention relates to image processing technologies in the field of signal processing, and in particular, to an image synthesis method and apparatus, and a storage medium.

Background technique

Currently, high dynamic range images (HDR, High-Dynamic Range) provide more dynamic range and image detail than normal images, and low dynamic range images (LDR, Low-Dynamic Range) according to different exposure times. The image, using the LDR image corresponding to the best detail for each exposure time to synthesize the final HDR image, can better reflect the visual effect in the real environment. The current HDR algorithm synthesis mainly has two strategies. The first one is to shoot different exposure images. By estimating the camera's illumination response curve, the image's brightness range is mapped from a low dynamic range to a high dynamic range, and then through the tone mapping algorithm. The image is mapped to the number of images suitable for image display viewing; the second is to capture a single image, and the image contrast and brightness adjustment will enhance the contrast of the underexposed area of the image, and the contrast suppression in the overexposed area; the first method Through the principle of physical imaging camera response curve, a more natural HDR image can be obtained, but the process is more complicated and the algorithm complexity is higher; the second method is relatively straightforward, the method complexity is not high, and the underexposed area can be repaired, but It is difficult to suppress the reduction to the actual scene brightness in the overexposed area.

Summary of the invention

In view of this, the main purpose of the embodiments of the present invention is to provide an image synthesizing method and apparatus, and a storage medium, which can solve at least the above problems in the prior art.

A first aspect of the embodiments of the present invention provides an image synthesizing method, including:

Obtaining at least two initial images, respectively converting the at least two initial images into at least two to-be-processed images; wherein the initial image is an image based on a first color space, and the image to be processed is based on a second An image of a color space;

Calculating a feature weight for the at least two to-be-processed images, wherein the feature weight is a set of weights for each pixel of the image to be processed;

Determining, according to the at least two images to be processed, a high frequency image portion and a low frequency image portion corresponding to each of the to-be-processed images;

And superimposing the at least two to-be-processed images to obtain a fused image corresponding to the at least two initial images, based on the high-frequency image portion of the image to be processed, the low-frequency image portion, and the feature weight of the image to be processed.

A second aspect of the embodiments of the present invention provides an image synthesizing apparatus, where the apparatus includes:

And an acquiring unit configured to acquire at least two initial images, respectively converting the at least two initial images into at least two to-be-processed images; wherein the initial image is an image based on the first color space, the to-be-processed The image is an image based on the second color space;

a calculating unit, configured to respectively determine a high frequency image portion and a low frequency image portion corresponding to each of the to-be-processed images based on the at least two to-be-processed images; and calculate feature rights for the at least two to-be-processed images a value; wherein the feature weight is a set of weights for each pixel of the image to be processed;

a merging unit configured to fuse the at least two to-be-processed images based on the high-frequency image portion of the image to be processed, the low-frequency image portion, and the feature weight of the image to be processed, to obtain the at least two The fused image corresponding to the initial image.

A third aspect of the embodiments of the present invention provides a computer storage medium, wherein the computer storage medium stores a computer program for executing the image synthesis method described above.

The image synthesizing method and device and the storage medium provided by the embodiments of the present invention are obtained. Having two initial images, converting the at least two initial images into at least two to-be-processed images; calculating feature weights for the at least two images to be processed; and obtaining high-frequency images based on the image to be processed And a part of the low-frequency image portion and the feature weight of the image to be processed, and merging the at least two to-be-processed images to obtain a fused image corresponding to the at least two initial images. In this way, the fusion of the plurality of images is performed based on the feature weights corresponding to the different pixel points in each image, so that the image obtained by the final fusion can ensure the quality of the image in detail.

DRAWINGS

1 is a schematic structural diagram of hardware of a mobile terminal that implements various embodiments of the present invention;

2 is a schematic diagram of a wireless communication system of the mobile terminal shown in FIG. 1;

3 is a schematic flowchart 1 of an image synthesizing method according to an embodiment of the present invention;

4 is a schematic diagram of two color spaces according to an embodiment of the present invention;

FIG. 5a is a schematic diagram of two initial images according to an embodiment of the present invention; FIG.

FIG. 5b is a schematic diagram of three initial images according to an embodiment of the present invention; FIG.

6 is a second schematic flowchart of an image synthesizing method according to an embodiment of the present invention;

FIG. 7 is a diagram showing an example of wavelet decomposition for an image according to an embodiment of the present invention; FIG.

8 is a schematic diagram of processing logic according to an embodiment of the present invention;

9 is a schematic diagram of a synthesis result according to an embodiment of the present invention;

10 is a schematic view showing the comparison of effects of another embodiment of the present invention;

11 is a schematic structural diagram of an image synthesizing apparatus according to an embodiment of the present invention.

detailed description

It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

It should be noted that the image processing apparatus in this embodiment may be a mobile terminal, or Think of the server, or it can be a terminal device such as a personal computer, a laptop, or a camera.

Hereinafter, the image processing apparatus may be described as an example of a mobile terminal, and the mobile terminal may be implemented in various forms. For example, the terminals described in the present invention may include, for example, mobile phones, smart phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), navigation devices, and the like. Mobile terminals and fixed terminals such as digital TVs, desktop computers, and the like. In the following, it is assumed that the terminal is a mobile terminal. However, those skilled in the art will appreciate that configurations in accordance with embodiments of the present invention can be applied to fixed type terminals in addition to components that are specifically for mobile purposes.

FIG. 1 is a schematic diagram showing the hardware structure of a mobile terminal embodying various embodiments of the present invention.

The mobile terminal 100 may include an audio/video (A/V) input unit 120, an output unit 150, a memory 160, an interface unit 170, a controller 180, a power supply unit 190, a sensing unit, and the like. Figure 1 illustrates a mobile terminal having various components, but it should be understood that not all illustrated components are required to be implemented. More or fewer components can be implemented instead. The elements of the mobile terminal will be described in detail below.

The A/V input unit 120 is for receiving an audio or video signal. The A/V input unit 120 may include a camera 121 that processes image data of still pictures or video obtained by an image capturing device in a video capturing mode or an image capturing mode. The processed image frame can be displayed on the display unit 151. The image frames processed by the camera 121 may be stored in the memory 160 (or other storage medium) or transmitted via the wireless communication unit, and two or more cameras 121 may be provided according to the configuration of the mobile terminal.

The interface unit 170 serves as an interface through which at least one external device can connect with the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, and an audio input/output. (I/O) port, video I/O port, headphone port, and more. The identification module may be storing various information for verifying that the user uses the mobile terminal 100 and And may include a User Identity Module (UIM), a Customer Identity Module (SIM), a Universal Customer Identity Module (USIM), and the like. In addition, the device having the identification module (hereinafter referred to as "identification device") may take the form of a smart card, and thus the identification device may be connected to the mobile terminal 100 via a port or other connection device. The interface unit 170 can be configured to receive input from an external device (eg, data information, power, etc.) and transmit the received input to one or more components within the mobile terminal 100 or can be used at the mobile terminal and external device Transfer data between. In addition, when the mobile terminal 100 is connected to the external base, the interface unit 170 may function as a path through which power is supplied from the base to the mobile terminal 100 or may be used as a transmission of various command signals allowing input from the base to the mobile terminal 100 The path to the terminal. Various command signals or power input from the base can be used as signals for identifying whether the mobile terminal is accurately mounted on the base. Output unit 150 is configured to provide an output signal (eg, an audio signal, a video signal, an alarm signal, a vibration signal, etc.) in a visual, audio, and/or tactile manner.

The output unit 150 may include a display unit 151, an audio output module, an alarm unit, and the like. The display unit 151 can display information processed in the mobile terminal 100. For example, when the mobile terminal 100 is in a phone call mode, the display unit 151 can display a user interface (UI) or a graphical user interface (GUI) related to a call or other communication (eg, text messaging, multimedia file download, etc.). When the mobile terminal 100 is in a video call mode or an image capturing mode, the display unit 151 may display a captured image and/or a received image, a UI or GUI showing a video or image and related functions, and the like. Meanwhile, when the display unit 151 and the touch panel are superposed on each other in the form of a layer to form a touch screen, the display unit 151 can function as an input device and an output device. The display unit 151 may include at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), an organic light emitting diode (OLED) display, a flexible display, a three-dimensional (3D) display, and the like. Some of these displays may be configured to be transparent to allow a user to view from the outside, which may be referred to as a transparent display, and a typical transparent display may be, for example, a TOLED (Transparent Organic Light Emitting Diode) display or the like. Mobile terminal 100 may include two, depending on the particular desired implementation. Or more display units (or other display devices), for example, the mobile terminal may include an external display unit (not shown) and an internal display unit (not shown). The touch screen can be used to detect touch input pressure as well as touch input position and touch input area.

The memory 160 may store a software program or the like that performs processing and control operations performed by the controller 180, or may temporarily store data (for example, a phone book, a message, a still image, a video, and the like) that has been output or is to be output. Moreover, the memory 160 can store data regarding vibrations and audio signals of various manners that are output when a touch is applied to the touch screen. The memory 160 may include at least one type of storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (eg, SD or DX memory, etc.), a random access memory (RAM), a static random access memory ( SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, and the like. Moreover, the mobile terminal 100 can cooperate with a network storage device that performs a storage function of the memory 160 through a network connection.

The controller 180 typically controls the overall operation of the mobile terminal. For example, the controller 180 performs the control and processing associated with voice calls, data communications, video calls, and the like. The controller 180 may perform a pattern recognition process to recognize a handwriting input or a picture drawing input performed on the touch screen as a character or an image.

The power supply unit 190 receives external power or internal power under the control of the controller 180 and provides appropriate power required to operate the various components and components.

The various embodiments described herein can be implemented in a computer readable medium using, for example, computer software, hardware, or any combination thereof. For hardware implementations, the embodiments described herein may be through the use of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays ( FPGA, processor, controller, microcontroller, microprocessor, at least one of the electronic units designed to perform the functions described herein, in some cases, such an embodiment may be under control Implemented in the controller 180. For software implementations, implementations such as procedures or functions may be implemented with separate software modules that permit the execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, which can be stored in memory 160 and executed by controller 180.

So far, the mobile terminal has been described in terms of its function. Hereinafter, for the sake of brevity, a slide type mobile terminal among various types of mobile terminals such as a folding type, a bar type, a swing type, a slide type mobile terminal, and the like will be described as an example. Therefore, the present invention can be applied to any type of mobile terminal, and is not limited to a slide type mobile terminal.

The mobile terminal 100 as shown in FIG. 1 may be configured to operate using a communication system such as a wired and wireless communication system and a satellite-based communication system that transmits data via frames or packets.

A communication system in which a mobile terminal according to the present invention can be operated will now be described with reference to FIG.

Such communication systems may use different air interfaces and/or physical layers. For example, air interfaces used by communication systems include, for example, Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), and Universal Mobile Telecommunications System (UMTS) (in particular, Long Term Evolution (LTE)). ), Global System for Mobile Communications (GSM), etc. As a non-limiting example, the following description relates to a CDMA communication system, but such teachings are equally applicable to other types of systems.

Referring to FIG. 2, a CDMA wireless communication system can include a plurality of mobile terminals 100, a plurality of base stations (BS) 270, a base station controller (BSC) 275, and a mobile switching center (MSC) 280. The MSC 280 is configured to interface with a public switched telephone network (PSTN) 290. The MSC 280 is also configured to interface with a BSC 275 that can be coupled to the BS 270 via a backhaul line. The backhaul line can be constructed in accordance with any of a number of known interfaces including, for example, E1/T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL, or xDSL. It will be appreciated that the system as shown in FIG. 2 can include multiple BSCs 275.

Each BS 270 can serve one or more partitions (or regions), each of which is covered by a multi-directional antenna or an antenna directed to a particular direction radially away from the BS 270. Or, each partition can Covered by two or more antennas for diversity reception. Each BS 270 can be configured to support multiple frequency allocations, and each frequency allocation has a particular frequency spectrum (eg, 1.25 MHz, 5 MHz, etc.).

The intersection of partitioning and frequency allocation can be referred to as a CDMA channel. BS 270 may also be referred to as a Base Transceiver Subsystem (BTS) or other equivalent terminology. In such a case, the term "base station" can be used to generally refer to a single BSC 275 and at least one BS 270. A base station can also be referred to as a "cell station." Alternatively, each partition of a particular BS 270 may be referred to as a plurality of cellular stations.

As shown in FIG. 2, a broadcast transmitter (BT) 295 transmits a broadcast signal to the mobile terminal 100 operating within the system. A broadcast receiving module may be provided at the mobile terminal 100 to receive a broadcast signal transmitted by the BT 295. In Figure 2, several Global Positioning System (GPS) satellites 300 are shown. The satellite 300 helps locate at least one of the plurality of mobile terminals 100.

In Figure 2, a plurality of satellites 300 are depicted, but it is understood that useful positioning information can be obtained using any number of satellites. A GPS module located in the mobile terminal is typically configured to cooperate with the satellite 300 to obtain desired positioning information. Instead of GPS tracking technology or in addition to GPS tracking technology, other techniques that can track the location of the mobile terminal can be used. Additionally, at least one GPS satellite 300 can selectively or additionally process satellite DMB transmissions.

As a typical operation of a wireless communication system, BS 270 receives reverse link signals from various mobile terminals 100. Mobile terminal 100 typically participates in calls, messaging, and other types of communications. Each reverse link signal received by a particular BS 270 is processed within a particular BS 270. The obtained data is forwarded to the relevant BSC 275. The BSC provides call resource allocation and coordinated mobility management functions including a soft handoff procedure between the BSs 270. The BSC 275 also routes the received data to the MSC 280, which provides additional routing services for interfacing with the PSTN 290. Similarly, PSTN 290 interfaces with MSC 280, which forms an interface with BSC 275, and BSC 275 controls BS 270 accordingly to transmit forward link signals to mobile terminal 100.

The method and device of the present invention are proposed based on the hardware structure and communication system of the above mobile terminal Various embodiments.

Embodiment 1

An embodiment of the present invention provides an image synthesizing method, as shown in FIG. 3, including:

Step 301: Acquire at least two initial images, and convert the at least two initial images into at least two to-be-processed images respectively; wherein the initial image is an image based on a first color space, and the image to be processed is An image based on the second color space;

Step 302: Calculate a feature weight for the at least two to-be-processed images; wherein the feature weight is a set of weights for each pixel of the image to be processed;

Step 303: Determine, according to the at least two to-be-processed images, a high-frequency image portion and a low-frequency image portion corresponding to each of the to-be-processed images;

Step 304: merging the at least two to-be-processed images to obtain corresponding to the at least two initial images, based on the high-frequency image portion of the image to be processed, the low-frequency image portion, and the feature weight of the image to be processed. Fusion image.

Here, the acquiring the at least two initial images includes: acquiring at least two initial images having different exposure amounts for the target object. The at least two initial images may be two initial images or three initial images. It can be understood that the target object may be for the same scene or for the same person, which is not limited in this embodiment.

The first color space may be a red (R), green (G), blue (B) color space; the second color space may be a hue (H), a saturation (S), a brightness (V) color space . The HSV space separates the color and brightness of the image, which is more in line with the human eye's visual experience than the RGB color space. The parameters of the color in the HSV model are: hue (H), saturation (S), and brightness (V). As shown in FIG. 4, the left side of the figure represents the model of the RGB color space, and the right side of the figure represents the model of the HSV color space; the image of the RGB color space into the HSV color space can be calculated by the following formula:

Max(R,G,B)→V

If H<0 then H←H+360.On output 0≤V≤1, 0≤S≤1, 0≤H≤360. That is, when H is less than zero, H is replaced by the value obtained by H+360; and the final output value V is less than or equal to 1 and greater than or equal to 0; S is less than or equal to 1 and greater than or equal to zero; H is less than or equal to 360, and is greater than or equal to 0.

This embodiment describes two scenarios:

Scene 1 : A scenario in which two initial images are used for subsequent processing. In the present scenario, the two initial images may be a first initial image and a second initial image, respectively, wherein the first initial image and the second initial The exposure amount corresponding to the image is different. It is assumed that the exposure amount of the first initial image is larger than the second initial image.

It should be noted that, in this embodiment, it is assumed that the two images have been registered, and the pixels are aligned. For example, the image is as shown in FIG. 5a, wherein the first initial image is a and the second initial image is b.

Calculating the feature weights for the at least two images to be processed, and calculating the feature weights corresponding to each pixel in each image to be processed, that is, for each image to be processed Each pixel is corresponding to a different adjustment value when it is fused.

Preferably, the feature weights in the embodiment may be normalized feature weights, so that the image after the fusion is not exceeded beyond the original range of values.

And combining the at least two to-be-processed images to obtain the at least two initial images corresponding to the high-frequency image portion of the image to be processed and the low-frequency image portion and the feature weight of the image to be processed Fusion image can be: each of the first pending image Calculating the feature weights of the high-frequency image portion of the pixel and the low-frequency image portion and the corresponding pixel, and obtaining a portion of the feature of the pixel to be processed that is retained in the fused image and capable of affecting the final fused image; The high-frequency image portion of each pixel in the second image to be processed and the feature weight of the low-frequency image portion and the corresponding pixel point are calculated, and the pixel point of the second image to be processed is retained in the final fused image. Some features that can affect the final fused image.

Scenario 2: a scenario in which two initial images are used for subsequent processing. In the present scenario, the two initial images may be a first initial image, a second initial image, and a third initial image, where the first initial image, The exposure amounts corresponding to the second initial image and the third initial image are different. It is assumed that the exposure amount of the first initial image is larger than the second initial image, and the exposure amount of the second initial image is larger than the third initial image.

It should be noted that, in this embodiment, it is assumed that three images have been registered, and the pixels are aligned. For example, the image is as shown in FIG. 5b, wherein the first initial image is a, and the second initial image is b. The third initial image is image c.

And combining the at least two to-be-processed images to obtain the at least two initial images corresponding to the high-frequency image portion of the image to be processed and the low-frequency image portion and the feature weight of the image to be processed The fused image may be: calculating a feature weight value of each of the first to-be-processed image, the second to-be-processed image, and the third image to be processed, and the low-frequency image portion and the corresponding pixel, respectively. The image to be processed The pixel retains some of the features in the fused image that can affect the final fused image.

It can be seen that, by adopting the above scheme, at least two initial images are acquired, and the at least two initial images are respectively converted into at least two to-be-processed images; the feature weights for the at least two to-be-processed images are calculated; The high-frequency image portion of the image to be processed and the low-frequency image portion and the feature weight of the image to be processed are fused to the at least two images to be processed to obtain a fused image corresponding to the at least two initial images. In this way, the fusion of the plurality of images is performed based on the feature weights corresponding to the different pixel points in each image, so that the image obtained by the final fusion can ensure the quality of the image in detail.

Embodiment 2

Here, the acquiring the at least two initial images includes: acquiring at least two initial images having different exposure amounts for the target object. The at least two initial images may be two initial images or three initial images. It can be understood that the target object may be for the same scene or for the same person. This embodiment is incorrect. It is limited.

Max(R,G,B)→V

If H<0 then H←H+360.On output 0≤V≤1, 0≤S≤1, 0≤H≤360.

Before the calculating the feature weights for the at least two to-be-processed images, the method further comprises: respectively determining, according to the at least two to-be-processed images, a high-frequency image corresponding to each of the to-be-processed images Part and low frequency image part.

The high frequency image portion and the low frequency image portion corresponding to each of the to-be-processed images may be decomposed for the pixel points in the image to be processed by using wavelet coefficients, for example, the formula Iwave _k (i, j) may be used. A calculation is performed in which I can represent the image to be processed, wave() is a wavelet decomposition function, and (i, j) represents the horizontal and vertical coordinates of the pixel.

The calculating the feature weights for the at least two to-be-processed images may include:

Calculating the area contrast of each pixel in each image to be processed, and the gradient value of each pixel;

Determining a feature weight of each of the to-be-processed images based on an area contrast of each of the pixels and the gradient value;

A normalized feature weight for each image to be processed is determined based on feature weights of the at least two images to be processed.

Wherein, the regional contrast of each pixel point can be calculated by the following formula:

p(i,j) is the pixel value of the pixel, and m(i,j) is the local region average; where M and N represent the position of the largest pixel in the selected region.

The soble operator is used to calculate the gradient of the image in the horizontal and vertical directions. The operator consists of two sets of 3x3 matrices, which are horizontal and vertical, respectively, and are planarly convolved with the image to obtain lateral and longitudinal luminance difference approximations. If I represents the original image, Gx and Gy represent the images detected by the longitudinal and lateral edges, respectively, and GL _{i,j is} the gradient of the pixel points of the image, and the formula is as follows:

Further, determining the feature weight of each of the to-be-processed images based on the region contrast of each of the pixel points and the gradient value may be: a region contrast of each of the pixel points and the Multiplying the gradient values to obtain feature weights corresponding to each pixel of each image to be processed; specifically, the following formula WM(i,j)=CL _i,j *GL _i,j can be used; wherein, WM (i, j) is used to characterize feature weights.

Based on the above solution, the determining the normalized feature weight for each image to be processed may adopt a formula:

Calculate, where n is the number of initial images, for example, n=2 means there are two initial images, n=3 means there are three initial images; WM(I,k) is the fusion weight of the kth image Coefficient, so that the weights of the fusion coefficients of different exposure images are normalized, satisfying

It ensures that the pixels will not exceed the original range of values after image fusion.

The merging the at least two to-be-processed images based on the high-frequency image portion of the image to be processed and the low-frequency image portion and the feature weight of the image to be processed includes:

Multiplying the high frequency image portion of the at least two images to be processed and the low frequency image portion by respective normalized feature weights of each image to be processed to obtain a normalized image to be processed;

The normalized at least two images to be processed are summed, and the at least two images to be processed are merged.

Specifically, the following formula can be used to illustrate:

n is the number of initial images, Iwave _k (i, j) is the wavelet decomposition of any pixel (i, j) point in image I. It can be seen from the above formula calculation that the larger the regional contrast, the larger the gradient feature The more obvious the regional feature of the pixel, the clearer the image detail is, and the image pixel needs to be preserved in the HDR image, so the fusion weight is also relatively large.

And obtaining the fused image corresponding to the at least two initial images, including:

Converting the fused image to obtain a fused image based on the first color space. It can be understood that the conversion described herein can be opposite to the manner in which the first color space provided in the embodiment is converted to the second color space, and finally the image of the RGB color space is obtained.

It can be seen that, by adopting the above scheme, at least two initial images are acquired, and the at least two initial images are respectively converted into at least two to-be-processed images; the feature weights for the at least two to-be-processed images are calculated; a high frequency image portion of the image to be processed and a low frequency image portion, a feature weight of the image to be processed, and the at least two to-be-processed maps The image is merged to obtain a fused image corresponding to the at least two initial images. In this way, the fusion of the plurality of images is performed based on the feature weights corresponding to the different pixel points in each image, so that the image obtained by the final fusion can ensure the quality of the image in detail.

Further, the high-frequency image portion and the low-frequency image portion of the pixel are obtained by wavelet transform, and the HDR image is synthesized by selecting the pixel point satisfying the HDR requirement with the region contrast feature and the gradient joint feature, and the generated HDR image can effectively highlight the scene. The dark details are suppressed and the image is overexposed.

Embodiment 3

The embodiment of the invention further describes a computer storage medium, wherein the computer storage medium stores a computer program for executing the image synthesis method described in the above embodiments.

Embodiment 4

This embodiment is based on the schemes given in the above two embodiments, and a detailed description of the image synthesis method using three initial images with different exposure amounts, as shown in FIG. 6, includes the following steps:

The S100 acquires three images of different exposures.

The S200 converts the image from the RGB color space to the HSV space.

The S300 uses wavelet transform to decompose the HSV image into high frequency and low frequency parts.

S400 regional contrast features and gradient joint feature weight fusion.

The specific implementation is as follows:

The S100 acquires three images of different exposures. The three images are low-exposure, normal-exposure, and over-exposure images. It should be noted that assuming that the three images have been registered, the pixels are aligned. The image is shown in Figure 5b.

The S200 converts the image from the RGB color space to the HSV space. Since the HSV space separates the color and brightness of the image, the intersecting RGB color space conforms to the visual perception of the human eye. The parameters of the color in the HSV model are: hue (H), saturation (S), and brightness (V). As shown in Figure 4.

The S300 uses wavelet transform to decompose the HSV image into high frequency and low frequency parts. Wavelet transform is a multi-scale, multi-resolution decomposition of an image that can be focused on any detail of an image and is called a mathematical microscope. In recent years, with the development of wavelet theory and its applications, wavelet multiresolution decomposition has been used for pixel-level image fusion. The inherent characteristics of wavelet transform have the following advantages in image processing: 1. Perfect reconstruction ability to ensure that there is no information loss and redundant information in the decomposition process; 2. Decompose the image into a combination of average image and detail image , respectively, represent the different structures of the image, so it is easy to extract the structural information and detailed information of the original image; 3. It has a fast algorithm, its role in wavelet transform is equivalent to the role of FFT algorithm in Fourier transform, providing for wavelet transform application The necessary means; 4. Two-dimensional wavelet analysis provides a selective image that matches the direction of the human visual system. Figure 7 shows a schematic diagram of an image of wavelet decomposition. (a) is the original image, and (b), (c), and (d) are wavelet coefficient images decomposed once, twice, and three times.

After S400 is decomposed by wavelet in S300, the wavelet decomposition coefficients of three different exposure images in HSV space are obtained. By observing the images of three different exposures in Figure 5b, it can be found that the underexposed image has better contrast in the area of the highlight detail, the image details are clear, such as the cloud part in the sky; the overexposed image is clear in the dark part. For example, the green grass under the city wall has clear image details; the normal exposed image is generally in the dark part detail and the highlight detail phenotype, and the overall image has a general visual effect.

HDR images are details that need to preserve the dark and highlight details in the scene, enhancing the overall brightness range of the image. Therefore, as shown in Figure 8, after wavelet decomposition, the coefficients of these relatively clear details need to be preserved, and the choice of fusion rules is the key to the fusion algorithm.

Therefore, in this paper, the local area contrast and the global gradient image feature calculation are performed on the decomposed wavelet coefficients, and the weight maps of the three different exposure image fusion coefficients are generated. The calculation process is as follows:

WM(i,j)=CL _i,j *GL _i,j (1)

In the formula (1), i, j is the coordinate of the p point of any pixel in the image, WM(i, j) is the initialization weight of the pixel participating in the fusion algorithm, and CL _{i, j} is the local area contrast of the pixel, GL _{i, j} is the gradient value of the pixel.

In the formula (2), p(i, j) is the pixel value of the pixel, and m(i, j) is the local region average.

The soble operator is used to calculate the gradient of the image in the horizontal and vertical directions. The operator consists of two sets of 3x3 matrices, which are horizontal and vertical, respectively, and are planarly convolved with the image to obtain lateral and longitudinal luminance difference approximations. If I represents the original image, Gx and Gy represent the images detected by the longitudinal and lateral edges, respectively, and G is the gradient of the pixel points of the image. The formula (3) is as follows:

According to the above calculation process, the fusion weight maps of three different exposure image images can be calculated respectively.

In equation (4), WM(I,k) is the fusion weight coefficient of the kth image, so that the weights of the fusion coefficients of different exposure images are normalized, satisfying

It ensures that the pixels will not exceed the original range of values after the image is blended. According to the formula (5), the wavelet coefficients of the three image decompositions can be fused, and the high-frequency coefficients and the low-frequency coefficients are fused in the same manner, and are multiplied by the fusion weight coefficients.

Iwave _k (i, j) is the wavelet decomposition coefficient of any pixel (i, j) point in the image. It can be seen from the above formula calculation that the larger the regional contrast, the larger the gradient feature, the regional characteristics of the pixel The more obvious, the clearer the image details, the more the image pixels need to be preserved in the HDR image, so the fusion weight is also large.

Figure 9 shows two sets of images of different exposure combined HDR. The first set of images is based on the three images of Figure 5b. The composite image is obtained. It can be seen that the sky in the HDR composite image retains the underexposed image area. In clear areas of the sky, the grass on the wall also retains the details of the dark parts of the exposed image area. See the area section of the ellipse mark. This is compared to another HDR algorithm effect. Since another HDR algorithm is not disclosed, first two points need to be clarified. 1 It is not possible to determine whether Qualcomm is synthesized by three images of different exposures; 2 If it is through 3 frames, the synthesis algorithm is unknown. Based on the above two points, this patent will be compared with the performance of Qualcomm HDR algorithm. MTK models currently have no models, which can not be compared temporarily. Here are three sets of test scenarios. Through the test scenario shown in Figure 10, it can be found that the HDR effect obtained by the algorithm of this patent is similar to the Qualcomm effect in terms of detail retention in the dark portion, and another HDR algorithm does not suppress the highlight in terms of highlight suppression in the overexposed region. The details caused the partial pixels to be overexposed. The algorithm of the scheme can effectively suppress the problem of high light overexposure. You can refer to the detail comparison of the red marked area. In terms of image sharpness and saturation, the patent has not been finalized at present, and the overall saturation and sharpness are inferior to the other HDR effects.

Embodiment 5

An embodiment of the present invention provides an image synthesizing apparatus, as shown in FIG.

The acquiring unit 1101 is configured to acquire at least two initial images, and convert the at least two initial images into at least two to-be-processed images respectively; wherein the initial image is an image based on the first color space, and the Processing the image as an image based on the second color space;

The calculating unit 1102 is configured to respectively determine a high frequency image portion and a low frequency image portion corresponding to each of the to-be-processed images based on the at least two to-be-processed images; and calculate features for the at least two to-be-processed images a weight; wherein the feature weight is a set of weights for each pixel of the image to be processed;

The merging unit 1103 is configured to fuse the at least two to-be-processed images based on the high-frequency image portion of the image to be processed, the low-frequency image portion, and the feature weight of the image to be processed, to obtain the at least two The fused image corresponding to the initial image.

Here, the obtaining unit 1101 is further configured to acquire at least two initial images having different exposure amounts for the target object.

Max(R,G,B)→V

If H<0 then H←H+360.On output 0≤V≤1, 0≤S≤1, 0≤H≤360.

Before the calculating the feature weights for the at least two images to be processed, the The calculating unit is configured to determine a high frequency image portion and a low frequency image portion corresponding to each of the to-be-processed images, respectively, based on the at least two to-be-processed images.

The calculating unit is configured to calculate a region contrast of each pixel in each image to be processed, and a gradient value of each pixel; and determine the region based on the region contrast of the each pixel and the gradient value Feature weights for each image to be processed; determining normalized feature weights for each image to be processed based on feature weights of the at least two images to be processed.

The merging unit is configured to multiply the high frequency image portion of the at least two to-be-processed images and the low-frequency image portion, respectively, with a normalized feature weight of each image to be processed, to obtain a normalized The image to be processed; the at least two images to be processed after the normalization are summed, and the at least two images to be processed are merged.

Specifically, the following formula can be used to illustrate:

The merging unit is configured to convert the fused image to obtain a fused image based on the first color space. It can be understood that the conversion described here can be combined with this implementation. The manner in which the first color space provided in the example is converted to the second color space is reversed, and the resulting image is the RGB color space.

In practical applications, the obtaining unit 1101, the computing unit 1102, and the merging unit 1103 can all run on a computer, and can be a central processing unit (CPU), a microprocessor (MPU), or a digital signal processor located on a computer. (DSP), or programmable gate array (FPGA) implementation.

It is to be understood that the term "comprises", "comprising", or any other variants thereof, is intended to encompass a non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes those elements. It also includes other elements that are not explicitly listed, or elements that are inherent to such a process, method, article, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device that comprises the element.

The serial numbers of the embodiments of the present invention are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The device embodiments described above are merely illustrative, examples For example, the division of the unit is only a logical function division, and the actual implementation may have another division manner, for example, multiple units or components may be combined, or may be integrated into another system, or some features may be ignored. Or not. In addition, the coupling, or direct coupling, or communication connection of the components shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may be electrical, mechanical or other forms. of.

The units described above as separate components may or may not be physically separated, and the components displayed as the unit may or may not be physical units, that is, may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated into one unit; The unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

A person skilled in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by using hardware related to the program instructions. The foregoing program may be stored in a computer readable storage medium, and the program is executed when executed. The foregoing storage device includes the following steps: the foregoing storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. A medium that can store program code.

Alternatively, the above-described integrated unit of the present invention may be stored in a computer readable storage medium if it is implemented in the form of a software function module and sold or used as a standalone product. Based on such understanding, the technical solution of the embodiments of the present invention may be embodied in the form of a software product in essence or in the form of a software product stored in a storage medium, including a plurality of instructions. Enabling a computer device (which may be a personal computer, server, or network device, etc.) to perform all of the methods of the various embodiments of the present invention or section. The foregoing storage medium includes various media that can store program codes, such as a mobile storage device, a ROM, a RAM, a magnetic disk, or an optical disk.

The above is only a specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention. It should be covered by the scope of the present invention. Therefore, the scope of the invention should be determined by the scope of the appended claims.

Industrial applicability

The embodiment of the invention can perform the fusion of multiple images based on the feature weights corresponding to different pixel points in each image, thereby ensuring that the image obtained by the final fusion guarantees the quality of the image in detail.

Claims

An image synthesis method, the method comprising:

Obtaining at least two initial images, respectively converting the at least two initial images into at least two to-be-processed images; wherein the initial image is an image based on a first color space, and the image to be processed is based on a second An image of a color space;

Calculating a feature weight for the at least two to-be-processed images, wherein the feature weight is a set of weights for each pixel of the image to be processed;

Determining, according to the at least two images to be processed, a high frequency image portion and a low frequency image portion corresponding to each of the to-be-processed images;

And superimposing the at least two to-be-processed images to obtain a fused image corresponding to the at least two initial images, based on the high-frequency image portion of the image to be processed, the low-frequency image portion, and the feature weight of the image to be processed.
The method of claim 1 wherein said first color space is a red R, green G, blue B color space.
The method of claim 1, wherein the second color space is a hue H, a saturation S, and a brightness V color space.
The method of claim 1 wherein the method further comprises:

Converting the at least two initial images into at least two to-be-processed images according to the following formula; wherein, the formula is:

Max(R,G,B)→V

When H is less than zero, H is replaced by the value obtained by H+360; and the final output value V is less than or equal to 1 and greater than or equal to 0; S is less than or equal to 1 and greater than or equal to zero; H is less than or equal to 360 and greater than or equal to zero.
The method of claim 1, wherein the calculating the feature weights for the at least two images to be processed comprises:

A feature weight corresponding to each pixel in each image to be processed is calculated.
The method of claim 5 wherein the feature weights can be normalized feature weights.
The method according to claim 1 or 5 or 6, wherein the calculating the feature weights for the at least two images to be processed comprises:

Calculating the area contrast of each pixel in each image to be processed, and the gradient value of each pixel;

Determining a feature weight of each of the to-be-processed images based on an area contrast of each of the pixels and the gradient value;

A normalized feature weight for each image to be processed is determined based on feature weights of the at least two images to be processed.
The method according to claim 7, wherein said merging said at least two images to be processed based on a high frequency image portion of said image to be processed, a low frequency image portion, and a feature weight of said image to be processed ,include:

Multiplying the high frequency image portion of the at least two images to be processed and the low frequency image portion by respective normalized feature weights of each image to be processed to obtain a normalized image to be processed;

The normalized at least two images to be processed are summed, and the at least two images to be processed are merged.
The method of claim 1 wherein said obtaining said at least two initials The fused image corresponding to the image, including:

Converting the fused image to obtain a fused image based on the first color space.
The method of claim 1 wherein said obtaining at least two initial images comprises:

At least two initial images having different exposure amounts for the target object are acquired.
An image synthesizing device, the device comprising:

And an acquiring unit configured to acquire at least two initial images, respectively converting the at least two initial images into at least two to-be-processed images; wherein the initial image is an image based on the first color space, the to-be-processed The image is an image based on the second color space;

a calculating unit, configured to respectively determine a high frequency image portion and a low frequency image portion corresponding to each of the to-be-processed images based on the at least two to-be-processed images; and calculate feature rights for the at least two to-be-processed images a value; wherein the feature weight is a set of weights for each pixel of the image to be processed;

a merging unit configured to fuse the at least two to-be-processed images based on the high-frequency image portion of the image to be processed, the low-frequency image portion, and the feature weight of the image to be processed, to obtain the at least two The fused image corresponding to the initial image.
The apparatus according to claim 11, wherein said first color space is a red R, green G, blue B color space; and/or said second color space is a hue H, a saturation S, a brightness V color space.
The apparatus according to claim 11, wherein the obtaining unit is further configured to separately convert the at least two initial images into at least two to-be-processed images according to the following formula; wherein the formula is:

Max(R,G,B)→V

When H is less than zero, H is replaced by the value obtained by H+360; and the final output value V is less than or equal to 1 and greater than or equal to 0; S is less than or equal to 1 and greater than or equal to zero; H is less than or equal to 360 and greater than or equal to zero.
The apparatus according to claim 11, wherein said calculating unit is further configured to calculate a feature weight corresponding to each pixel point in each image to be processed.
The apparatus of claim 14, wherein the feature weights can be normalized feature weights.
The device according to claim 11 or 14 or 15, wherein

The calculating unit is further configured to calculate a region contrast of each pixel in each image to be processed, and a gradient value of each pixel; and determine the region contrast and the gradient value of each pixel a feature weight of each of the images to be processed; determining a normalized feature weight for each image to be processed based on feature weights of the at least two images to be processed.
The device according to claim 16, wherein

The merging unit is further configured to multiply the high frequency image portion of the at least two to-be-processed images and the low-frequency image portion, respectively, with a normalized feature weight of each image to be processed to obtain a normalization a subsequent image to be processed; summing at least two images to be processed after normalization, and fusing the at least two images to be processed.
The apparatus according to claim 11, wherein

The merging unit is further configured to convert the fused image to obtain a basis A fused image of the first color space.
The apparatus according to claim 11, wherein

The acquiring unit is further configured to acquire at least two initial images having different exposure amounts for the target object.
A computer storage medium having stored therein a computer program for performing the image composition method according to any one of claims 1 to 10.