WO2023231583A1 - Image processing method and related device thereof - Google Patents

Abstract

The present application relates to the field of image processing, and provides an image processing method and a related device thereof. The image processing method comprises: displaying a first interface, the first interface comprising a first control; detecting a first operation on the first control; in response to the first operation, obtaining an original image; preprocessing the original image to obtain a first initial image, a second initial image, and a third initial image; performing front-end processing on the second initial image and the third initial image to obtain front-end processed images; and performing fusion processing on the first initial image and the front-end processed images to obtain a target image. In this way, better restoration of details and colors of the images can be realized.

Description

Image processing methods and related equipment

This application claims priority to the Chinese patent application submitted to the State Intellectual Property Office on May 31, 2022, with application number 202210606523.5 and application name "Image processing method and related equipment", the entire content of which is incorporated into this application by reference. middle.

Technical field

The present application relates to the field of image processing, and in particular, to an image processing method and related equipment.

Background technique

Most of the complementary metal oxide semiconductor (CMOS) image sensors currently used for visible light imaging are traditional RGB (red, green, blue) sensors. In other words, this image sensor can only receive red. channel signal, green channel signal and blue channel signal.

Since its small number of spectral response channels limits the upper limit of imaging color restoration, some multispectral response visible light imaging CMOS image sensors, also known as multispectral sensors, have appeared on the market, hoping to solve the problem of imaging color restoration. However, noise problems will occur when using this multispectral sensor for imaging, and usually as the number of spectral response channels increases, the noise problems that occur during imaging will become more serious. Currently, there is no mature processing solution to make good use of this multispectral sensor to achieve the goal of accurate color restoration and noise reduction. Therefore, a new solution is urgently needed.

Contents of the invention

This application provides an image processing method and related equipment. By merging and reorganizing the channel signals of the original image, two frames of images with better details and better color information are generated, and then the two frames of images are fused to generate a target image. This enables better restoration of image details and colors.

In order to achieve the above purpose, this application adopts the following technical solutions:

In the first aspect, an image processing method is provided, which is applied to electronic devices. The method includes:

Display a first interface, the first interface including a first control;

detecting a first operation on the first control;

In response to the first operation, acquiring an original image, the original image including channel signals of at least 4 colors;

Perform preprocessing on the original image to obtain a first initial image, a second initial image and a third initial image, where the preprocessing is used to merge and recombine channel signals of multiple colors included in the original image;

Perform front-end processing on the second initial image and the third initial image to obtain a front-end processed image;

The first initial image and the front-end processed image are fused to obtain a target image.

The image processing method provided by the embodiment of the present application obtains an original image including at least 4 channel signals, and then merges and reorganizes the channel signals in the original image to generate a first initial image, a second initial image, and a third initial image, and then Perform front-end processing on the second initial image and the third initial image to obtain the front-end processed image. Since The first initial image has only undergone merging and reorganization, so the detail richness is higher, and the color accuracy of the front-end processed image is higher. Based on this, the first initial image and the front-end processed image are used to generate the target image, so that image details and Better color restoration.

In a possible implementation of the first aspect, the original image is preprocessed to obtain a first initial image, a second initial image and a third initial image, including:

Perform a four-in-one pixel merging process on the original image to obtain the first initial image;

Perform diagonal pixel merging processing on the original image to obtain the second initial image and the third initial image.

It should be understood that four-in-one pixel merging processing refers to a processing method in which four adjacent pixel values are weighted and averaged and output as a single pixel value, and diagonal pixel merging processing refers to two pixels in the diagonal direction. The weighted average value is output as a single pixel value.

In this implementation, by merging four-in-one pixels of multiple color channel signals in the original image, a first initial image with high detail richness can be obtained; and by merging diagonal pixels on the original image, Two second initial images and a third initial image including different color channel signals are obtained, so that the second initial image and the third initial image can provide better color information for subsequent color restoration.

In a possible implementation of the first aspect, the front-end processing at least includes: downsampling, noise reduction, automatic white balance and/or color correction, and upsampling.

In this implementation, front-end processing is used to process the colors of the second initial image and the third initial image, so that the processed front-end processed image can provide better colors for subsequent processing.

In a possible implementation of the first aspect, fusion processing of the first initial image and the front-end processed image to obtain a target image includes: merging the first initial image and the front-end processed image using The following formula is used for fusion processing: E(f)=w ^d (fd) ² +w ^x (f _x -g _x ) ² +w ^y (f _y -g _y ) ² ; where f is the pixel of the target image value, g is the pixel value of the first initial image, d is the pixel value of the front-end processed image, f _x and f _y are the gradient values of the target image in the x direction and y direction, g _x and g _y is the gradient value of the first initial image, w ^d , w ^x , and ^wy are weight values, and E(f) is the energy function of f; determine the minimum value of E(f) to obtain the target image.

In this implementation, the energy function formula can be used to fuse the first initial image and the front-end processed image, so that the target image is expected to be close to both the gradient value of the first initial image and the pixel value of the front-end processed image, so that The restored target image carries better detailed texture information and color information.

In a possible implementation of the first aspect, the image processing method further includes:

Perform back-end processing on the target image to obtain a color image.

In this implementation, back-end processing is used to further enhance the detail and color of the target image.

In a possible implementation of the first aspect, the back-end processing includes: at least one of demosaicing, gamma correction, and style transformation.

In a possible implementation of the first aspect, the original image includes a red channel signal, a green channel signal, a blue channel signal, a yellow channel signal, a cyan channel signal and a magenta channel signal.

Optionally, the first operation refers to an operation of clicking the camera application.

In a possible implementation manner of the first aspect, the first interface refers to a photographing interface of the electronic device, and the first control refers to a control for instructing to photograph.

Optionally, the first operation refers to an operation of clicking a control for instructing to take a photo. In the first aspect a possible In an implementation manner, the first interface refers to a video shooting interface of the electronic device, and the first control refers to a control used to instruct video shooting.

Optionally, the first operation refers to an operation of clicking a control indicating shooting a video.

The above description takes the first operation as a click operation as an example; the first operation may also include a voice instruction operation, or other operations of instructing the electronic device to take photos or videos; the above is an example and does not limit the application in any way. .

In a second aspect, an electronic device is provided, including a module/unit for performing the first aspect or any method in the first aspect.

In a third aspect, an electronic device is provided, including one or more processors and memories;

The memory is coupled to one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors invoke the computer instructions to cause the electronic device to Perform the first aspect or any method in the first aspect.

In a fourth aspect, a chip system is provided. The chip system is applied to an electronic device. The chip system includes one or more processors. The processor is used to call computer instructions to cause the electronic device to execute the first aspect. Or any method in the first aspect.

In a fifth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When executed by a processor, the program instructions cause the processor to Perform the first aspect or any method in the first aspect.

In a sixth aspect, a computer program product is provided. The computer program product includes: computer program code. When the computer program code is run by an electronic device, the electronic device causes the electronic device to execute the first aspect or any one of the first aspects. method.

This application provides an image processing method and related equipment. By acquiring an original image including at least 4 channel signals, and then merging and reorganizing the channel signals in the original image, a first initial image, a second initial image and a third initial image are generated. The initial image is then subjected to front-end processing on the second initial image and the third initial image to obtain the front-end processed image. Since the first initial image has only undergone merger and reorganization, the richness of details is higher, while the color accuracy of the front-end processed image is lower. High, based on this, the first initial image and the front-end processed image are used to generate the target image, so that better restoration of image details and colors can be achieved.

Description of the drawings

Figure 1 is an imaging schematic diagram of an RGBCMY sensor;

Figure 2 is a spectral response curve of RGBCMY;

Figure 3 is a schematic diagram of using 24 color blocks to determine the CCM matrix;

Figure 4 is a comparison before and after using CCM matrix for processing;

Figure 5 is a schematic diagram of an application scenario;

Figure 6 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

Figure 7 is a schematic diagram of an original image provided by an embodiment of the present application;

Figure 8 is a schematic diagram of Qbin processing of an original image provided by an embodiment of the present application;

Figure 9 is a Dbin processing process provided by the embodiment of the present application;

Figure 10 is a schematic diagram of Dbin processing of an original image provided by an embodiment of the present application;

Figure 11 is a schematic flow chart of another image processing method provided by an embodiment of the present application;

Figure 12 is a schematic diagram of front-end processing provided by an embodiment of the present application;

Figure 13 is a schematic diagram of the effect of the fusion process provided by the embodiment of the present application;

Figure 14 is a schematic flow chart of another image processing method provided by an embodiment of the present application;

Figure 15 is a schematic diagram of back-end processing provided by an embodiment of the present application;

Figure 16 is a schematic diagram of the effect of the image processing method provided by the embodiment of the present application;

Figure 17 is a schematic diagram of a display interface of an electronic device provided by an embodiment of the present application;

Figure 18 is a schematic diagram of a display interface of another electronic device provided by an embodiment of the present application;

Figure 19 is a schematic diagram of a hardware system suitable for the electronic device of the present application;

Figure 20 is a schematic diagram of a software system suitable for the electronic device of the present application;

Figure 21 is a schematic structural diagram of an image processing device provided by an embodiment of the present application;

Figure 22 is a schematic structural diagram of a chip system provided by an embodiment of the present application.

Detailed ways

The technical solutions in this application will be described below with reference to the accompanying drawings.

In the description of the embodiments of this application, unless otherwise stated, "/" means or, for example, A/B can mean A or B; "and/or" in this article is just a way to describe the association of related objects. Relationship means that there can be three relationships. For example, A and/or B can mean: A alone exists, A and B exist at the same time, and B exists alone.

Hereinafter, the terms “first” and “second” are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

First, some terms used in the embodiments of this application are explained to facilitate understanding by those skilled in the art.

1. RGB (red, green, blue) color space or RGB domain refers to a color model related to the structure of the human visual system. Think of all colors as different combinations of red, green, and blue, based on the structure of the human eye. Red, green and blue are called the three primary colors. It should be understood that a primary color refers to a "basic color" that cannot be obtained by mixing other colors.

2. YUV color space or YUV domain refers to a color encoding method. Y represents brightness, and U and V represent chroma. The above-mentioned RGB color space focuses on the human eye's perception of color, while the YUV color space focuses on the visual sensitivity to brightness. The RGB color space and the YUV color space can be converted into each other.

3. Pixel value refers to a set of color components corresponding to each pixel in a color image located in the RGB color space. For example, each pixel corresponds to a set of three primary color components, where the three primary color components are the red component R, the green component G, and the blue component B respectively.

4. Bayer pattern color filter array (CFA). When the image is converted from the actual scene into image data, the image sensor usually receives the red channel signal, the green channel signal and the blue channel signal respectively. information of three channel signals, and then synthesize the information of the three channel signals into a color image. However, in this scheme, three filters are required for each pixel position, which is expensive and difficult to produce. Therefore, it can be used on the image sensor. The surface is covered with a color filter array to obtain information from the three channel signals. Bayer format color filter array refers to filters arranged in a checkerboard format. For example, the minimum repeating unit in the Bayer format color filter array is: one filter to obtain the red channel signal, two filters to obtain the green channel signal, A filter that acquires the blue channel signal is arranged in a 2×2 pattern.

5. Bayer image (bayer image), that is, the image output by the image sensor based on the Bayer format color filter array. Pixels of multiple colors in this image are arranged in a Bayer format. Among them, each pixel in the Bayer format image only corresponds to the channel signal of one color. For example, since human vision is more sensitive to green, it can be set that green pixels (pixels corresponding to green channel signals) account for 50% of all pixels, blue pixels (pixels corresponding to blue channel signals) and red pixels (pixels corresponding to the red channel signal) each account for 25% of all pixels. Among them, the smallest repeating unit of the Bayer format image is: one red pixel, two green pixels and one blue pixel arranged in a 2×2 manner. It should be understood that the RAW domain is the RAW color space, and the Bayer format image can be called an image located in the RAW domain.

6. Grayscale image. Grayscale image is a single-channel image, used to represent different brightness levels. The brightest is completely white, and the darkest is completely black. That is, each pixel in a grayscale image corresponds to a different degree of brightness between black and white. Usually in order to describe the brightness change from the brightest to the darkest, it is divided, for example, into 256 parts, which represents 256 levels of brightness, and is called 256 gray levels (the 0th gray level to the 0th gray level). 255 grayscale).

7. Spectral response (spectral response), which can also be called spectral sensitivity. Spectral response represents the ability of the image sensor to convert incident light energy of different wavelengths into electrical energy. Among them, if the light energy of a certain wavelength of light incident on the image sensor is converted into the number of photons, and the current generated by the image sensor and transmitted to the external circuit is expressed in the number of electrons, it means that each incident photon can be converted into The ability of electrons in the external circuit is called quantum efficiency (QE), and the unit is expressed in percentage. The spectral responsivity of the image sensor depends on the quantum efficiency, as well as parameters such as wavelength and integration time.

8. Auto white balance (AWB)

The human eye has the characteristic of color constancy. In most cases, the color of the same object seen in various light source scenarios is consistent. For example, white paper looks white. Then, in order to eliminate the impact of the light source on the imaging of the image sensor, simulate the color constancy of human vision, and ensure that the white seen in any scene is truly white, therefore, it is necessary to correct the color temperature and automatically adjust the white balance to the appropriate Location.

Different cameras have their own color filter arrays, and the filter colors of different color filter arrays constitute the camera color space (RAW domain or RAW color space). Therefore, the camera color space is not a universal color space. For example, a color filter array with a filter color of RGGB forms a camera color space of RAW RGB. If the Bayer format image or RAW image generated by the color filter array is directly displayed, the image will be greenish. Generally, monitors display based on the standard color space (sRGB), and their reference light source is D65. Therefore, the automatic white balance algorithm needs to correct the image in the RAW domain to the D65 reference light source. Among them, D65 refers to the standard light source with a color temperature of 6500K. Under the D65 light source, white color is generally specified as R=G=B.

9. Color correction. Since there will be a certain gap between the image acquired by the camera and the color people expect, the color needs to be corrected. And because the automatic white balance has already calibrated white, colors other than white can be calibrated through color correction.

10. Color correction matrix (CCM)

The CCM matrix is mainly used to convert image data obtained by automatic white balance into standard color space (sRGB). Since there is a big difference between the spectral response of the CMOS sensor and the spectral response of the human eye to visible light, the color reproduction of the camera is very different from the color of the object perceived by the observer. Therefore, it is necessary to improve the color of the object through the CCM matrix. Color saturation makes the color of the image captured by the camera closer to the perception of the human eye. Among them, the process of correction using the CCM matrix is the process of color correction.

The above is a brief introduction to the terms involved in the embodiments of this application, and will not be described in detail below.

Most of the CMOS image sensors currently used for visible light imaging are traditional RGB sensors. Due to hardware limitations, this image sensor can only receive red channel signals, green channel signals, and blue channel signals. In this way, the number of spectral response channels of the image sensor is very limited, and a small number of spectral response channels will limit the color restoration capability of the image sensor and affect the color and other information of the restored image.

Since the small number of spectral response channels of RGB sensors limits the upper limit of imaging color restoration, some multispectral response visible light imaging CMOS sensors, also known as multispectral sensors, have appeared on the market, hoping to solve the problem of imaging color restoration. , but noise problems will occur when using multispectral sensors for imaging, and usually as the number of spectral response channels increases, the noise problems that occur during imaging will become more serious. Currently, there is no mature processing solution to make good use of this sensor to achieve the goal of accurate color restoration and noise reduction.

It should be understood that multispectral means that the spectral bands used for imaging include 2 or more bands. According to this definition, since the RGB sensor utilizes three bands of red, green and blue, the RGB sensor is also strictly a multispectral response. However, it should be noted that the visible light referred to in this application as a multispectral response CMOS sensors actually refer to other multispectral sensors that have a larger number of spectral response channels than RGB sensors.

For example, the multispectral sensor may be an RGBC sensor, an RGBM sensor, an RGBY sensor, an RGBCM sensor, an RGBCY sensor, an RGBMY sensor, an RGBCMY sensor, etc. It should be understood that the RGBCMY sensor receives red channel signal, green channel signal, blue channel signal, cyan channel signal, magenta channel signal and yellow channel signal. The channel colors received by other sensors are deduced in sequence and will not be described again here.

Of course, the multispectral sensor can also be a sensor that receives signals from other color channels, and can be specifically selected and set according to needs. The embodiments of the present application do not impose any restrictions on this.

For example, Figure 1 provides an imaging schematic diagram of an RGBCMY sensor. The color filter array covered on the surface of the RGBCMY sensor can obtain information from six color channel signals. For example, the minimum repeating unit in the Bayer format color filter array is: two filters to obtain the red channel signal, four filters to obtain the green channel signal, two filters to obtain the blue channel signal, and two filters to obtain the cyan channel signal. Filters for channel signals, two filters for acquiring magenta channel signals, and four filters for acquiring yellow channel signals, arranged in a 4×4 matrix.

Correspondingly, as shown in Figure 1, the minimum repeating unit of the Bayer format image obtained using the RGBCMY sensor is: two red pixels, four green pixels, two blue pixels, two cyan pixels, two magenta pixels, Four yellow pixels, arranged in a 4×4 matrix.

Figure 2 provides a schematic diagram of the spectral response curve of RGBCMY. The horizontal axis represents the wavelength, and the vertical axis represents the spectral responsivity corresponding to different spectra. Among them, the spectral response curve indicated by R represents the different spectral responsivity of red light at different wavelengths, the spectral response curve indicated by G represents the different spectral responsivity of green light at different wavelengths, and the spectral response indicated by B The curve represents the different spectral responsivity of blue light at different wavelengths; the spectral response curve indicated by C represents the different spectral responsivity of cyan light at different wavelengths; the spectral response curve indicated by M represents the different spectral responsivity of magenta light at different wavelengths. Corresponding different spectral responsivities, the spectrum indicated by Y The response curve represents the different spectral responsivity of yellow light at different wavelengths.

Take the RGBCMY sensor as an example. Compared with the RGB sensor, due to the increase in the number of primary colors and the increase in the number of spectral response channels, the RGBCMY sensor can generally achieve relatively better color restoration capabilities, that is, color accuracy.

In the related art, the Bayer format image acquired by the sensor is usually processed through automatic white balance and CCM matrix to restore the scene color. Then, for the Bayer format image acquired by the RGBCMY sensor, it is usually also possible to process the Bayer format image through automatic white balance and CCM. Matrix is processed to restore the scene color. The CCM matrix used in this process needs to be fitted in advance.

However, compared to the CCM matrix corresponding to the RGB sensor, which is a 3×3 matrix, the CCM matrix corresponding to the RGBCMY sensor is a 6×3 matrix, which includes more parameter values. Moreover, for the RGBCMY sensor, usually when fitting the CCM Matrix will encounter over-fitting phenomenon, which will cause some parameter values in the fitted CCM matrix to be too large. In this way, when the CCM matrix is actually used for processing, the noise in the Bayer format image obtained by the RGBCMY sensor will be amplified. , causing serious color noise problems. Among them, color noise refers to colored noise.

Figure 3 provides a schematic diagram of determining the CCM matrix using 24 color blocks.

Illustratively, taking a color temperature of 6500K as an example, as shown in (a) in Figure 3, at this color temperature of 6500K, the image data acquired by the RGBCMY sensor is generated after automatic white balance processing and demosaic (DM) 24 color cards. As shown in (b) in Figure 3, this is a standard 24 color card at a color temperature of 6500K.

Using the 24 color cards shown in (a) and (b) in Figure 3 for matrix fitting, the corresponding CCM matrix at a color temperature of 6500K can be obtained. This CCM matrix represents the 24 colors shown in (a) in Figure 3 When the card is calibrated to the standard 24-color card shown in (b) in Figure 3, the coefficient matrix that needs to be multiplied is required.

Since each color shown in (a) in Figure 3 corresponds to the six primary color values of R, G, B, C, M and Y, while each color shown in (b) in Figure 3 only corresponds to R, There are three primary color values of G and B, so the fitted CCM matrix is a 6×3 matrix, that is, the fitted CCM matrix includes 18 parameter values. Since in the fitting process, over-fitting phenomenon is usually encountered, causing some of the 18 parameter values included in the CCM matrix to be too large, which will lead to the problem of using the fitted parameters during actual processing. The noise of the image processed by the CCM matrix is amplified.

Figure 4 provides a before-and-after comparison of processing using the CCM matrix.

For example, as shown in (a) in Figure 4, the image on the left is a Bayer format image acquired by the RGB sensor, the image on the right is an image processed using the corresponding 3×3 CCM matrix, and the image on the right is Compared to the image on the left, although the noise is amplified, it is not very obvious.

As shown in (b) in Figure 4, the image on the left is a Bayer format image obtained by the RGBCMY sensor, the image on the right is an image processed using the corresponding 6×3 CCM matrix, and the image on the right is relative to the left For the image on the side, the noise is amplified; compared to (a) in Figure 4, the noise problem is more serious.

Therefore, a new solution is urgently needed that can effectively solve many of the above problems.

In view of this, embodiments of the present application provide an image processing method that combines and reorganizes the channel signals of the original image to generate two frames of images with better details and better color information respectively, and then fuses the two frames of images to generate The target image can achieve better restoration of the details and colors of the target image.

The application scenarios of the image processing method provided by the embodiment of the present application are illustrated below with reference to FIG. 5 .

The image processing method provided by the embodiment of the present application can be applied to the field of photography. For example, can be applied in dark halo Take images or record videos in any environment.

Figure 5 shows a schematic diagram of an application scenario provided by the embodiment of the present application. In one example, the electronic device is a mobile phone, which includes a multispectral sensor other than an RGB sensor.

As shown in Figure 5, in response to the user's operation, the electronic device can start the camera application and display a graphical user interface (GUI) as shown in Figure 5. The GUI interface can be called a first interface. The first interface includes multiple shooting mode options and first controls. The multiple shooting modes include, for example, photo taking mode, video recording mode, etc. The first control is, for example, the shooting key 11 , and the shooting key 11 is used to indicate that the current shooting mode is one of the multiple shooting modes.

For example, as shown in Figure 5, when the user starts the camera application and wants to take pictures of outdoor grass and trees at night, the user clicks the shooting button 11 on the first interface, and the electronic device detects the user's click on the shooting button 11. After the operation, in response to the click operation, run the program corresponding to the image processing method provided by the embodiment of the present application to obtain the image.

It should be understood that the multispectral sensor included in this electronic device is not an RGB sensor, for example, an RGBCMY sensor. The spectral response range of this electronic device has been expanded relative to the existing technology, that is to say, the color reduction capability has been improved. However, due to The CCM matrix may have an over-fitting problem, so after processing with the CCM matrix, the noise of the image may be amplified. In this regard, if the electronic device is processed using the image processing method provided by the embodiment of the present application, it can ensure color restoration and reduce noise, thereby improving the quality of the captured image or video.

It should be understood that the above are examples of application scenarios and do not limit the application scenarios of the present application in any way.

The image processing method provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

FIG. 6 shows a schematic flowchart of an image processing method provided by an embodiment of the present application. As shown in Figure 6, the embodiment of the present application provides an image processing method 1. The image processing method 1 includes the following S11 to S16.

S11. Display the first interface, which includes the first control.

S12. Detect the first operation on the first control.

The first control is, for example, the shooting key 11 shown in FIG. 5 , and the first operation is, for example, a click operation. Of course, the first operation can also be other operations, and the embodiment of the present application does not impose any limitation on this.

S13. In response to the first operation, obtain the original image. The original image includes channel signals of at least four colors.

The original image is a Bayer format image, or is located in the RAW domain.

Figure 7 shows a schematic diagram of an original image.

For example, as shown in (a) in Figure 7, the original image includes channel signals that may include 4 colors (for example, t1, t2, t3, and t4), or, as shown in (b) in Figure 7, The original image may include channel signals of 5 colors (eg, t1, t2, t3, t4, and t5), or, as shown in (c) in FIG. 7 , the original image may include channel signals of 6 colors (eg, t1, t2, t3, t4, and t5). t1, t2, t3, t4, t5 and t6). Of course, the original image may also include channel signals of more colors, and the embodiments of the present application do not impose any limitation on this.

The arrangement of channel signals included in the original image can be set and modified as needed. The arrangement shown in FIG. 7 is only an example, and the embodiment of the present application does not impose any restrictions on this.

In response to the first operation, 1 frame, 2 frames, or more than 2 frames of original images may be acquired. Specifically, it can be obtained as needed, and the embodiments of this application do not impose any restrictions on this.

It should be understood that the multi-frame original image can be collected using a multispectral sensor included in the electronic device itself or obtained from other devices. The specific settings can be set as needed, and the embodiments of the present application do not impose any restrictions on this.

It should be understood that when using its own multispectral sensor to acquire multiple frames of original images, the multispectral sensor can simultaneously Multiple frames of original images can be output simultaneously, or multiple frames of original images can be output serially. Specific selections and settings may be required, and the embodiments of the present application do not impose any restrictions on this.

It should also be understood that although multiple frames of original images are output from the multispectral sensor, they can be output simultaneously or serially, but no matter how they are output, the multiple frames of original images are actually images generated by taking the same shot of the scene to be shot. The scene to be shot refers to all objects in the camera's shooting perspective. The scene to be shot can also be called the target scene, or it can also be understood as the scene that the user expects to shoot.

S14. Preprocess the original image to obtain the first initial image, the second initial image and the third initial image.

Among them, the preprocessing is used to merge and recombine multiple color channel signals included in the original image. For example, the preprocessing can include four-in-one pixel binning (quarter binning, Qbin) processing and diagonal pixel binning (diagonal binning, Dbin). )deal with. Of course, other methods can also be used for merging and reorganization, and specific settings and changes can be made as needed. The embodiments of the present application do not impose any restrictions on this.

It should be noted that Qbin processing refers to a processing method in which four adjacent pixel values are weighted and averaged and output as a single pixel value. Dbin processing refers to a weighted average of two pixel values in the diagonal direction and is output as a single pixel value. How a single pixel value is output. The weights assigned during weighting can be set and modified as needed, and the embodiments of the present application do not impose any restrictions on this. For example, the weights can all be set to 1.

Optionally, as shown in Figure 11, the above S14 may include:

S141. Perform Qbin processing on the original image to obtain the first initial image.

S142. Perform Dbin processing on the original image to obtain the second initial image and the third initial image.

For example, FIG. 8 shows a schematic diagram of Qbin processing of original images.

As shown in (a) of Figure 8, it is an original image provided by an embodiment of the present application. The original image includes channel signals of 6 colors. The channel signals of the 6 colors are respectively the red channel signal (R), Green channel signal (G), blue channel signal (B), cyan channel signal (C), magenta channel signal (M) and yellow channel signal (Y), these 6 colors are arranged in a 4×4 arrangement. And repeat with the minimum repeating unit as shown in Figure 1.

As shown in (b) in Figure 8, when Qbin processing is performed on the original image, it is equivalent to combining the pixel channel signal G in the 1st row and 1st column, the pixel channel signal Y in the 1st row and 2nd column, and the pixel channel signal Y in the 1st row and 2nd column. The pixel channel signal Y in the first column of row 2 and the pixel channel signal G in the second row and column of the second row. The pixel values corresponding to the four adjacent pixel channel signals are weighted and averaged to obtain a single pixel value, for example, T1 , the T1 is the pixel value of the pixel in the first row and the first column corresponding to the first initial image.

Then, the pixel channel signal B in the 1st row and 3rd column, the pixel channel C in the 1st row and 4th column, the pixel channel signal C in the 2nd row and 2nd column, and the pixel channel signal B in the 2nd row and 4th column can be , the pixel values corresponding to the four adjacent pixel channel signals are weighted and averaged to obtain a single pixel value, for example, T2. The T2 is the pixel value of the pixel in the first row and the second column corresponding to the first initial image.

By analogy, the first initial image can be determined based on the original image. Each pixel in the first initial image corresponds to a pixel value. Therefore, the first initial image can still be considered as an image in the RAW domain.

It should be understood that since one pixel value of the first initial image is obtained by a weighted average of four adjacent pixel values of the original image, the size of each side of the first initial image is half of the original image. 1. The area of the entire image is one quarter of the original image.

For example, Figure 9 shows a schematic diagram of Dbin processing, and Figure 10 shows a raw image processing Schematic diagram of Dbin processing.

As shown in (a) in Figure 9, when Dbin processing is performed on the image, if the pixel value corresponding to the pixel in row 1 and column 1 is p11, and the pixel value corresponding to the pixel in row 1 and column 2 is p12, then the pixel value corresponding to the pixel in row 1 and column 2 is p12. The pixel value corresponding to the pixel in row 2 and column 2 is p21, and the pixel value corresponding to the pixel in row 2 and column 2 is p22. After Dbin processing, as shown in (b) in Figure 9, the pixel value of the first frame of the image is p21. The pixel value corresponding to the pixel in row 1 and column 1 is b1, b1=(p11+p22)/2, that is, the pixel value corresponding to each pixel in the first frame image is the image phase shown in (a) in Figure 9 The average value of the pixel values of the upper left corner and the lower right corner of the four adjacent pixels. At this time, the weight is assumed to be 1, and so on. As shown in (c) in Figure 9, the first row of the second frame image The pixel value corresponding to the pixel in the first column is c1, c1=(p12+p21)/2, that is, the pixel value corresponding to each pixel in the second frame image is the image shown in (a) in Figure 9. The average value of the pixel values of the lower left corner and the upper right corner of each pixel. At this time, the weight is assumed to be 1, and so on for the others.

In combination with the above, as shown in (a) in Figure 10, for the original image including channel signals of 6 colors provided by the embodiment of the present application, after Dbin processing, as shown in (b) in Figure 10, The pixel value of the pixel in row 1 and column 1 in the second initial image is the average of the pixel values of the pixel in row 1 and column 1 and the pixel in row 2 and column 2 in the original image, for example, it is the pixel corresponding to the green channel signal. value, and so on; as shown in (c) in Figure 10, the pixel value of the pixel in row 1, column 1 in the third initial image is the pixel in row 1, column 2 and the pixel in row 2, column 1 in the original image The average value of the pixel values of the column pixels, for example, the pixel value corresponding to the yellow channel signal, and so on for the others.

Since each pixel in the second initial image and the third initial image still corresponds to a pixel value, the second initial image and the third initial image are also images in the RAW domain.

It should be understood that since the pixel value in the second initial image is obtained by the weighted average of two pixels in the upper left corner and the lower right corner of the four adjacent pixels in the original image, the pixel value in the third initial image is obtained by the adjacent pixels in the original image. It is obtained by weighting the average of the two pixels in the lower left corner and the upper right corner of the four pixels, so the pixel values of the second initial image and the third initial image are different.

It should also be understood that since one pixel value of the second initial image and the third initial image is obtained by a weighted average of two pixel values in the diagonal direction among the four adjacent pixels of the original image, therefore, the second initial image, The size of each side of the third initial image is half the size of the original image, and the size of the entire image is one-fourth the size of the original image.

In addition, in the above processing process, Qbin processing and Dbin processing can be processed in the same image signal processor (image signal processing, ISP), or they can be processed separately in two image signal processors, or they can also be processed in two image signal processors. The processing can be performed in a multispectral sensor, and the specific settings can be set as needed. The embodiments of the present application do not impose any restrictions on this.

S15. Perform front-end processing on the second initial image and the third initial image to obtain a front-end processed image.

Here, it should be noted that the front-end processing described in this application only means that this step is before fusion, so it is called "front-end" processing and has no other meaning. Front-end processing may also be called first processing, etc., and the embodiments of this application do not impose any restrictions on this.

The front-end processing is used to process the colors of the second initial image and the third initial image, so that the front-end processed image obtained after processing can provide better colors for subsequent processing.

It should be understood that since the channel signals included in the second initial image and the third initial image respectively have different colors, the color information of the front-end processed image obtained according to the second initial image and the third initial image is better preserved and also That is to say, front-end processing of images can provide better color reproduction capabilities for subsequent processing.

The front-end processing provided by the embodiment of the present application can be processed in the same ISP as the above-mentioned Qbin processing or Dbin processing, or can be processed in the same ISP as Qbin processing and Dbin processing, or it can also be processed in another The processing can be carried out in separate different ISPs. Of course, the processing can also be carried out in a multispectral sensor. The specific settings can be set according to needs. The embodiments of the present application do not impose any restrictions on this.

Optionally, the front-end processing may at least include: down sampling (down scale sample), noise reduction (denoise), automatic white balance and/or color correction, and up sampling (up sample).

Among them, downsampling is used to split and recombine the channel signals included in the image to reduce the size of the image.

Noise reduction is used to reduce noise in images. Common methods include mean filtering, Gaussian filtering, bilateral filtering, etc. Of course, other methods can also be used for noise reduction, and the embodiments of the present application do not impose any limitations on this.

Automatic white balance is used to correct the down-sampled and noise-reduced image to the D65 reference light source so that its white color appears truly white.

Color correction is used to calibrate the accuracy of colors other than white. Here, it is equivalent to using the CCM matrix to correct multiple color channel signals into three color channel signals, for example, a red channel signal, a green channel signal, and a blue channel signal respectively.

Among them, when performing color correction, the CCM matrix used can be a previously fitted CCM matrix. When there is no CCM matrix under the D65 reference light source, the CCM matrix under the D65 reference light source can be determined by interpolating the CCM matrices corresponding to other color temperatures.

Upsampling is used to enlarge the image size. It should be understood that due to the previous downsampling, the image size is reduced, so accordingly, upsampling needs to be performed to enlarge the image size and restore the image size to facilitate subsequent fusion.

Figure 12 shows a schematic diagram of front-end processing. For example, as shown in Figure 12, front-end processing includes: downsampling, noise reduction, automatic white balance, color correction, and upsampling in the processing order.

If down-sampling is performed on the second initial image and the third initial image obtained in Figure 11, the three color channels included in the second initial image can be split, and then the red channel signals are recombined to generate only one frame. A monochrome channel image that includes a red channel signal, recombining the green channel signals together to produce a frame of a monochrome channel image that includes only the green channel signal, and recombining the blue channel signals together to produce a frame that includes only the blue channel signal Monochrome channel image. In the same way, the three color channels included in the third initial image can be split, and then the yellow channel signals are reorganized together to generate a single-color channel image that only includes the yellow channel signal, and the cyan channel signals are reassembled together. Generating a frame of a monochrome channel image including only the cyan channel signal, and recombining the magenta channel signals together to generate a frame of a monochrome channel image including only the magenta channel signal.

Therefore, after downsampling the second initial image and the third initial image, six frames of monochromatic channel images including different color channel signals can be obtained. The side length of each frame of the monochromatic channel image is the original second initial image. One-half of the image or the third initial image, or in other words, the overall area of each frame of the monochromatic channel image is one-quarter of the original second initial image or the third initial image.

Then, noise reduction is performed on the 6 frames of monochromatic channel images to reduce the original and noise generated during the downsampling process; then, the 6 frames of monochromatic channel images after noise processing are subjected to automatic white balance and color correction processing. Thus, one frame of a monochromatic channel image including only red channel signals, one frame of monochromatic channel images including only green channel signals, and one frame of monochromatic channel images including only blue channel signals can be obtained.

Here, multiple frames of monochromatic channel images are all of the same size.

It can be understood that since the size of the three-frame single-color channel image obtained after color correction is smaller than the size of the first initial image, in order to facilitate subsequent fusion processing, the three-frame three-color channel image can be upsampled. The included red channel signal, green channel signal and blue channel signal are spliced and reorganized to determine a front-end processed image including three color channel signals. As a result, front-end processed images can include better color information. It should be understood that the front-end processed image is a Bayer format image, that is to say, the front-end processed image is an image located in the RAW domain.

On this basis, front-end processing can also include: at least one of dynamic dead pixel compensation (defect pixel correction, DPC), lens shading correction (lens shading correction, LSC) and wide dynamic range adjustment (wide range compression, WDR) .

It should be understood that dynamic bad pixel compensation is used to solve the defects in the array formed by the light collection points on the multi-spectral sensor, or the errors in the process of converting the light signal; usually by taking the average value of other surrounding pixels in the brightness domain to eliminate bad pixels.

It should be understood that lens shading correction is used to eliminate the problem of inconsistency between the color and brightness around the image and the center of the image due to the lens optical system.

Wide dynamic range adjustment refers to: when high-brightness areas illuminated by strong light sources (sunlight, lamps, reflections, etc.) and relatively low-brightness areas such as shadows and backlights coexist in the image, bright areas will appear in the image. Overexposure becomes white, while dark areas become black due to underexposure, seriously affecting image quality. Therefore, the brightest area and the darker area can be adjusted in the same scene, for example, the dark area is made brighter in the image, and the bright area is darkened in the image, so that the processed image can show the difference between the dark area and the bright area. more details.

It should be understood that the front-end processing may include one or more of the above-mentioned processing steps. When the front-end processing includes multiple processing steps, the order of the multiple processing steps may be adjusted as needed. This embodiment of the present application does not impose any limitation on this. In addition, the front-end processing may also include other steps, which may be added as needed. The embodiments of the present application do not impose any restrictions on this.

S16. Fusion of the first initial image and the front-end processed image to obtain the target image.

It should be understood that since the first initial image is directly reconstructed from the channel signals of the original image without any other processing, the first initial image carries more texture details. Then, in order to ensure the richness of details, the first initial image can be used for fusion processing, so that while the color of the scene is restored, the restored image carries more details.

The front-end processed image is obtained from the second initial image and the third initial image after a series of color processing. Some details are missing, but good color information is retained. Therefore, in order to ensure the color richness, the front-end processed image can be used. Fusion processing, so that when the scene color is restored, the restored image has good color.

The resolution of the first initial image is relatively high and can be called a high resolution (HR) image. The resolution of the front-end processed image is relatively low and can be called a low resolution (LR) image.

Among them, the first initial image and the front-end processed image are fused, and the pixel values corresponding to the same position can be added or multiplied according to different weights, or a network model can be used for fusion; of course, it can also be Fusion processing can be performed using other methods, which can be specifically selected and set according to needs. The embodiments of this application do not impose any restrictions on this.

Optionally, the first initial image and the front-end processed image can be fused using the following formula:
E(f)＝w ^d (fd) ² +w ^x (f _x -g _x ) ² +w ^y (f _y -g _y ) ² ;

Among them, f is the pixel value of the target image, g is the pixel value of the first initial image, d is the pixel value of the front-end processed image, f _x and f _y are the gradient values of the target image in the x direction and y direction, g _x and g _y are the gradient values of the first initial image, w ^d , w ^x , and w ^y are weight values, and E(f) is the energy function of f.

Determine the minimum value of E(f) and obtain the target image.

It should be understood that the high-resolution first initial image is equivalent to a constraint map of the gradient value of the target image. Since the first initial image has a higher degree of detail, the closer the gradient value of the target image is to the gradient value of the first initial image. The better. Among them, the gradient value is used to reflect the change rate of image data.

The low-resolution front-end processed image is equivalent to the constraint map of the pixel values of the target image. Since the color richness of the front-end processed image is higher, the closer the pixel value of the target image is to the pixel value of the front-end processed image, the better.

It should be understood that the smaller the value of the sum of the three terms on the right side of the above formula, the smaller the determined value of E(f); when the value of the sum of the three terms on the right side of the formula is the smallest, the determined value of E(f) is minimum value. At this time, the target image can ensure that it is close to the gradient value of the first initial image and the pixel value of the front-end processed image. Therefore, the restored target image carries better detailed texture information and color information. .

Exemplarily, FIG. 13 is a schematic diagram of the effect of the fusion process provided by the embodiment of the present application.

As shown in (a) in Figure 13, it is the first initial image obtained after Qbin processing, with high detail richness; as shown in (b) in Figure 13, it is the front-end processing obtained after Dbin processing and front-end processing Images are rich in color information but low resolution. Based on this, after fusion using the fusion method described in S16 above, the target image as shown in (c) in Figure 13 can be obtained.

Since this fusion method can combine the detail information of the first initial image and the color information of the front-end processed image, the resulting target image has better color compared to the first initial image, and has better texture information compared to the front-end processed image. Richer and higher resolution.

It should be understood that the target image will be displayed on the interface of the electronic device as a captured image, or will only be stored. The specific selection can be made as needed, and the embodiments of the present application do not impose any restrictions on this.

It should also be understood that the above process is only an example, and the sequence can be adjusted as needed. Of course, steps can also be added or reduced, and the embodiments of the present application do not impose any limitations on this.

The image processing method provided by the embodiment of the present application obtains an original image including at least 4 channel signals, and then merges and reorganizes the channel signals in the original image to generate a first initial image, a second initial image, and a third initial image, and then Perform front-end processing on the second initial image and the third initial image to obtain the front-end processed image. Since the first initial image has only undergone merger and reorganization, the richness of details is higher, and the color accuracy of the front-end processed image is higher. Based on this , using the first initial image and the front-end processed image to generate the target image, so that better restoration of image details and colors can be achieved.

Based on the above, Figure 14 provides a schematic flow chart of another image processing method. As shown in Figure 14, the above method may also include S17.

S17. Perform back-end processing on the target image to obtain a color image.

Here, it should be noted that the back-end processing described in this application only means that this step is located after the fusion, so it is called "back-end" processing and has no other meaning. Back-end processing may also be called second processing, etc., and the embodiments of this application do not impose any restrictions on this.

Optionally, backend processing can include: demosaicing.

In this application, demosaicing is used to complement the single-channel signal in each pixel into a multi-channel signal, i.e. A color image in the RGB domain is reconstructed from the image in the RAW domain.

For example, for a target image that includes red, green, and blue channel signals, before demosaicing, a certain pixel in the image only corresponds to one color channel signal, such as only the red channel signal; after demosaicing, the pixel Corresponding to three color channel signals, they are red, green, and blue channel signals. In other words, for pixels with only red channel signals, green and blue channel signals are supplemented. The supplementation of pixels of other colors is analogous and will not be described again here.

Optionally, the first back-end processing may also include: at least one of gamma correction, style transformation (3dimensional look up table, 3DLUT), and RGB domain conversion to YUV domain.

Among them, gamma correction is used to adjust the brightness, contrast, dynamic range, etc. of the image by adjusting the gamma curve; style transformation indicates the style transformation of the color, that is, using a color filter to change the original image style into other image styles. Common styles include movie style, Japanese style, spooky style, etc. RGB domain to YUV domain conversion refers to converting an image in the RGB domain into an image in the YUV domain.

It should be understood that the back-end processing may include one or more of the above-mentioned processing steps. When the back-end processing includes multiple processing steps, the order of the multiple processing steps may be adjusted as needed. The embodiments of the present application do not impose any limitation on this. . In addition, the back-end processing may also include other steps, which may be added as needed. The embodiments of this application do not impose any restrictions on this.

It should be understood that the back-end processing can be processed in the same image signal processor as the pre-processing, front-end processing and/or fusion processing, or the back-end processing can also be processed separately in other image signal processors, as needed. Settings are made, and the embodiments of this application do not impose any restrictions on this.

Figure 15 shows a schematic diagram of back-end processing. For example, as shown in Figure 15, the back-end processing includes: demosaic, gamma correction, style transformation and RGB domain conversion to YUV domain in processing order.

It should be understood that after back-end processing, the target image is converted from the RAW domain to the YUV domain, which can reduce the amount of subsequent transmission data and save bandwidth.

It should be understood that color images are in the YUV domain. The color image may be displayed on the interface of the electronic device 100 as a captured image, or may only be stored. Specifically, the color image may be set as needed, and the embodiment of the present application does not impose any limitation on this.

In this embodiment, a target image containing better detail information and better color information is generated based on the fusion of the first initial image and the front-end processed image, and then back-end processing is performed on the fused target image to obtain its color and The details are further adjusted to achieve better restoration of image details and colors.

It should also be understood that the above process is only an example, and the sequence can be adjusted as needed. Of course, steps can also be added or reduced, and the embodiments of the present application do not impose any limitations on this.

For example, FIG. 16 is a schematic diagram of the effect of the image processing method provided by the embodiment of the present application.

As shown in (a) of Figure 16 , it is a color image obtained without using the image processing method provided by the embodiment of the present application. The content shown in (c) of Figure 16 is part of (a).

As shown in (b) of Figure 16 , it is a color image processed by the image processing method provided by the embodiment of the present application, and the content shown in (d) of Figure 16 is part of (b). Compared with (a) and (c), after being processed by the image processing method provided by the embodiment of the present application, the details and color restoration effect of the image are relatively better.

The image processing method provided by the embodiment of the present application is introduced in detail above. The following describes how the user activates the image processing method provided by the embodiment of the present application in conjunction with the display interface of the electronic device.

FIG. 17 is a schematic diagram of a display interface of an electronic device provided by an embodiment of the present application.

Exemplarily, in response to the user's click operation, when the electronic device 100 runs the camera application, the electronic device 100 displays a shooting interface as shown in (a) of FIG. 17 . The user can perform a sliding operation on the interface so that the shooting key 11 indicates the shooting option "more".

In response to the user's click operation on the shooting option "More", the electronic device 100 displays a shooting interface as shown in (b) of Figure 17 , on which multiple shooting mode options are displayed, such as: professional mode, panorama mode, HDR mode, time-lapse photography mode, watermark mode, detailed color restoration mode, etc. It should be understood that the above shooting mode options are only examples, and can be set and modified as needed. The embodiments of the present application do not impose any restrictions on this.

In response to the user's click operation on the "detailed color restoration" mode, the electronic device 100 can enable the program related to the image processing method provided by the embodiment of the present application during shooting.

FIG. 18 is a schematic diagram of a display interface of another electronic device provided by an embodiment of the present application.

Exemplarily, in response to the user's click operation, when the electronic device 100 runs the camera application, the electronic device 100 displays a shooting interface as shown in (a) of Figure 18 , with "Settings" displayed in the upper right corner of the shooting interface. button. Users can click the "Settings" button on this interface to enter the setting interface to set related functions.

In response to the user's click operation on the "Settings" button, the electronic device 100 displays a setting interface as shown in (b) of FIG. 18 . Multiple functions are displayed on this interface. For example, the photo ratio is used to realize the photo-taking mode. For setting the photo ratio, voice-activated photography is used to set whether to trigger by sound in the photo mode, video resolution is used to adjust the video resolution, and video frame rate is used to adjust the video frame rate. In addition, there are universal reference lines, levels, detailed color restoration, etc.

In response to the user's drag operation of the switch button corresponding to "Detailed Color Restoration", the electronic device 100 can activate the program related to the image processing method provided by the embodiment of the present application when shooting.

It should be understood that the above are only two examples in which the user enables the image processing method provided by the embodiment of the present application from the display interface of the electronic device. Of course, the image processing method provided by the embodiment of the present application can also be enabled in other ways, or it can also be In the shooting process, the image processing method provided by the embodiment of the present application is directly used by default, and the embodiment of the present application does not impose any restrictions on this.

The image processing method and related display interfaces and renderings provided by the embodiments of the present application are described in detail above with reference to Figures 1 to 18; below, the electronic equipment, devices and devices provided by the embodiments of the present application will be described in detail with reference to Figures 19 to 22. chip. It should be understood that the electronic equipment, devices and chips in the embodiments of the present application can perform various image processing methods in the embodiments of the present application. That is, for the specific working processes of the following various products, reference can be made to the corresponding processes in the foregoing method embodiments.

Figure 19 shows a hardware system suitable for the electronic device of the present application. The electronic device 100 may be used to implement the image processing method described in the above method embodiment.

The electronic device 100 may be a mobile phone, a smart screen, a tablet, a wearable electronic device, a vehicle-mounted electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a notebook computer, or a super mobile personal computer ( ultra-mobile personal computer (UMPC), netbook, personal computer Personal digital assistant (personal digital assistant, PDA), projector, etc., the embodiment of the present application does not place any restrictions on the specific type of the electronic device 100.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It should be noted that the structure shown in FIG. 19 does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or less components than those shown in FIG. 19 , or the electronic device 100 may include a combination of some of the components shown in FIG. 19 , or , the electronic device 100 may include sub-components of some of the components shown in FIG. 19 . The components shown in Figure 19 may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), an image signal processor (image signal processor) , ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, neural network processing unit (NPU). Among them, different processing units can be independent devices or integrated devices.

The controller may be the nerve center and command center of the electronic device 100 . The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In this embodiment of the present application, the processor 110 may display a first interface, the first interface including a first control; detect a first operation on the first control; and in response to the first operation, obtain an original image, the original image including at least Channel signals of 4 colors; preprocess the original image to obtain the first initial image, the second initial image and the third initial image; then perform front-end processing on the second initial image and the third initial image to obtain the front-end processed image ; Fusion process the first initial image and the front-end processed image to obtain the target image.

The connection relationship between the modules shown in FIG. 19 is only a schematic illustration and does not constitute a limitation on the connection relationship between the modules of the electronic device 100 . Optionally, each module of the electronic device 100 may also adopt a combination of various connection methods in the above embodiments.

The wireless communication function of the electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, baseband processor and other components.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to Covers single or multiple communication bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.

The electronic device 100 may implement display functions through a GPU, a display screen 194, and an application processor. The GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display 194 may be used to display images or videos.

The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can algorithmically optimize the noise, brightness and color of the image. ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

Camera 193 is used to capture still images or video. The object passes through the lens to produce an optical image that is projected onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard red green blue (RGB), YUV and other format image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3 and MPEG4.

The hardware system of the electronic device 100 is described in detail above, and the software system of the electronic device 100 is introduced below.

Figure 20 is a schematic diagram of a software system of an electronic device provided by an embodiment of the present application.

As shown in Figure 20, the system architecture may include an application layer 210, an application framework layer 220, a hardware abstraction layer 230, a driver layer 240 and a hardware layer 250.

The application layer 210 may include a camera application or other applications. Other applications include but are not limited to: camera, gallery and other applications.

The application framework layer 220 can provide an application programming interface (API) and programming framework to applications in the application layer; the application framework layer can include some predefined functions.

For example, the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and camera equipment; where camera management may be used to provide an access interface for managing cameras; and the camera device may be used to provide an interface for accessing cameras.

Hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer can include the camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call the camera algorithm in the camera algorithm library.

For example, the hardware abstraction layer 230 includes a camera hardware abstraction layer 2301 and a camera algorithm library; the camera algorithm library may include software algorithms; for example, Algorithm 1, Algorithm 2, etc. may be software algorithms for image processing.

The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include camera device drivers, digital signal processor drivers, and graphics processor drivers.

The hardware layer 250 may include multiple image sensors, multiple image signal processors, digital signal processors, graphics processors, and other hardware devices.

For example, the hardware layer 250 includes a sensor and an image signal processor; the sensor may include sensor 1, sensor 2, depth sensor (time of flight, TOF), multispectral sensor, etc. The image signal processor may include image signal processor 1, image signal processor 2, etc.

In this application, by calling the hardware abstraction layer interface in the hardware abstraction layer 230, the connection between the application layer 210 and the application framework layer 220 above the hardware abstraction layer 230 and the driver layer 240 and the hardware layer 250 below can be realized. Camera data transmission and function control.

Among them, in the camera hardware interface layer in the hardware abstraction layer 230, manufacturers can customize functions here according to needs. Compared with the hardware abstraction layer interface, the camera hardware interface layer is more efficient, flexible, and low-latency, and can also make more abundant calls to ISP and GPU to implement image processing. The image input to the hardware abstraction layer 230 may come from an image sensor or a stored picture.

The scheduling layer in the hardware abstraction layer 230 includes general functional interfaces for implementing management and control.

The camera service layer in the hardware abstraction layer 230 is used to access interfaces of ISP and other hardware.

The following exemplifies the workflow of the software and hardware of the electronic device 100 in conjunction with capturing the photographing scene.

The camera application in the application layer may be displayed on the screen of the electronic device 100 in the form of an icon. When the icon of the camera application is clicked by the user to be triggered, the electronic device 100 starts to run the camera application. When the camera application is running on the electronic device 100, the camera application calls the interface corresponding to the camera application in the application framework layer 210, and then starts the camera driver by calling the hardware abstraction layer 230, turning on the multispectral sensor on the electronic device 100. camera and collect raw images through multispectral sensors. At this time, the multispectral sensor can collect according to a certain operating frequency, and the collected images are processed inside the multispectral sensor or transmitted to one or more image signal processors, and then the processed target image or Color images are saved and/or transferred to the monitor for display.

An image processing device 300 for implementing the above image processing method provided by an embodiment of the present application is introduced below. FIG. 21 is a schematic diagram of an image processing device 300 provided by an embodiment of the present application.

As shown in FIG. 21 , the image processing device 300 includes a display unit 310 , an acquisition unit 320 and a processing unit 330 .

The display unit 310 is used to display a first interface, and the first interface includes first controls.

The obtaining unit 320 is used to detect the first operation on the first control.

The processing unit 330 is configured to acquire an original image in response to the first operation, where the original image includes channel signals of at least 4 colors.

The processing unit 330 is also used to pre-process the original image to obtain the first initial image, the second initial image and the third initial image; perform front-end processing on the second initial image and the third initial image to obtain the front-end processed image; An initial image and the front-end processed image are fused to obtain the target image.

It should be noted that the above image processing device 300 is embodied in the form of functional units. The term "unit" here can be implemented in the form of software and/or hardware, and is not specifically limited.

For example, a "unit" may be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor) for executing one or more software or firmware programs. etc.) and memory, merged logic circuitry, and/or other suitable components to support the described functionality.

Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

Embodiments of the present application also provide a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium is run on the image processing device 300, the image processing device 300 causes the image processing device 300 to Execute the image processing method shown previously.

The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or include one or more data storage devices such as servers and data centers that can be integrated with the medium. The available media may be magnetic media (eg, floppy disk, hard disk, tape), optical media, or semiconductor media (eg, solid state disk (SSD)), etc.

Embodiments of the present application also provide a computer program product containing computer instructions, which when run on the image processing device 300 enables the image processing device 300 to execute the image processing method shown above.

Figure 22 is a schematic structural diagram of a chip provided by an embodiment of the present application. The chip shown in Figure 22 can be a general-purpose processor or a special-purpose processor. The chip includes a processor 401. The processor 401 is used to support the image processing device 300 in executing the technical solutions shown above.

Optionally, the chip also includes a transceiver 402, which is used to accept the control of the processor 401 and to support the image processing device 300 in executing the technical solution shown above.

Optionally, the chip shown in Figure 22 may also include: a storage medium 403.

It should be noted that the chip shown in Figure 22 can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGA), programmable logic devices (PLD) , controller, state machine, gate logic, discrete hardware components, any other suitable circuit, or any combination of circuits capable of performing the various functions described throughout this application.

The electronic equipment, image processing device 300, computer storage media, computer program products, and chips provided by the embodiments of the present application are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the methods provided above. The beneficial effects corresponding to the method will not be repeated here.

It should be understood that the above is only to help those skilled in the art better understand the embodiments of the present application, but is not intended to limit the scope of the embodiments of the present application. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples given. For example, some steps in the various embodiments of the above detection method may not be necessary. Or you can add some new steps, etc. Or a combination of any two or more of the above embodiments. Such modified, changed or combined solutions also fall within the scope of the embodiments of the present application.

It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments. The similarities or similarities that are not mentioned can be referred to each other. For the sake of brevity, they will not be described again here.

It should also be understood that the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

It should also be understood that in the embodiments of the present application, "preset" and "predefined" can be realized by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in the device (for example, including electronic devices). , this application does not limit its specific implementation.

It should also be understood that the division of modes, situations, categories and embodiments in the embodiments of this application is only for the convenience of description and should not constitute a special limitation. The features in various modes, categories, situations and embodiments are not contradictory unless cases can be combined.

It should also be understood that in the various embodiments of the present application, if there are no special instructions or logical conflicts, the terms and/or descriptions between different embodiments are consistent and can be referenced to each other. The technical features in different embodiments New embodiments can be formed based on their internal logical relationships.

Finally, it should be noted that the above are only specific implementation modes of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be covered by this application. within the scope of protection applied for. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims (14)

1. An image processing method, characterized in that it is applied to electronic equipment, and the method includes:

   Display a first interface, the first interface including a first control;

   detecting a first operation on the first control;

   In response to the first operation, acquiring an original image, the original image including channel signals of at least 4 colors;

   Perform preprocessing on the original image to obtain a first initial image, a second initial image and a third initial image, where the preprocessing is used to merge and recombine channel signals of multiple colors included in the original image;

   Perform front-end processing on the second initial image and the third initial image to obtain a front-end processed image;

   The first initial image and the front-end processed image are fused to obtain a target image.
2. The image processing method according to claim 1, characterized in that preprocessing the original image to obtain the first initial image, the second initial image and the third initial image includes:

   Perform a four-in-one pixel merging process on the original image to obtain the first initial image;

   Perform diagonal pixel merging processing on the original image to obtain the second initial image and the third initial image.
3. The image processing method according to claim 1 or 2, characterized in that the front-end processing at least includes: downsampling, noise reduction, automatic white balance and/or color correction, and upsampling.
4. The image processing method according to any one of claims 1 to 3, characterized in that, the first initial image and the front-end processed image are fused to obtain a target image, including:

   The first initial image and the front-end processed image are fused using the following formula:
   E(f)＝w ^d (fd) ² +w ^x (f _x -g _x ) ² +w ^y (f _y -g _y ) ² ;

   Where, f is the pixel value of the target image, g is the pixel value of the first initial image, d is the pixel value of the front-end processed image, f _x and f _y are the x and y directions of the target image. The gradient values in the direction, g _x and g _y are the gradient values of the first initial image, w ^d , w ^x , and w ^y are weight values, and E(f) is the energy function of f;

   Determine the minimum value of E(f) to obtain the target image.
5. The image processing method according to claim 4, characterized in that the image processing method further includes:

   Perform back-end processing on the target image to obtain a color image.
6. The image processing method according to claim 5, characterized in that the back-end processing includes: at least one of demosaicing, gamma correction, and style transformation.
7. The image processing method according to any one of claims 1 to 6, characterized in that the original image includes a red channel signal, a green channel signal, a blue channel signal, a yellow channel signal, a cyan channel signal and a magenta channel Signal.
8. An electronic device, characterized in that the electronic device includes:

   one or more processors and memories;

   The memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors invoke the computer instructions to cause the The electronic device executes the image processing method according to any one of claims 1 to 7.
9. A chip system, characterized in that the chip system is applied to electronic equipment, and the chip system includes It includes one or more processors, the processor is used to call computer instructions to cause the electronic device to execute the image processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor causes the processor to execute any one of claims 1 to 7. image processing methods.
11. The image processing method according to claim 1, characterized in that the method includes:

    Display a first interface, the first interface including a first control;

    detecting a first operation on the first control;

    In response to the first operation, acquiring an original image, the original image including channel signals of at least 4 colors;

    Perform a four-in-one pixel merging process on the original image to obtain a first initial image;

    Perform diagonal pixel merging processing in different diagonal directions on the original image to obtain a second initial image and a third initial image respectively;

    Perform front-end processing on the second initial image and the third initial image to obtain a front-end processed image; the front-end processing at least includes: downsampling, noise reduction, automatic white balance and/or color correction, and upsampling;

    The first initial image and the front-end processed image are fused to obtain a target image.
12. The image processing method according to claim 11, characterized in that, the first initial image and the front-end processed image are fused to obtain a target image, including:

    The first initial image and the front-end processed image are fused using the following formula:
    E(f)＝w ^d (fd) ² +w ^x (f _x -g _x ) ² +w ^y (f _y -g _y ) ² ;

    Where, f is the pixel value of the target image, g is the pixel value of the first initial image, d is the pixel value of the front-end processed image, f _x and f _y are the x and y directions of the target image. The gradient values in the direction, g _x and g _y are the gradient values of the first initial image, w ^d , w ^x , and w ^y are weight values, and E(f) is the energy function of f;

    Determine the minimum value of E(f) to obtain the target image.
13. The image processing method according to claim 12, characterized in that the image processing method further includes:

    Perform back-end processing on the target image to obtain a color image; the back-end processing includes: at least one of demosaicing, gamma correction, and style transformation.
14. The image processing method according to any one of claims 11 to 13, characterized in that the original image includes a red channel signal, a green channel signal, a blue channel signal, a yellow channel signal, a cyan channel signal and a magenta channel Signal.