CN115706869A - Terminal image processing method and device and terminal equipment - Google Patents

Abstract

The embodiment of the application provides an image processing method and device for a terminal and terminal equipment, wherein the terminal comprises a black-and-white camera and a color camera, and the method comprises the following steps: after the photographing function of the terminal is operated, acquiring a color RAW image through a color camera and acquiring a black-and-white RAW image through a black-and-white camera in the previewing process; acquiring a shooting instruction, and controlling a color camera to shoot a current shooting scene to acquire a color RAW image of the shooting scene; carrying out RAW domain processing on the color RAW image to obtain a processed color RAW image; processing the processed color RAW image to obtain a color YUV image; processing the black and white RAW image acquired in the preview process through a second Bayer domain processing algorithm link to obtain a black and white YUV image; and fusing the color YUV image and the black-and-white YUV image to obtain a target color image, thereby retaining more image details and improving the color reduction degree of the target color image.

Description

Terminal image processing method, device and terminal equipment

technical field

The embodiments of the present application relate to the technical field of smart terminals, and in particular, to a terminal image processing method, device, and terminal equipment.

Background technique

Taking pictures is now a common function in mobile phones. With the popularization of smart phones, the application of cameras has become more and more extensive, and the camera function has become more and more excellent. Excellent camera performance has become a major selling point of smartphones. Technological innovations around the camera function are emerging in an endless stream, and the aperture and sensor size are getting larger and larger. These technology and hardware upgrades have a significant impact on smartphone photography, both in terms of photo quality and overall quality. The camera experience has been greatly improved. While pursuing the quality of photos, the thinning and lightening of smartphone bodies is also a major trend in the development of smartphones. Therefore, most smartphones now use at least two camera modules, which can not only improve the quality of photos taken, but also meet the needs of mobile phones. The need for a thinner and lighter body.

However, taking a smart phone with two camera modules as an example, two different images are obtained directly by using the dual cameras. Therefore, image fusion algorithms need to be used to fuse the two dual-camera images to obtain the final Enhance images.

The existing image fusion method loses more image details, resulting in poor color reproduction of the final image; especially when shooting scenery and/or still life in a forward-lit scene, due to the ability of a single camera device and the detail enhancement ability of the algorithm Limited, some details, especially high-frequency details, cannot be fully recovered.

Contents of the invention

Embodiments of the present application provide a terminal image processing method, device, server, and terminal equipment. Embodiments of the present application also provide a computer-readable storage medium, so that the terminal equipment first performs RAW on the color RAW image collected by the color camera. After the color YUV image is converted into a color YUV image by domain processing, it is then fused with the black and white YUV image, thereby retaining more image details and improving the color reproduction of the final image.

In the first aspect, the present application provides an image processing method of a terminal. The above-mentioned terminal includes a black-and-white camera and a color camera. The above-mentioned method includes: after the camera function of the terminal is run, during the preview process, the color RAW image is collected by the color camera, and the The black-and-white camera collects black-and-white RAW images; after the terminal device obtains the shooting instruction, it responds to the above-mentioned shooting instruction to obtain the color RAW image collected during the preview process, and performs RAW domain processing on the color RAW image collected during the preview process to obtain a color YUV image; The two-Bayer domain processing algorithm link processes the black and white RAW images collected during the preview process to obtain black and white YUV images. Finally, the terminal device fuses the above color YUV image and the above black and white YUV image to obtain the target color image.

In one possible implementation manner, acquiring the color RAW image captured during the preview process in response to the above shooting instruction may be: the terminal device acquires the color RAW image captured during the preview process from a cache in response to the above shooting instruction.

In one possible implementation, the terminal device performs RAW domain processing on the color RAW image collected during the preview process, and before obtaining the color YUV image, it can also process the color RAW image collected during the preview process through the first Bayer domain processing algorithm link. Image, linearization and dead point correction processing.

In one possible implementation manner, before the terminal device acquires the shooting instruction, it may also determine that the current shooting scene is a non-high dynamic scene according to the color RAW image collected during the preview process.

In one possible implementation manner, the color RAW image collected by the terminal device during the preview process includes at least two frames of color RAW images. In this way, performing RAW domain processing on the color RAW image collected during the preview process to obtain a color YUV image includes: Perform preprocessing on at least two frames of color RAW images; perform noise reduction processing on at least two frames of color RAW images after preprocessing, so as to fuse at least two frames of color RAW images into one frame of color RAW images; convert the color RAW images obtained by fusion Become a color RGB image; adjust the color and brightness of the above color RGB image; convert the color RGB image after adjusting the color and brightness into a color YUV image.

In one possible implementation manner, the terminal device's preprocessing of the above-mentioned at least two frames of color RAW images includes: aligning the sizes of at least two frames of color RAW images; Mapping is performed to obtain the corresponding relationship between pixels; error detection is performed on the corresponding relationship, and the corresponding relationship that occurs in error is corrected.

In one of the possible implementations, the terminal device fuses the color YUV image and the black-and-white YUV image, and the target color image can be obtained by: aligning the size of the color YUV image and the black-and-white YUV image; from the aligned size of the color YUV image and In the black-and-white YUV image after aligning the size, obtain the area that needs to be fused in the color YUV image and the area that needs to be fused in the black-and-white YUV image; The pixels are mapped to obtain the corresponding relationship between pixels; next, the error detection of the above-mentioned corresponding relationship is performed, and the corresponding relationship that occurs is corrected; finally, the color YUV image needs to be fused with the black and white YUV image. The sharp pixels in the region are fused to the corresponding blurred pixels.

In one possible implementation, after the current shooting scene is determined to be a non-high dynamic scene during the preview process, the color RAW images collected by the terminal device through the color camera include short-time exposure color RAW images and normal exposure color RAW images. Image; the above method may also include: using overlapping exposure high dynamic technology to fuse the short-time exposure color RAW image and the normal exposure color RAW image; in the preview process, display the color image obtained by fusion.

In one possible implementation manner, during the preview process, after the terminal device captures the color RAW image through the color camera, it may also cache the above normal exposed color RAW image.

In the terminal image processing method provided in the embodiment of the present application, the terminal device first performs RAW domain processing on the color RAW image, converts the color RAW image into a color YUV image, and then fuses it with the black and white YUV image, so that more images can be retained Details, improve the color reproduction of the target color image.

In a second aspect, an embodiment of the present application provides an image processing apparatus for a terminal, the apparatus is included in a terminal device, and the apparatus has a function of realizing the behavior of the terminal device in the first aspect and possible implementation manners of the first aspect. The functions may be implemented by hardware, or may be implemented by executing corresponding software through hardware. Hardware or software includes one or more modules or units corresponding to the functions described above. For example, a receiving module or unit, a processing module or unit, a sending module or unit, and the like.

In the third aspect, the embodiment of the present application provides a terminal device, including: including a black and white camera and a color camera; one or more processors; memory; multiple application programs; and one or more computer programs, wherein one or more The computer program is stored in the memory, and one or more computer programs include instructions. When the above instructions are executed by the terminal device, the terminal device performs the following steps: For color RAW images, black and white RAW images are collected through a black and white camera; after obtaining the shooting instructions, the color RAW images collected during the preview process are obtained in response to the above shooting instructions, and RAW domain processing is performed on the color RAW images collected during the preview process to obtain color YUV images ; Process the black-and-white RAW image collected in the preview process through the second Bayer domain processing algorithm link to obtain a black-and-white YUV image; fuse the color YUV image and the black-and-white YUV image to obtain a target color image.

In one possible implementation manner, when the above instruction is executed by the terminal device, making the terminal device execute the step of obtaining the color RAW image captured during the preview process in response to the above shooting instruction includes: in response to the above shooting instruction, obtaining from the cache Color RAW image captured during preview.

In one of the possible implementations, when the above instruction is executed by the terminal device, the terminal device performs RAW domain processing on the color RAW image collected during the preview process, and before the step of obtaining the color YUV image, the following steps are also performed: by The first Bayer domain processing algorithm link performs linearization and dead point correction processing on the color RAW images collected during the preview process.

In one possible implementation manner, when the above instruction is executed by the terminal device, before the terminal device executes the step of obtaining the shooting instruction, the following step is also performed: according to the color RAW image collected during the preview process, determine that the current shooting scene is non-high dynamic scene.

In one of the possible implementations, the color RAW image collected during the preview process includes at least two frames of color RAW images; when the above instruction is executed by the terminal device, the terminal device is made to execute the RAW domain image processing on the color RAW image collected during the preview process. Processing, the step of obtaining the color YUV image may include: preprocessing the above-mentioned at least two frames of color RAW images; performing noise reduction processing on the preprocessed at least two frames of color RAW images, so as to fuse at least two frames of color RAW images into one frame Color RAW image; convert the color RAW image obtained by fusion into a color RGB image; adjust the color and brightness of the color RGB image; convert the color RGB image after adjusting color and brightness into a color YUV image.

In one possible implementation manner, when the above instruction is executed by the terminal device, the step of making the terminal device perform the preprocessing of the at least two frames of color RAW images includes: aligning the sizes of the at least two frames of color RAW images; Mapping pixels in every two frames of the at least two frames of color RAW images to obtain correspondences between pixels; performing error detection on the above correspondences, and correcting error correspondences.

In one of the possible implementations, when the above instructions are executed by the terminal device, the terminal device is made to perform fusion of the color YUV image and the black and white YUV image, and the step of obtaining the target color image may be: combining the color YUV image and the black and white YUV image The size alignment; from the color YUV image after the size alignment and the black and white YUV image after the size alignment, obtain the area that needs to be fused in the color YUV image and the area that needs to be fused in the black and white YUV image; the area that needs to be fused in the color YUV image The pixels in the black-and-white YUV image need to be fused with the pixels in the region to obtain the corresponding relationship between the pixels; the above-mentioned corresponding relationship is detected incorrectly, and the wrong corresponding relationship is corrected; the color YUV image needs to be fused. The region and The clear pixels in the black-and-white YUV image that need to be fused are fused to the corresponding blurred pixels.

In one possible implementation, after the current shooting scene is determined to be a non-high dynamic scene during the preview process, the color RAW images collected by the terminal device through the color camera include short-time exposure color RAW images and normal exposure color RAW images. image; when the above instructions are executed by the terminal device, the terminal device will also perform the following steps: using overlapping exposure high dynamic technology to fuse the short-time exposure color RAW image and the normal exposure color RAW image; during the preview process, display Fused color images obtained.

In one of the possible implementations, when the above instruction is executed by the terminal device, the terminal device executes the step of capturing the color RAW image through the color camera during the preview process, and then performs the following step: cache the color RAW image of normal exposure .

It should be understood that the second aspect and the third aspect of the embodiment of the present application are consistent with the technical solution of the first aspect of the embodiment of the present application, and the beneficial effects obtained in each aspect and the corresponding feasible implementation manners are similar, so details are not repeated here.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the above-mentioned computer-readable storage medium, and when running on a computer, causes the computer to execute the method provided in the first aspect.

In a fifth aspect, an embodiment of the present application provides a computer program, which is used to execute the method provided in the first aspect when the above computer program is executed by a computer.

In a possible design, all or part of the program in the fifth aspect may be stored on a storage medium packaged with the processor, or part or all may be stored on a memory not packaged with the processor.

Description of drawings

FIG. 1 is a schematic structural diagram of a terminal device provided by an embodiment of the present application;

FIG. 2(a) is a schematic diagram of a display interface of a terminal device provided in an embodiment of the present application;

FIG. 2(b) is a schematic diagram of a display interface of a terminal device provided in another embodiment of the present application;

FIG. 3 is a schematic diagram of a terminal image processing method provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of an image processing method of a terminal provided in another embodiment of the present application;

Fig. 5 is the flowchart of the fusion of color and black-and-white images provided by one embodiment of the present application;

Fig. 6 is a schematic structural diagram of a terminal device provided by another embodiment of the present application.

Detailed ways

The terms used in the embodiments of the present application are only used to explain specific embodiments of the present application, and are not intended to limit the present application.

In an image fusion scheme provided by the existing related technology, the original (RAW) images collected by the color camera and the black and white camera are respectively converted into a color YUV image and a black and white YUV image, and then the color YUV image and the black and white YUV image are combined Fusion, this image fusion scheme loses more image details, resulting in poor color reproduction of the final image.

Especially when shooting scenery and/or still life in a forward-lit scene, some details, especially high-frequency details, cannot be fully restored due to the limited capabilities of a single camera device and the ability to enhance details of an algorithm.

Based on the above problems, the embodiment of the present application provides a terminal image processing method, which can realize that the terminal device first performs RAW domain processing on the color RAW image collected by the color camera, and then converts it into a color YUV image, and then fuses it with the black and white YUV image. Thereby retaining more image details and improving the color reproduction of the final image.

The terminal image processing method provided in the embodiment of the present application can be applied to a terminal device, wherein the above-mentioned terminal device can be a smart phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (augmented reality, AR)/virtual reality (virtual reality) reality, VR) equipment, notebook computer, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook or personal digital assistant (personal digital assistant, PDA) and other equipment; the embodiment of this application does not make any restrictions on the specific type of terminal equipment .

Exemplarily, FIG. 1 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 1 , the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus, USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, Earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.

It can be understood that, the structure shown in the embodiment of the present application does not constitute a specific limitation on the terminal device 100 . In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor (ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver (universal asynchronous receiver) /transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or A universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, DCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flashlight, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 communicates with the touch sensor 180K through an I2C bus interface to realize the touch function of the terminal device 100 .

The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.

The PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example: the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to realize the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . The MIPI interface includes a camera serial interface (camera serial interface, CSI), a display serial interface (displayserial interface, DSI), and the like. In some embodiments, the processor 110 communicates with the camera 193 through a CSI interface to realize the shooting function of the terminal device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the terminal device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, specifically, it may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present application is only a schematic illustration, and does not constitute a structural limitation of the terminal device 100 . In other embodiments of the present application, the terminal device 100 may also adopt different interface connection modes in the foregoing embodiments, or a combination of multiple interface connection modes.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through the wireless charging coil of the terminal device 100 . While the charging management module 140 is charging the battery 142 , it can also supply power to the terminal device 100 through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be disposed in the processor 110 . In some other embodiments, the power management module 141 and the charging management module 140 may also be set in the same device.

The wireless communication function of the terminal device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the terminal device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide wireless local area network (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system, etc. applied on the terminal device 100. (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (codedivision multiple access, CDMA), wideband code Wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM , and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou satellite navigation system (beidounavigation satellite system, BDS), a quasi-zenith satellite system (quasi- zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The terminal device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light emitting diode). AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

The terminal device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide-semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the terminal device 100 may include N cameras 193, where N is a positive integer greater than 1. In the embodiment of the present application, the N cameras 193 include a black and white camera and a color camera.

Specifically, during the shooting process, the user turns on the camera, and the light is transmitted to the photosensitive element through the lens (which may correspond to the previously described camera 193). The element undergoes photoelectric conversion, which converts the light signal into an image visible to the naked eye. The photosensitive element then transmits the internal original image (Bayer format, also called Bayer format or Bayer format) to the ISP module, and the ISP module outputs the image in the RGB space domain to the back-end acquisition unit after algorithm processing, and displays it on the terminal device 100. The image preview area or the display screen of the terminal device 100. In this process, the processor controls the lens, photosensitive element and ISP module through the firmware program running on it, and then completes the image preview or shooting function.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the terminal device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 121 may be used to store computer-executable program codes including instructions. The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the terminal device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 110 executes various functional applications and data processing of the terminal device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The terminal device 100 may implement an audio function through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, and an application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .

Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music through the speaker 170A, or listen to hands-free calls.

Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a phone call or voice information, the receiver 170B can be placed close to the human ear to receive the voice.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can put his mouth close to the microphone 170C to make a sound, and input the sound signal to the microphone 170C. The terminal device 100 may be provided with at least one microphone 170C. In some other embodiments, the terminal device 100 may be provided with two microphones 170C, which may also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, identify sound sources, and realize directional recording functions, etc.

The earphone interface 170D is used for connecting wired earphones. The earphone interface 170D may be the USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, pressure sensor 180A may be disposed on display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may be comprised of at least two parallel plates with conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The terminal device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view short messages is executed. When a touch operation whose intensity is greater than or equal to the first pressure threshold acts on the icon of the short message application, the instruction of creating a new short message is executed.

The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100 . In some embodiments, the angular velocity of the terminal device 100 around three axes (ie, x, y and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the terminal device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the terminal device 100 through reverse motion to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The terminal device 100 may use the magnetic sensor 180D to detect the opening and closing of the flip holster. In some embodiments, when the terminal device 100 is a clamshell machine, the terminal device 100 may detect opening and closing of the clamshell according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, features such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The terminal device 100 emits infrared light through the light emitting diode. The terminal device 100 detects infrared reflected light from nearby objects using a photodiode. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100 . When insufficient reflected light is detected, the terminal device 100 may determine that there is no object near the terminal device 100 . The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to make a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.

The ambient light sensor 180L is used for sensing ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in the pocket to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access to the application lock, take pictures with fingerprints, answer incoming calls with fingerprints, and so on.

The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to implement a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds the threshold, the terminal device 100 may reduce the performance of a processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection. In some other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 caused by the low temperature. In some other embodiments, when the temperature is lower than another threshold, the terminal device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

The touch sensor 180K is also called "touch device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the terminal device 100 , which is different from the position of the display screen 194 .

The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the human pulse and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone, combined into a bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the vocal part acquired by the bone conduction sensor 180M, so as to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M, so as to realize the heart rate detection function.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100 .

The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.

The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the terminal device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The terminal device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards may be the same or different. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calling and data communication. In some embodiments, the terminal device 100 adopts an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100 .

For ease of understanding, the Bayer domain and the RAW domain mentioned in the following embodiments are first described here.

1. Bayer field: Each lens on a digital camera has a light sensor to measure the brightness of the light, but to obtain a full-color image, generally three light sensors are required to obtain red, green, and blue images respectively. Three-color information, and in order to reduce the cost and volume of digital cameras, manufacturers usually use CCD or CMOS image sensors. Usually, the original image output by CMOS image sensors is in Bayer domain RGB format, and a single pixel contains only one color value, to get the gray value of the image, it is necessary to interpolate the complete color information of each pixel, and then calculate the gray value of each pixel. In other words, the Bayer field refers to a raw picture format inside a digital camera.

2. RAW domain: RAW domain image, that is, the original image contains data processed from the image sensor of a digital camera, scanner or film scanner. It is so named because RAW domain images have not been processed, printed or used for editing. The RAW domain image contains the most original information of the image without the nonlinear processing in the ISP process.

The following embodiments of the present application will take the terminal device having the structure shown in FIG. 1 as an example, and describe the image processing method of the terminal provided in the embodiments of the present application in detail in combination with the accompanying drawings and application scenarios.

Specifically, after the user clicks the "camera" icon on the display interface of the terminal device, in response to the user's operation, the terminal device runs the camera function. During the preview process, the terminal device collects color RAW images through the color camera, and collects black and white RAW image, and then, according to the color RAW image collected in the preview process, the terminal device determines that the current shooting scene is a non-high dynamic scene, where the non-high dynamic scene refers to the current shooting scene with sufficient light and a small brightness span Scene, such as: architecture or scenery, etc. After determining that the current shooting scene is a non-high dynamic scene, the terminal device can display the current shooting scene as a non-high dynamic scene on the current display interface, as shown in Figure 2(a), which is the A schematic diagram of a display interface of a terminal device provided in an embodiment is shown in FIG. 2( a ) by taking a non-high dynamic scene as a building as an example.

Specifically, the terminal device determines that the current shooting scene is a non-high dynamic scene according to the color RAW image collected during the preview process may be: the terminal device acquires the maximum brightness and minimum brightness in the color RAW image collected during the preview process, and calculates the maximum brightness A ratio to the minimum brightness; if the ratio is smaller than a predetermined ratio threshold, it can be determined that the current shooting scene is a non-high dynamic scene. Wherein, the size of the above-mentioned predetermined ratio threshold may be set according to system performance and/or implementation requirements during specific implementation, and the size of the above-mentioned predetermined ratio threshold is not limited in this embodiment.

In addition, it should be noted that the current shooting scene may be determined by the terminal device in the above manner, or may be selected by the user in the interface of the camera function.

In addition, if during the preview process, the terminal device detects that the black-and-white camera among the N cameras 193 is blocked, it can display a prompt message of "black-and-white camera is blocked" in the current display interface, as shown in FIG. 2(b) , to remind the user that the black and white camera is blocked, so that the user can remove the blocking object of the black and white camera as soon as possible to ensure the quality of the captured image. Fig. 2(b) is a schematic diagram of a display interface of a terminal device provided in another embodiment of the present application.

Further, referring to FIG. 3, FIG. 3 is a schematic diagram of a terminal image processing method provided by an embodiment of the present application. It can be seen from FIG. 3 that after determining that the current shooting scene is a non-high dynamic scene (35), the terminal device acquires A shooting instruction, in response to the above shooting instruction, the terminal device acquires the color RAW image captured during the preview process. Specifically, in response to the above shooting instruction, the terminal device can acquire the color RAW image captured during the preview process from the cache; The image is stored in the cache, and after obtaining the shooting instruction, the color RAW image captured during the preview process is obtained from the cache.

Then, the terminal device performs linearization and dead point correction processing on the color RAW image collected during the preview process through the Bayer domain processing algorithm link 31 . Wherein, the Bayer domain processing algorithm link 31 is the first Bayer domain processing algorithm link, which is used to process the color RAW image.

Furthermore, the terminal device performs RAW domain processing 32 on the color RAW image collected during the preview process to obtain a color YUV image.

For the black-and-white RAW image collected by the terminal device through the black-and-white camera during the preview process, the terminal device responds to the above shooting instructions to obtain the black-and-white RAW image collected during the preview process, and uses the Bayer domain processing algorithm link 33 to pair the black-and-white RAW image collected during the preview process. The image is processed to obtain a black and white YUV image. Wherein, the Bayer domain processing algorithm link 33 is the second Bayer domain processing algorithm link, which is used to process black and white RAW images.

Finally, the terminal device fuses the above-mentioned color YUV image and the above-mentioned black-and-white YUV image (34), obtains the target color image, and displays the above-mentioned target color image (that is, taking pictures in FIG. 3 ).

In addition, referring to Figure 3, in the preview process, after the current shooting scene is determined to be a non-high dynamic scene, the color RAW image collected by the terminal device through the color camera may include a short-time exposure color RAW image and a normal exposure color RAW image In this way, the terminal device can also use the overlapping exposure high dynamic range (stagger high dynamic range, Stagger HDR) technology to fuse the above-mentioned short-time exposure color RAW image and the normal exposure color RAW image (36); then, in the above-mentioned preview During the process, the color image (37) obtained by fusion is displayed.

In specific implementation, the terminal device can process the color RAW image of normal exposure through the image processing front end 0, and process the color RAW image of the short-time exposure through the image processing front end 1, and then use the Stagger HDR to combine the image processing front end 0 and the image processing front end 1 The processed color RAW image is fused (36), and then the fused color image is previewed and displayed (37). When using Stagger HDR to fuse the color RAW images processed by image processing front-end 0 and image processing front-end 1, the difference between the color RAW images processed by image processing front-end 0 and image processing front-end 1 can be determined according to the light intensity in the current shooting scene. Fusion ratio; for example, if the current shooting scene has sufficient light and is a forward-lit scene, then it can be determined that the fusion ratios of the color RAW images processed by image processing front-end 0 and image processing front-end 1 are 95% and 5%, respectively, The effect of the color image obtained by such fusion is similar to that of the image without fusion.

In addition, for the black-and-white RAW images collected by the black-and-white camera during the preview process, one path is sent to the Bayer domain processing algorithm link 33 for processing, and the other path is sent to the image processing front-end 2 for processing, and the black-and-white RAW image processed by the image processing front-end 2 The image is sent to the 3A statistical/semantic understanding module 38, which mainly realizes the functions of automatic color, brightness and/or focusing according to the black and white RAW image, and recognizes the semantic content of the image and uses it for decision-making.

The image processing method of the terminal provided in the embodiment of the present application will be described in detail below with reference to FIG. 4 . FIG. 4 is a schematic diagram of the image processing method of the terminal provided in another embodiment of the present application.

In this embodiment, during the preview process, the terminal device collects a color RAW image through a color camera, and then, according to the color RAW image collected during the preview process, the terminal device determines that the current shooting scene is a non-high dynamic scene, and then, the terminal device passes The images collected by the color camera include normal-exposure color RAW images and short-time-exposure color RAW images, and the normal-exposure color RAW images are cached in the color image cache.

For the black-and-white RAW image collected by the terminal device through the black-and-white camera, the terminal device caches the black-and-white RAW image in the black-and-white image cache, and the black-and-white RAW image is a normally exposed black-and-white RAW image.

Next, the terminal device obtains a shooting instruction, and in response to the above shooting instruction, obtains 8 frames of normally exposed color RAW images from the color image cache.

Then, the terminal device performs linearization and dead point correction processing on the 8 frames of color RAW images collected during the preview process through the Bayer domain processing algorithm link 41 . In this embodiment, the Bayer-domain processing algorithm link 41 in FIG. 4 corresponds to the Bayer-domain processing algorithm link 31 in FIG. 3 , and is used to process the color RAW image.

Next, the terminal device buffers the 8 frames of color RAW images after linearization and dead point correction processing into the image buffer pool 42, and then performs RAW domain processing on the 8 frames of color RAW images in the image buffer pool 42 (43), to obtain Color YUV image, cache color YUV image to image buffer 44. It can be seen from FIG. 4 that the number of bits of the color YUV image buffered in the image buffer 44 is 10.

Specifically, the 8 frames of color RAW images in the image buffer pool 42 are processed in the RAW domain (43), and obtaining the color YUV image may include:

Step 1, preprocessing the above-mentioned 8 frames of color RAW images; wherein, performing pre-processing on the above-mentioned 8 frames of color RAW images may include: aligning the sizes of the above-mentioned 8 frames of color RAW images, The pixels in the frame image are mapped to obtain the corresponding relationship between pixels, the error detection is performed on the above-mentioned corresponding relationship, and the error-prone corresponding relationship is corrected.

Step 2, performing noise reduction processing on the preprocessed 8 frames of color RAW images, so as to fuse the above 8 frames of color RAW images into one frame of color RAW images.

Step 3, convert the color RAW image obtained by fusion into a color RGB image.

Step 4, adjusting the color and brightness of the above-mentioned color RGB image; specifically, adjusting the color and brightness of the above-mentioned color RGB image may include: performing automatic white balance (automatic white balance, AWB) and vignetting (shading) on the above-mentioned color RGB image And color correction (Gamma) and other brightness and color processing flow.

Step 5, converting the color RGB image after adjusting color and brightness into a color YUV image.

In addition, continue to refer to Figure 4, for the black-and-white RAW image collected by the black-and-white camera during the preview process of the terminal device, after the terminal device obtains the shooting instruction, in response to the above-mentioned shooting instruction, the terminal device obtains 1 frame of black-and-white RAW image from the black-and-white image cache , through the Bayer domain processing algorithm link 45, sequentially perform linearization, bad pixel correction, Bayer processing, graph transformation matching (graph transformation matching, GTM) and color correction (Gamma) processing on the above-mentioned 1 frame of black and white RAW image to obtain a black and white YUV image , the obtained black and white YUV image is buffered into the image buffer 46. The number of bits of the black and white YUV image buffered in the image buffer 46 is 10. In this embodiment, the Bayer domain processing algorithm link 45 corresponds to the Bayer domain processing algorithm link 33 in FIG. 3 , and is used for processing black and white RAW images.

Finally, the terminal device fuses (47) the color YUV image in the image buffer 44 and the black and white YUV image in the image buffer 46 to obtain the target color image, and displays the target color image.

Specifically, Fig. 5 is a flow chart of color and black and white image fusion provided by one embodiment of the present application. As shown in Fig. 5, the fusion of the color YUV image and the black and white YUV image to obtain the target color image may include:

Step 501, aligning the sizes of the color YUV image and the black and white YUV image.

Step 502, from the aligned color YUV image and the aligned black-and-white YUV image, obtain the region to be fused in the color YUV image and the region to be fused in the black-and-white YUV image.

Step 503 , mapping the pixels in the region to be fused of the color YUV image with the pixels in the region to be fused of the black and white YUV image to obtain a correspondence between pixels.

Step 504, performing error detection on the above-mentioned corresponding relationship, and correcting the wrong corresponding relationship.

Step 505, merging the clear pixels in the regions to be fused of the color YUV image and the regions to be fused of the black and white YUV images to corresponding blurred pixels. For example, the pixel A in the area where the color YUV image needs to be fused corresponds to the pixel A' in the area where the black and white YUV image needs to be fused; in one case, assume that the pixel A is a clear pixel in the pixel A and the pixel A', If pixel A' is a blurred pixel in the two, then the color of pixel A can be fused to pixel A'; in another case, assuming that pixel A is a blurred pixel in pixel A and pixel A', pixel A' is If there are clear pixels in the two, then the details of pixel A' can be blended to pixel A.

In the terminal image processing method provided in the embodiment of the present application, the terminal device first performs RAW domain processing on the color RAW image, converts the color RAW image into a color YUV image, and then fuses the color YUV image with the black and white YUV image, so that Preserve more image details, especially when shooting scenery and/or still life in front light scenes, it can better restore image details, especially high-frequency details, and improve the color reproduction of the target color image.

In addition, during the preview process, after determining that the current shooting scene is a non-high dynamic scene, the color RAW images collected by the terminal device through the color camera include short-time exposure color RAW images and normal exposure color RAW images, and the above short-time After the exposed color RAW image and the normally exposed color RAW image are fused, the fused image is previewed and displayed, which can reduce the difference between the image obtained by shooting and the image previewed and displayed, and improve user experience.

It can be understood that some or all of the steps or operations in the foregoing embodiments are only examples, and other operations or modifications of various operations may also be performed in the embodiment of the present application. In addition, various steps may be performed in different orders presented in the above embodiments, and it may not be necessary to perform all operations in the above embodiments.

It can be understood that, in order to realize the above functions, the terminal device includes hardware and/or software modules corresponding to each function. Combining the algorithm steps of each example described in the embodiments disclosed in the application, the application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions in combination with the embodiments for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In this embodiment, the functional modules of the terminal device may be divided according to the foregoing method embodiments. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one module. The above integrated modules may be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division methods in actual implementation.

Fig. 6 is a schematic structural diagram of a terminal device provided by another embodiment of the present application. In the case of dividing each functional module corresponding to each function, Fig. 6 shows a possible composition of the terminal device 600 involved in the above embodiment Schematic diagram, as shown in FIG. 6, the terminal device 600 may include: a receiving unit 601, a processing unit 602, and a sending unit 603;

Wherein, the processing unit 602 may be used to support the terminal device 600 to implement the technical solutions described in the embodiments shown in FIG. 2(a) to FIG. 5 of this application.

It should be noted that all relevant content of the steps involved in the above method embodiments can be referred to the function description of the corresponding function module, and will not be repeated here.

The terminal device 600 provided in this embodiment is used to execute the above terminal image processing method, so the same effect as the above method can be achieved.

It should be understood that the terminal device 600 may correspond to the terminal device 100 shown in FIG. 1 . Wherein, the functions of the receiving unit 601 and the sending unit 603 may be implemented by the processor 110, the antenna 1 and the mobile communication module 60 in the terminal device 100 shown in FIG. Implementation; the function of the processing unit 602 may be implemented by the processor 110 in the terminal device 100 shown in FIG. 1 .

In the case of using an integrated unit, the terminal device 600 may include a processing module, a storage module and a communication module.

Wherein, the processing module may be used to control and manage the actions of the terminal device 600 , for example, may be used to support the terminal device 600 to execute the steps performed by the receiving unit 601 , the processing unit 602 and the sending unit 603 . The storage module can be used to support the terminal device 600 to store program codes and data. The communication module can be used to support the communication between the terminal device 600 and other devices.

Wherein, the processing module may be a processor or a controller, which may realize or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (digital signal processing, DSP) and a microprocessor, and the like. The storage module may be a memory. Specifically, the communication module may be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, and/or a Wi-Fi chip.

In an embodiment, when the processing module is a processor and the storage module is a memory, the terminal device 600 involved in this embodiment may be a device having the structure shown in FIG. 1 .

The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when it is run on a computer, the computer executes the implementation shown in Fig. 2(a) to Fig. 5 of the present application. The method provided by the example.

The embodiment of the present application also provides a computer program product, the computer program product includes a computer program, and when it is run on a computer, the computer executes the method provided by the embodiment shown in Fig. 2(a) to Fig. 5 of the present application.

In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three kinds of relationships, for example, A and/or B may indicate that A exists alone, A and B exist simultaneously, or B exists alone. Among them, A and B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single, or Can be multiple.

Those of ordinary skill in the art can appreciate that each unit and algorithm steps described in the embodiments disclosed herein can be realized by a combination of electronic hardware, computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In several embodiments provided in this application, if any function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other various media that can store program codes. .

The foregoing is only a specific implementation of the present application. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should be covered by the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims (20)

1. An image processing method for a terminal, wherein the terminal includes a black and white camera and a color camera, and the method comprises: After the camera function of the terminal runs, during the preview process, the color RAW image is collected by the color camera, and the black and white RAW image is collected by the black and white camera; After obtaining the shooting instruction, obtain the color RAW image collected in the preview process in response to the shooting instruction, and perform RAW domain processing on the color RAW image collected in the preview process to obtain a color YUV image; Process the black-and-white RAW image collected in the preview process through the second Bayer domain processing algorithm link to obtain a black-and-white YUV image; The color YUV image and the black-and-white YUV image are fused to obtain a target color image. 2. The method according to claim 1, wherein the acquisition of the color RAW image captured in the preview process in response to the shooting instruction comprises: In response to the shooting instruction, the color RAW image captured during the preview process is acquired from the cache. 3. The method according to claim 1, characterized in that, before performing RAW domain processing on the color RAW image collected in the preview process, before obtaining the color YUV image, it also includes: The color RAW image collected in the preview process is linearized and dead point corrected through the first Bayer domain processing algorithm link. 4. The method according to claim 1, further comprising: According to the color RAW images collected during the preview process, it is determined that the current shooting scene is a non-high dynamic scene. 5. The method according to claim 1, wherein the color RAW images collected during the preview process comprise at least two frames of color RAW images, and the color RAW images collected during the preview process are processed in the RAW domain to obtain Color YUV images include: Preprocessing the at least two frames of color RAW images; Performing noise reduction processing on at least two frames of color RAW images after preprocessing, so as to fuse at least two frames of color RAW images into one frame of color RAW images; Convert the color RAW image obtained by fusion into a color RGB image; adjust the color and brightness of the color RGB image; Convert a color RGB image with adjusted color and brightness to a color YUV image. 6. The method according to claim 5, wherein said preprocessing the at least two frames of color RAW images comprises: aligning the dimensions of the at least two frames of color RAW images; Mapping pixels in every two frames of images in the at least two frames of color RAW images to obtain correspondence between pixels; An error detection is performed on the corresponding relationship, and an erroneous corresponding relationship is corrected. 7. The method according to claim 1, wherein said merging said color YUV image and said black-and-white YUV image to obtain a target color image comprises: Aligning the size of the color YUV image and the black and white YUV image; From the color YUV image after the size alignment and the black-and-white YUV image after the size alignment, obtain the region that needs to be fused in the color YUV image and the region that needs to be fused in the black-and-white YUV image; Mapping the pixels in the region where the color YUV image needs to be fused with the pixels in the region where the black and white YUV image needs to be fused, to obtain the correspondence between the pixels; performing error detection on the corresponding relationship, and correcting the wrong corresponding relationship; The clear pixels in the area to be fused of the color YUV image and the area to be fused of the black and white YUV image are fused to corresponding blurred pixels. 8. The method according to claim 4, wherein, during the preview process, after determining that the current shooting scene is a non-high dynamic scene, the color RAW image captured by the terminal device through the color camera includes short-time exposure color RAW images and normally exposed color RAW images; The method also includes: Fusing the short-time exposed color RAW image and the normally exposed color RAW image by using overlapping exposure high dynamic technology; During the preview process, the fused color image is displayed. 9. The method according to claim 8, wherein, in the preview process, after the color RAW image is collected by the color camera, further comprising: Cache the normally exposed color RAW image. 10. An image processing device for a terminal, which is used to execute the method according to any one of claims 1 to 9. 11. A terminal device, characterized in that it includes a black-and-white camera and a color camera; one or more processors; memory; multiple application programs; and one or more computer programs, wherein the one or more computer programs are programmed Stored in said memory, said one or more computer programs comprising instructions which, when executed by said terminal device, cause said terminal device to perform the following steps: After the camera function of the terminal runs, during the preview process, the color RAW image is collected by the color camera, and the black and white RAW image is collected by the black and white camera; After obtaining the shooting instruction, obtain the color RAW image collected in the preview process in response to the shooting instruction, and perform RAW domain processing on the color RAW image collected in the preview process to obtain a color YUV image; Process the black-and-white RAW image collected in the preview process through the second Bayer domain processing algorithm link to obtain a black-and-white YUV image; The color YUV image and the black-and-white YUV image are fused to obtain a target color image. 12. The terminal device according to claim 11, characterized in that, when the instruction is executed by the terminal device, the terminal device is made to execute the color RAW captured in the preview process in response to the shooting instruction The image steps include: In response to the shooting instruction, the color RAW image captured during the preview process is acquired from the cache. 13. The terminal device according to claim 11, wherein when the instruction is executed by the terminal device, the terminal device is made to perform RAW domain processing on the color RAW image collected during the preview process to obtain a color Before the step of YUV image, also perform the following steps: The color RAW image collected in the preview process is linearized and dead point corrected through the first Bayer domain processing algorithm link. 14. The terminal device according to claim 11, wherein when the instruction is executed by the terminal device, before the terminal device executes the step of obtaining the shooting instruction, the following steps are further performed: According to the color RAW images collected during the preview process, it is determined that the current shooting scene is a non-high dynamic scene. 15. The terminal device according to claim 11, wherein the color RAW image collected during the preview process includes at least two frames of color RAW images; when the instruction is executed by the terminal device, the terminal The device performs the RAW domain processing on the color RAW image collected in the preview process, and the steps of obtaining the color YUV image include: Preprocessing the at least two frames of color RAW images; Performing noise reduction processing on at least two frames of color RAW images after preprocessing, so as to fuse at least two frames of color RAW images into one frame of color RAW images; Convert the color RAW image obtained by fusion into a color RGB image; adjust the color and brightness of the color RGB image; Convert a color RGB image with adjusted color and brightness to a color YUV image. 16. The terminal device according to claim 15, wherein when the instruction is executed by the terminal device, the terminal device is made to perform the step of preprocessing the at least two frames of color RAW images include: aligning the dimensions of the at least two frames of color RAW images; Mapping pixels in every two frames of images in the at least two frames of color RAW images to obtain correspondence between pixels; An error detection is performed on the corresponding relationship, and an erroneous corresponding relationship is corrected. 17. The terminal device according to claim 11, wherein when the instruction is executed by the terminal device, the terminal device is made to perform the fusing of the color YUV image and the black and white YUV image , the steps to obtain the target color image include: Aligning the size of the color YUV image and the black and white YUV image; From the color YUV image after the size alignment and the black-and-white YUV image after the size alignment, obtain the region that needs to be fused in the color YUV image and the region that needs to be fused in the black-and-white YUV image; Mapping the pixels in the region where the color YUV image needs to be fused with the pixels in the region where the black and white YUV image needs to be fused, to obtain the correspondence between the pixels; performing error detection on the corresponding relationship, and correcting the wrong corresponding relationship; The clear pixels in the area to be fused of the color YUV image and the area to be fused of the black and white YUV image are fused to corresponding blurred pixels. 18. The terminal device according to claim 14, wherein, during the preview process, after determining that the current shooting scene is a non-high dynamic scene, the color RAW image collected by the terminal device through the color camera includes short-time exposure Color RAW images and normally exposed color RAW images; When the instruction is executed by the terminal device, the terminal device is made to further perform the following steps: Fusing the short-time exposed color RAW image and the normally exposed color RAW image by using overlapping exposure high dynamic technology; During the preview process, the fused color image is displayed. 19. The terminal device according to claim 18, characterized in that, when the instruction is executed by the terminal device, the terminal device executes the step of capturing a color RAW image through the color camera during the preview process Afterwards, the following steps are also performed: The normal exposure color RAW image is cached. 20. A computer-readable storage medium, characterized in that, a computer program is stored in the computer-readable storage medium, and when it is run on a computer, the computer executes the program according to any one of claims 1-9. method.

Images