WO2023040540A1 - 图像处理芯片、应用处理芯片、电子设备和图像处理方法 - Google Patents

图像处理芯片、应用处理芯片、电子设备和图像处理方法 Download PDF

Info

Publication number
WO2023040540A1
WO2023040540A1 PCT/CN2022/112534 CN2022112534W WO2023040540A1 WO 2023040540 A1 WO2023040540 A1 WO 2023040540A1 CN 2022112534 W CN2022112534 W CN 2022112534W WO 2023040540 A1 WO2023040540 A1 WO 2023040540A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
processing
tone mapping
processing chip
fused
Prior art date
Application number
PCT/CN2022/112534
Other languages
English (en)
French (fr)
Inventor
曾玉宝
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2023040540A1 publication Critical patent/WO2023040540A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/65—Control of camera operation in relation to power supply
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio

Definitions

  • the application belongs to the technical field of image processing, and in particular relates to an image processing chip, an application processing chip, electronic equipment and an image processing method.
  • Cameras have become an essential device for various digital products, such as mobile phones, tablet computers, etc., all of which are equipped with cameras.
  • the number of cameras has also changed from one to multiple, and it is necessary to transmit multiple channels of raw RAW data acquired by the image sensor of the camera to the application processing chip for processing.
  • an object of the present application is to propose an image processing chip.
  • the second purpose of the present application is to propose an application processing chip.
  • the third object of the present application is to provide an electronic device.
  • the fourth purpose of the present application is to propose an image processing method.
  • the embodiment of the first aspect of the present application proposes an image processing chip, the image processing chip includes: a first image signal processor, which is used to perform fusion processing on M channels of original image data to obtain N channels Fusion image data, wherein M and N are both positive integers, and M>N; the image processing chip is also used to send the fusion image data to an application processing chip.
  • the embodiment of the second aspect of the present application proposes an application processing chip, the application processing chip is used to obtain N channels of fused image data from the image processing chip, and the application processing chip includes: a second image signal A processor, configured to perform calibration processing on the N channels of fused image data; wherein, the N channels of fused image data are obtained by performing fusion processing on M channels of original image data, wherein M and N are both positive integers, and M> N.
  • the embodiment of the third aspect of the present application proposes an electronic device, the electronic device includes an image processing chip for performing fusion processing on M channels of original image data to obtain N channels of fused image data, wherein, Both M and N are positive integers, and M>N; the application processing chip is used to obtain N channels of fused image data from the image processing chip, and perform calibration processing on the N channels of fused image data.
  • the embodiment of the fourth aspect of the present application proposes an image processing method, the method includes: acquiring M channels of original image data; performing fusion processing on the M channels of original image data to obtain N channels of fused images data; performing calibration processing on the N-way fused image data.
  • Fig. 1 is a schematic structural diagram of image data processing in one embodiment of the present application
  • Fig. 2 is a schematic structural diagram of an image processing chip according to an embodiment of the present application.
  • Fig. 3 is a schematic structural diagram of an image processing chip according to a specific embodiment of the present application.
  • Fig. 4 is a schematic diagram of image size comparison before and after fusion processing according to a specific embodiment of the present application.
  • Fig. 5 is a schematic diagram of image size comparison before and after tone mapping processing according to a specific embodiment of the present application
  • Fig. 6 is a schematic structural diagram of an application processing chip according to an embodiment of the present application
  • FIG. 7 is a schematic structural diagram of an application processing chip according to a specific embodiment of the present application.
  • FIG. 8 is a schematic flow chart of calibration processing in a specific embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of an electronic device according to another embodiment of the present application.
  • FIG. 11 is a schematic flowchart of an image processing method according to an embodiment of the present application.
  • the image processing chip of the present application includes a first image signal processor, which is used to perform fusion processing on M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers, and M>N;
  • the processing chip is also used to send the fused image data to the application processing chip.
  • the first image signal processor is specifically configured to: divide M channels of original image data into N groups, where each group includes m channels of original image data, m is an integer, and 2 ⁇ m ⁇ M;
  • the pixel value of the i-th original image, ki represents the ratio of the longest exposure time among the exposure times of the m-th original image to the exposure time of the i-th original image, i is an integer, and 1 ⁇ i ⁇ m.
  • the first image signal processor is further configured to: perform tone mapping processing on each channel of fused image data to obtain fused image data after tone mapping processing and tone mapping processing parameters; wherein, the image processing chip is also It is used to send the fused image data processed by the N channels of tone mapping and the corresponding tone mapping processing parameters to the application processing chip.
  • the first image signal processor when the first image signal processor performs tone mapping processing on the fused image data, it is specifically used to: determine the region of interest of the fused image data; perform histogram equalization processing based on the region of interest to obtain the histogram An image equalization mapping relationship, wherein the histogram equalization mapping relationship is a tone mapping processing parameter; the histogram equalization mapping relationship is mapped to a full image of the fused image data.
  • the first image signal processor is further configured to: obtain 3A statistical information of M channels of raw image data, wherein the 3A statistical information includes automatic exposure statistical information, automatic white balance statistical information, and automatic focus statistical information ; Wherein, the image processing chip is also used to send 3A statistical information to the application processing chip.
  • the image processing chip is further configured to: encode the 3A statistical information, the fused image data after tone mapping processing, and the tone mapping processing parameters to obtain encoding information, and send the encoding information to the application processing chip.
  • the application processing chip of the present application is used to obtain N-way fused image data from the image processing chip, and the application processing chip includes: a second image signal processor, which is used to perform calibration processing on N-way fused image data; It is obtained by fusing M channels of original image data, where M and N are both positive integers, and M>N.
  • the application processing chip further includes: a second central processing unit, configured to use the 3A algorithm to obtain the N-way fused
  • the AWB calibration parameters and CCM parameters of the image data, and the AWB gain parameters are calibrated according to the tone mapping processing parameters; wherein, the second image signal processor is specifically used to use the calibrated AWB gain parameters and CCM parameters for M-channel fusion image data Perform automatic white balance calibration and color calibration.
  • the electronic device of the present application includes an image processing chip and an application processing chip.
  • the image processing chip is used to fuse M channels of original image data to obtain N channels of fused image data, where M and N are both positive integers, and M>N;
  • the application processing chip is used to obtain N channels from the image processing chip.
  • the fused image data of the channels is fused, and calibration processing is performed on the fused image data of the N channels.
  • the image processing chip is specifically used to: divide M channels of original image data into N groups, wherein each group includes m channels of original image data, m is an integer, and 2 ⁇ m ⁇ M; according to the following formula
  • the pixel value of the original image, ki represents the ratio of the longest exposure time among the exposure times of m original images to the exposure time of the i-th original image, i is an integer, and 1 ⁇ i ⁇ m.
  • the image processing chip is further configured to: perform tone mapping processing on each channel of fused image data to obtain the fused image data and tone mapping processing parameters after tone mapping processing, and convert the N channels of tone mapping processing The fused image data and its corresponding tone mapping processing parameters are sent to the application processing chip.
  • the first image signal processor when the first image signal processor performs tone mapping processing on the fused image data, it is specifically used to: determine the region of interest of the fused image data; perform histogram equalization processing based on the region of interest to obtain the histogram An image equalization mapping relationship, wherein the histogram equalization mapping relationship is a tone mapping processing parameter; the histogram equalization mapping relationship is mapped to a full image of the fused image data.
  • the first image signal processor is further configured to: obtain 3A statistical information of M channels of raw image data, wherein the 3A statistical information includes automatic exposure statistical information, automatic white balance statistical information, and automatic focus statistical information ; Wherein, the image processing chip is also used to send 3A statistical information to the application processing chip.
  • the image processing chip is also used to: encode the 3A statistical information, the fused image data after tone mapping processing, and the tone mapping processing parameters to obtain encoding information, and send the encoding information to the application processing chip.
  • the application processing chip further includes: a second central processing unit, configured to use the 3A algorithm to obtain the N-way fused
  • the AWB calibration parameters and CCM parameters of the image data, and the AWB gain parameters are calibrated according to the tone mapping processing parameters; wherein, the second image signal processor is specifically used to use the calibrated AWB gain parameters and CCM parameters for M-channel fusion image data Perform automatic white balance calibration and color calibration.
  • the image processing method of the present application includes acquiring M channels of original image data; performing fusion processing on the M channels of original image data to obtain N channels of fused image data; and performing calibration processing on the N channels of fused image data.
  • the image processing method further includes: performing tone mapping processing on each channel of fused image data to obtain fused image data after tone mapping processing and tone mapping processing parameters.
  • FIG. 1 when an electronic device that can collect multiple channels of raw image data performs image collection, it is necessary to continuously transmit multiple channels of original RAW data acquired by the image sensor of the camera to the An image processing chip and an application processing chip perform processing. If multiple channels of original image data are transmitted to the application processing chip for processing, the amount of transmitted data will be large, the bandwidth requirements are high, and the power consumption is also high. And, see Figure 1, if MIPI (Mobile Industry Processor Interface, mobile industry processor interface) is used for data transmission, limited by hardware and cost, it is difficult to realize data transmission of too many channels.
  • MIPI Mobile Industry Processor Interface, mobile industry processor interface
  • 3A statistics include automatic exposure statistics, automatic white balance statistics and auto focus statistics, the data transmission volume is relatively large, the requirements for transmission bandwidth are high, and the power consumption of transmission data is high.
  • an electronic device when it captures an image in DOL (Digital overlap) mode, it needs to sequentially transmit the multiple exposure images output by the image sensor of the camera to the image processing chip and the application processing chip, and each exposure image 3A statistics and PD.
  • 3 channels*2 cameras*3A need to be counted, and at least 18 categories of statistical data need to be used and transmitted, plus (3 channels of Raw images + 3 channels of PD)*2 (cameras) data, a total of 30 channels of data, limited by hardware and cost, the number of data channels of MIPI (Mobile Industry Processor Interface, mobile industry processor interface) hardware cannot meet the demand.
  • PD Phase Data (phase information), and PD is used for focusing.
  • this application proposes an image processing chip, an application processing chip, electronic equipment and an image processing method, aiming at solving the problem of large amount of data and the small number of data paths of MIPI hardware, which cannot meet the needs of data transmission.
  • the image processing chip, application processing chip, electronic device, and image processing method of the embodiments of the present application will be described in detail below with reference to FIGS. 2-11 of the specification and specific implementation manners.
  • FIG. 2 is a schematic structural diagram of an image processing chip according to an embodiment of the present application.
  • the image processing chip 2 includes a first image signal processor 21 .
  • the first image signal processor 21 is used to perform fusion processing on M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers, and M>N; the image processing chip 2 also uses to send the fused image data to the application processing chip 3 .
  • the requirement for transmission bandwidth when the image processing chip 2 returns data to the application processing chip 3 can be reduced, and the power consumption when returning data can be reduced.
  • the image sensor can use photosensitive elements such as CMOS (Complementary Metal Oxide Semiconductor, Complementary Metal Oxide Semiconductor), CCD (Charge-coupled Device, photosensitive element or charge-coupled device).
  • CMOS Complementary Metal Oxide Semiconductor, Complementary Metal Oxide Semiconductor
  • CCD Charge-coupled Device, photosensitive element or charge-coupled device
  • the original image data is the unprocessed image data collected by the image sensor, which is CMOS (Complementary Metal Oxide Semiconductor, Complementary Metal Oxide Semiconductor), CCD (Charge-coupled Device, photosensitive element or charge-coupled Component) and other photosensitive elements convert the captured light source signal into the original data of the digital signal.
  • Raw image data records the original information of the image sensor, and also records some metadata generated by the camera, such as ISO settings, shutter speed, aperture value, white balance, etc.
  • the image sensor can work in the digital overlay DOL mode, the original image data obtained by each image sensor includes multiple exposure images. For example, when acquiring raw image data in 3DOL mode, the acquired raw image data will include 3 exposure images, such as a long exposure image, a middle exposure image and a short exposure image.
  • the number of image sensors may be one or more (greater than or equal to two), for acquiring M channels of raw image data.
  • the image sensor acquires raw image data in the DOL mode
  • the multi-channel raw image data acquired by each image sensor is multi-channel exposure image data.
  • the image processing chip 2 can be used in an electronic device with a camera.
  • ZSL Zero Shutter Lang, zero-delay camera
  • the M The original image data of one path is input to the image processing chip 2, and after the original image data of M paths are fused and processed into N (N ⁇ M) paths of fused image data by the first image signal processor 21, the image processing chip 2 converts the fused image data of N paths transmitted to the application processing chip 3.
  • the requirements for transmission bandwidth when the image processing chip 2 returns data to the application processing chip 3 can be reduced, and the power consumption when returning data is reduced, which is conducive to the landing of zero-delay camera technology on low-end platforms.
  • the first image signal processor 21 is specifically configured to divide M channels of original image data into N groups, where each group includes m channels of original image data, m is an integer, and 2 ⁇ m ⁇ M; According to the following formula, the original image data of m paths in each group are fused:
  • Pixel_Value_j_Fusioned (Pixel_Value_i*ki)(1)
  • Pixel_Value_j_Fusioned represents the pixel value of the j-th fusion image among the N-path fusion images
  • Pixel_Value_i represents the pixel value of the i-th path original image data among the m-path original image data
  • ki represents the longest exposure time of the m-path original image data
  • the ratio of the exposure time to the exposure time of the i-th raw image data, i is an integer, and 1 ⁇ i ⁇ m.
  • the first image signal processor 21 can include a first ISP (Image Signal Processing, image signal processing) module and a fusion module, wherein the number of the first ISP module and the fusion module can be One, or N. If there are N, the first ISP module and the fusion module are in one-to-one correspondence with the m channels of original image data in each group. At this time, the m channels of original image data are sequentially input to the corresponding first ISP module , a fusion module for processing; if there is one, the first ISP module and the fusion module can process N sets of original image data in parallel. Thereby, image processing efficiency can be ensured.
  • the image processing chip 2 may also include a neural network processor, denoted as an NPU (Neural-network Processing Unit, neural network processing unit) module.
  • NPU Neuro-network Processing Unit
  • the N first ISP modules are used to receive M channels of original image data, and preprocess the received original image data to obtain preview images of the channels.
  • the first ISP module processes the raw image data transmitted by the image sensor to match different types of image sensors.
  • the first ISP module completes the effect processing of the original image data through a series of digital image processing algorithms, mainly including 3A (auto white balance, auto focus, auto exposure), dead point correction, denoising, strong light suppression, backlight Compensation, color enhancement, lens shading correction and other processing to get the preview image.
  • 3A auto white balance, auto focus, auto exposure
  • dead point correction denoising, strong light suppression, backlight Compensation
  • color enhancement color enhancement
  • lens shading correction other processing to get the preview image.
  • the NPU module is used to process each preview image separately by using the AI algorithm.
  • the NPU module uses the AI algorithm to complete the Demosaic (anti-mosaic) difference algorithm, automatic white balance, color correction, noise reduction, HDR (High-Dynamic Range, high-dynamic range image), super-resolution, etc. for each preview image. deal with.
  • Demosaic anti-mosaic
  • HDR High-Dynamic Range, high-dynamic range image
  • the fusion module is used to perform fusion processing on the corresponding preview images processed by the AI algorithm to obtain N-way fusion images.
  • the fusion module performs fusion processing on the images processed by the first ISP module and the NPU module, and converts M channels of original image data into N channels of fusion images, which can reduce data transmission bandwidth and save power consumption.
  • the raw image data acquired by each image sensor includes 3-way exposure images (long exposure image, intermediate exposure image and short exposure image), so When fusing long-exposure images, intermediate-exposure images, and short-exposure images, the original images can be fused according to the following formula:
  • Pixel_Value_Fusioned Pixel_Value_Long+Pixel_Value_Middle*4+Pixel_Value_Short*16,
  • Pixel_Value_Fusioned indicates the pixel value of the fused image
  • Pixel_Value_long indicates the pixel value of the long exposure image
  • Pixel_Value_in indicates the pixel value of the intermediate exposure image
  • Pixel_Value_short indicates the pixel value of the short exposure image.
  • the exposure time t of the long-exposure image is long
  • the exposure time t of the intermediate exposure image is short
  • the fusion module when the fusion module processes the exposure images in the preview image, it rearranges the exposure images. As an example, as shown in FIG. 4 , the fusion module can fuse three 10-bit exposure images into one 30-bit fusion image.
  • the first image signal processor 21 is also used to perform tone mapping processing on each channel of fused image data, so as to obtain the fused image data and tone mapping processing parameters after tone mapping processing; wherein, the image processing chip 2 is also used to send the fused image data processed by the N channels of tone mapping and the corresponding tone mapping processing parameters to the application processing chip 3 .
  • the first image signal processor 21 may include a tone mapping module.
  • the tone mapping module can be in one-to-one correspondence with the first ISP module and the fusion module, that is, the number of tone mapping modules is the same as the number of the first ISP module and the number of fusion modules, and the number of the first ISP module and the number of fusion modules are both N , the quantity of the tone mapping module is also N, and when the quantity of the first ISP module and the quantity of the fusion module are both 1, the quantity of the tone mapping module is also 1, so as to realize the fused image processed by the first ISP module and the fusion module, It can be transmitted to the corresponding tone mapping module for processing to ensure the reliability of data processing.
  • the tone mapping module is used to perform tone mapping processing on the fused image to obtain the fused image after the tone mapping processing and tone mapping processing parameters.
  • the tone mapping module may use a tone mapping algorithm (tone mapping) to perform tone mapping processing on the high-bit-width fusion image obtained through fusion processing.
  • tone mapping algorithm tone mapping
  • FIG. 5 the fused image of 30 bits obtained after fusion processing can be processed by tone mapping to obtain an image of 10 bits.
  • the first image signal processor 21 when the first image signal processor 21 performs tone mapping processing on the fused image data, it is specifically used to: determine the region of interest of the fused image data; perform histogram equalization processing based on the region of interest, A histogram equalization mapping relationship is obtained, wherein the histogram equalization mapping relationship is a tone mapping processing parameter; and the histogram equalization mapping relationship is mapped to a full image of the fused image data.
  • the region of interest of the fused image is determined to enhance a certain part of the image in a targeted manner, and the method of delineating the region of interest may be a user input method.
  • the number of defined images of interest can be one or more.
  • the shape of the acquired image of interest may be a polygon, an ellipse, or the like.
  • Histogram equalization is to stretch the image non-linearly and redistribute the pixel values of the image so that the number of pixels in a certain gray scale range is roughly the same, so that the given histogram distribution is transformed into a uniform histogram graph distribution for maximum contrast.
  • the histogram equalization mapping relationship is recorded. Based on the histogram equalization mapping relationship, it is mapped to the full image of the fused image to perform histogram equalization processing on the full image of the fused image to ensure the highest information fidelity in the ROI region.
  • the extended area can be further obtained, and the size of the extended area can be (width and height of the ROI area*1.25), such as the ROI area is a rectangular area, the extended area is specifically a rectangular area, and the extended area
  • the length of is the length of the ROI area*1.5
  • the width of the extension area is the width of the ROI area*1.5
  • the centers of the two areas coincide.
  • the histogram equalization process is performed based on the extended area, and the histogram equalization mapping relationship is obtained.
  • histogram equalization is very useful for images where both the background and the foreground are too bright or too dark, and it can better reveal the details in overexposed or underexposed photos.
  • a major advantage of this approach is that it is quite intuitive and reversible, and if the equalization function is known, the original histogram can be restored with little computation.
  • the first image signal processor 21 is also used for statistically obtaining 3A statistical information of M channels of raw image data, wherein the 3A statistical information includes automatic exposure statistical information, automatic white balance statistical information, and automatic focus statistical information. information; wherein, the image processing chip 2 is also used to send 3A statistical information to the application processing chip 3 .
  • the first image signal processor 21 may use the first ISP module to obtain 3A statistical information of M channels of original image data.
  • 3A statistics include automatic exposure statistics (AE, Auto Exposure), automatic white balance statistics (AWB, Auto White Balance) and auto focus statistics (AF, Auto Focus).
  • the image processing chip 2 is also used to encode 3A statistical information, fused image data after tone mapping processing, tone mapping processing parameters, and PD data to obtain encoding information, and send the encoding information to Application processing chip 3.
  • the image processing chip 2 may include a MIPI-TX encoding submodule, wherein the MIPI-TX encoding submodule may correspond to the above-mentioned tone mapping module, that is, the MIPI-TX encoding submodule
  • the number is the same as the number of tone mapping modules, which can be one or N.
  • the MIPI-TX encoding sub-module receives the 3A statistical information of the original image data, the fused image after the tone mapping process, the tone mapping processing parameters and PD data to encode the 3A statistical information of the original image data, the fused image after the tone mapping process , tone mapping processing parameters and PD data are encoded, and the encoded information is transmitted to the application processing chip 3 through the MIPI protocol.
  • the image processing chip provided by this application performs fusion processing on M channels of original image data to obtain N channels of fused image data, and performs tone mapping processing on N channels of fused image data, which greatly reduces the amount of data transmission and the data transmission process. It also has the effect of reducing power consumption, which is helpful for the application of zero-delay camera technology in low-end platforms.
  • the present application provides an application processing chip.
  • FIG. 6 is a schematic structural diagram of an application processing chip according to an embodiment of the present application.
  • the application processing chip 3 is used to obtain N channels of fused image data from the image processing chip 2 .
  • the application processing chip 3 includes a second image signal processor 31 .
  • the second image signal processor 31 is used to perform calibration processing on the N-way fused image data; wherein, the N-way fused image is obtained by performing fusion processing on the M-way original image data, wherein M and N are both positive integers, and M>N.
  • the image processing chip 2 After the original image data is fused or fused and tone-mapped by the image processing chip 2, the amount of data is greatly reduced. However, after the image processing chip 2 performs tone mapping processing on the fused image, the 3A accuracy of the image is affected, so calibration processing needs to be performed on the fused image after the tone mapping processing. As an example, the fused image after tone mapping processing, as well as 3A statistical information, tone mapping processing parameters and PD data may be obtained, so as to perform calibration processing on the fused image data to obtain a target image.
  • the application processing chip 3 may include a MIPI-RX decoding submodule
  • the second image signal processor 31 may include a second ISP module.
  • the number of the MIPI-RX decoding sub-module and the second ISP module can be one or N, which can be specifically the same as the number of the MIPI-TX encoding sub-module in the image processing chip 2 .
  • the MIPI-RX decoding submodule is used to receive the encoding information corresponding to the MIPI-TX encoding submodule, and decode the encoding information to obtain 3A statistical information, the fused image after tone mapping processing, and the tone mapping processing parameters and PD data, and then transmit the fused image processed by the tone mapping to the second ISP module.
  • the second ISP module is configured to, after receiving the corresponding fused image processed by tone mapping, use a digital image processing algorithm to preprocess the fused image processed by tone mapping.
  • the preprocessing performed by the second ISP module on the fused image after the tone mapping process is the same as the preprocessing performed by the first ISP module, and will not be repeated here.
  • the application processing chip 3 also includes a second central processing unit 32, and the number of the second central processing unit 32 can be one or N, and can be specifically compared with MIPI -
  • the numbers of the RX decoding sub-module and the second ISP module are the same.
  • the second central processing unit 32 is used to obtain the AWB calibration parameters and CCM parameters of the N-way fused image data according to the 3A statistical information of the M-way original image data and the tone mapping processing parameters of the N-way fused image data by using the 3A algorithm, and according to the tone
  • the mapping processing parameters calibrate the AWB gain parameters; wherein, the second image signal processor 31 is specifically configured to use the calibrated AWB gain parameters and CCM parameters to perform automatic white balance calibration and color calibration on the M-channel fused image data.
  • the second central processing unit 32 is configured to use the 3A algorithm to obtain the AWB calibration parameters and CCM according to the 3A statistical information, tone mapping processing parameters and PD data after receiving the corresponding 3A statistical information, tone mapping processing parameters and PD data (Color Correct Matrix, color correction) parameters, and calibrate the AWB gain parameters according to the tone mapping processing parameters.
  • the second central processing unit 32 can use the 3A statistical information before image fusion compression and the 3A statistical information after image fusion compression to compare the color of the RAW image received by the application processing chip 3.
  • the ratio coefficient is obtained by combining RGB statistics before and after compression, using this ratio to correct the result of the AWB algorithm (RGB Gain) on the application processing chip side, and using the corrected 3A algorithm result to color the Raw image of the application processing chip 3 calibration.
  • the second central processing unit 32 calibrates the AWB gain parameter according to the tone mapping processing parameters, it can be specifically used for:
  • RGain calibration RGain/Cr/Cg
  • BGain calibration BGain/Cb/Cg
  • RGain calibration is the R gain after calibration
  • BGain calibration is the B gain after calibration
  • RGain is the R gain before calibration
  • Cr/Cg is the relative G gain of R
  • Cb/Cg is the relative G gain of B
  • Cr Rsum/Rsum_untonemapping
  • Cg Gsum/Gsum_untonemapping
  • Cb Bsum/Bsum_untonemapping
  • Rsum, Gsum, and Bsum are the total values of the R, G, and B components of the fused image after the tone mapping process, respectively
  • Rsum_untonemapping, Gsum_untonemapping, and Bsum_untonemapping are the reverse tone The total value of the R, G, and B components of the fused image after mapping.
  • the application processing chip of the embodiment of the present application can ensure the image display effect by performing calibration processing on N channels of fused image data obtained by fusing M channels of original image data.
  • the application also provides an electronic device.
  • the electronic device 10 includes an image processing chip 2 and an application processing chip 3 .
  • the image processing chip 2 is used to perform fusion processing on M channels of original image data to obtain N channels of fused image data, wherein M and N are both positive integers, and M>N.
  • the image processing chip 2 is specifically used to: divide M channels of original image data into N groups, wherein each group includes m channels of original image data, m is an integer, and 2 ⁇ m ⁇ M; each group is divided according to the following formula
  • the m-way original image data in the fusion process :
  • Pixel_Value_j_Fusioned (Pixel_Value_i*ki),
  • Pixel_Value_j_Fusioned represents the pixel value of the j-th fusion image among the N-path fusion images
  • Pixel_Value_i represents the pixel value of the i-th path original image data among the m-path original image data
  • ki represents the longest exposure time of the m-path original image data
  • the ratio of the exposure time to the exposure time of the i-th raw image data, i is an integer, and 1 ⁇ i ⁇ m.
  • the image processing chip 2 is also used to: perform tone mapping processing on each channel of fused image data to obtain the fused image data and tone mapping processing parameters after the tone mapping process, and perform tone mapping processing on the N channels of tone mapping
  • the processed fused image data and its corresponding tone mapping processing parameters are sent to the application processing chip 3 .
  • the application processing chip 3 is used to obtain N channels of fused image data from the image processing chip, and perform calibration processing on the N channels of fused image data.
  • the electronic device in this embodiment of the present application may be a mobile terminal, such as a smart phone, a tablet computer, and the like.
  • the image processing chip 2 may also include a CPU, a memory, and a computer vision engine, wherein the CPU may be responsible for the control of the image processing chip 2, such as power on and off, firmware loading, runtime control, etc.; the memory may be used to store images Data that needs to be stored during data processing; a computer vision engine can be configured to process a scene, generate an information stream that represents the observed activity, and transmit the information stream to other modules through the system bus to learn the object behavior of the corresponding scene.
  • the application processor chip 3 may also include a memory for storing data to be stored during image data processing.
  • the electronic device provided by the embodiment of the present application performs fusion or fusion and tone mapping processing on the original image transmitted by the image sensor through the image processing chip, and sends the compressed fusion image to the application processing chip, which greatly reduces the amount of data transmission. It reduces the bandwidth requirement during the data transmission process, and also has the effect of reducing power consumption.
  • the electronic device provided by the embodiment of the present application can be applied to a multi-camera (such as two cameras, respectively, the primary and secondary cameras) scene, the primary and secondary cameras use this method synchronously to reduce the bandwidth, and adjust the parameters of the tone mapping during the fusion of the primary and secondary cameras. Synchronization and synthesis for more accurate tone mapping.
  • the application also provides an image processing method.
  • FIG. 11 is a schematic flowchart of an image processing method according to an embodiment of the present application. As shown in Figure 11, image processing methods include:
  • an image sensor may be used to acquire M channels of original image data, where the original image is acquired in a digital overlay DOL mode.
  • the image sensor is a photosensitive element, which uses the photoelectric conversion function of the photoelectric device to convert the light image on the photosensitive surface into an electrical signal proportional to the light image.
  • the image sensor can use photosensitive elements such as CMOS and CCD.
  • CMOS image sensor is essentially a chip, which mainly includes modules such as photosensitive area array (Bayer array), timing control, analog signal processing, and analog-to-digital conversion.
  • the main function is to convert the optical signal into an electrical signal, and then convert it into a digital signal through an ADC (Analog-to-digital converter, analog-to-digital converter).
  • performing fusion processing on M channels of original image data may include:
  • Pixel_Value_j_Fusioned (Pixel_Value_i*ki),
  • Pixel_Value_j_Fusioned represents the pixel value of the j-th fusion image among the N-path fusion images
  • Pixel_Value_i represents the pixel value of the i-th path original image data among the m-path original image data
  • ki represents the longest exposure time of the m-path original image data
  • the ratio of the exposure time to the exposure time of the i-th raw image data, i is an integer, and 1 ⁇ i ⁇ m.
  • the image processing method further includes: performing tone mapping processing on each channel of fused image data to obtain fused image data after tone mapping processing and tone mapping processing parameters.
  • the image processing method provided by the embodiment of the present application performs fusion or fusion and tone mapping processing on the M channels of original images, and corrects the fusion image after the tone mapping processing, which greatly reduces the amount of data transmission, and reduces Bandwidth requirements, and also has the effect of reducing power consumption.
  • the image processing method provided by the embodiment of the present application can be applied to a multi-camera (such as two cameras, respectively, the main and the sub-camera) scene, and the main and sub-cameras use this method synchronously to reduce the bandwidth, and the tone mapping when the main and sub-camera pairs are fused Parameters are synchronized and synthesized for more accurate tone mapping.
  • a "computer-readable medium” may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device.
  • computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM).
  • the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary.
  • the program is processed electronically and stored in computer memory.
  • each part of the present application may be realized by hardware, software, firmware or a combination thereof.
  • various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.
  • first and second are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features.
  • the features defined as “first” and “second” may explicitly or implicitly include at least one of these features.
  • “plurality” means at least two, such as two, three, etc., unless otherwise specifically defined.
  • a first feature being "on” or “under” a second feature may mean that the first and second features are in direct contact, or that the first and second features are indirect through an intermediary. touch.
  • “above”, “above” and “above” the first feature on the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature.
  • “Below”, “beneath” and “beneath” the first feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is less horizontally than the second feature.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

一种图像处理芯片(2)、应用处理芯片(3)、电子设备(10)和图像处理方法。图像处理芯片(2)包括第一图像信号处理器(21)，用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N；所述图像处理芯片(2)还用于将所述融合图像数据发送给应用处理芯片(3)。

Description

图像处理芯片、应用处理芯片、电子设备和图像处理方法
优先权信息
本申请请求2021年9月15日向中国国家知识产权局提交的、专利申请号为2021110811258的专利申请的优先权和权益，并且通过参照将其全文并入此处。
技术领域
本申请属于图像处理技术领域，尤其涉及一种图像处理芯片、应用处理芯片、电子设备和图像处理方法。
背景技术
摄像头已成为各种数码产品必备的设备，例如：手机、平板电脑等，均设置了摄像头。为保证图像采集效果，摄像头的数量也从一个变成多个，且需将摄像头的图像传感器获取的多路原始RAW数据传输给应用处理芯片进行处理。
发明内容
本申请旨在至少在一定程度上解决相关技术中的技术问题之一。为此，本申请的一个目的在于提出一种图像处理芯片。
本申请的第二个目的在于提出一种应用处理芯片。
本申请的第三个目的在于提出一种电子设备。
本申请的第四个目的在于提出一种图像处理方法。
为达到上述目的，本申请第一方面实施例提出了一种图像处理芯片，所述图像处理芯片包括：第一图像信号处理器，用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N；所述图像处理芯片还用于将所述融合图像数据发送给应用处理芯片。
为达到上述目的，本申请第二方面实施例提出了一种应用处理芯片，所述应用处理芯片用于从图像处理芯片处获得N路融合图像数据，所述应用处理芯片包括：第二图像信号处理器，用于对所述N路融合图像数据进行校准处理；其中，所述N路融合图像是对M路原始图像数据进行融合处理而得到，其中M和N均为正整数，且M>N。
为达到上述目的，本申请第三方面实施例提出了一种电子设备，所述电子设备包括图像处理芯片，用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N；应用处理芯片，用于从所述图像处理芯片处获得N路融合图像数据，并对所述N路融合图像数据进行校准处理。
为达到上述目的，本申请第四方面实施例提出了一种图像处理方法，所述方法包括：获取M路原始图像数据；对所述M路原始图像数据进行融合处理，以得到N路融合图像数据；对所述N路融合图像数据进行校准处理。
本申请附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。
附图说明
图1是本申请一个实施例的图像数据处理的结构示意图；
图2是本申请实施例的图像处理芯片的结构示意图；
图3是本申请一个具体实施例的图像处理芯片的结构示意图；
图4是本申请一个具体实施例的融合处理前后图像大小对比示意图；
图5是本申请一个具体实施例的色调映射处理前后图像大小对比示意图；
图6是本申请一个实施例的应用处理芯片的结构示意图
图7是本申请一个具体实施例的应用处理芯片的结构示意图；
图8是本申请一个具体实施例的校准处理的流程示意图；
图9是本申请一个实施例的电子设备的结构示意图；
图10是本申请另一个实施例的电子设备的结构示意图；
图11是本申请实施例的图像处理方法的流程示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。
本申请的图像处理芯片包括第一图像信号处理器，用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N；图像处理芯片还用于将融合图像数据发送给应用处理芯片。
在某些实施方式中，第一图像信号处理器具体用于：将M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数，且2≤m≤M；根据如下公式对各组中的m路原始图像数据进行融合处理：Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)，其中，Pixel_Value_j_Fusioned表示N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像中第i路原始图像的像素值，ki表示m路原始图像的曝光时间中的最长曝光时间与第i路原始图像的曝光时间的比值，i为整数，且1＜i≤m。
在某些实施方式中，第一图像信号处理器还用于：对各路融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数；其中，图像处理芯片还用于将N路色调映射处理后的融合图像数据及其对应的色调映射处理参数发送给应用处理芯片。
在某些实施方式中，第一图像信号处理器在对融合图像数据进行色调映射处理时，具体用于：确定融合图像数据的感兴趣区域；基于感兴趣区域进行直方图均衡化处理，得到直方图均衡化映射关系，其中，直方图均衡化映射关系为色调映射处理参数；将直方图均衡化映射关系映射到融合图像数据的全图。
在某些实施方式中，第一图像信号处理器还用于：统计得到M路原始图像数据的3A统计信息，其中，3A统计信息包括自动曝光统计信息、自动白平衡统计信息和自动对焦统计信息；其中，图像处理芯片还用于将3A统计信息发送给应用处理芯片。
在某些实施方式中，图像处理芯片还用于：对3A统计信息、色调映射处理后的融合图像数据和色调映射处理参数进行编码，以得到编码信息，并将编码信息发送给应用处理 芯片。
本申请的应用处理芯片用于从图像处理芯片处获得N路融合图像数据，应用处理芯片包括：第二图像信号处理器，用于对N路融合图像数据进行校准处理；其中，N路融合图像是对M路原始图像数据进行融合处理而得到，其中M和N均为正整数，且M>N。
在某些实施方式中，应用处理芯片还包括：第二中央处理器，用于利用3A算法根据M路原始图像数据的3A统计信息、N路融合图像数据的色调映射处理参数，得到N路融合图像数据的AWB校准参数和CCM参数，并根据色调映射处理参数对AWB增益参数进行校准；其中，第二图像信号处理器具体用于利用校准后的AWB增益参数和CCM参数对M路融合图像数据进行自动白平衡校准和色彩校准。
在某些实施方式中，第二中央处理器在根据色调映射处理参数对AWB增益参数进行校准时，具体用于：对色调映射处理后的融合图像数据进行反向色调映射处理；根据如下公式计算AWB增益校准参数：R Gain校准＝R Gain/Cr/Cg；B Gain校准＝B Gain/Cb/Cg；其中，R Gain校准为校准后的R增益，B Gain校准为校准后的B增益，R Gain为校准前的R增益，Cr/Cg为R的相对G增益，Cb/Cg为B的相对G增益，Cr＝Rsum/Rsum_untonemapping，Cg＝Gsum/Gsum_untonemapping，Cb＝Bsum/Bsum_untonemapping，Rsum、Gsum、Bsum分别为色调映射处理后的融合图像的R、G、B分量总值，Rsum_untonemapping、Gsum_untonemapping、Bsum_untonemapping分别为反向色调映射处理后的融合图像的R、G、B分量总值。
本申请的电子设备包括图像处理芯片和应用处理芯片。图像处理芯片用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N；应用处理芯片用于从图像处理芯片处获得N路融合图像数据，并对N路融合图像数据进行校准处理。
在某些实施方式中，图像处理芯片具体用于：将M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数，且2≤m≤M；根据如下公式对各组中的m路原始图像数据进行融合处理：Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)，其中，Pixel_Value_j_Fusioned表示N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像中第i路原始图像的像素值，ki表示m路原始图像的曝光时间中的最长曝光时间与第i路原始图像的曝光时间的比值，i为整数，且1＜i≤m。
在某些实施方式中，图像处理芯片还用于：对各路融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数，并将N路色调映射处理后的融合图像数据及其对应的色调映射处理参数发送给应用处理芯片。
在某些实施方式中，第一图像信号处理器在对融合图像数据进行色调映射处理时，具体用于：确定融合图像数据的感兴趣区域；基于感兴趣区域进行直方图均衡化处理，得到直方图均衡化映射关系，其中，直方图均衡化映射关系为色调映射处理参数；将直方图均衡化映射关系映射到融合图像数据的全图。
在某些实施方式中，第一图像信号处理器还用于：统计得到M路原始图像数据的3A统计信息，其中，3A统计信息包括自动曝光统计信息、自动白平衡统计信息和自动对焦统计信息；其中，图像处理芯片还用于将3A统计信息发送给应用处理芯片。
在某些实施方式中，图像处理芯片还用于：对3A统计信息、色调映射处理后的融合 图像数据和色调映射处理参数进行编码，以得到编码信息，并将编码信息发送给应用处理芯片。
在某些实施方式中，应用处理芯片还包括：第二中央处理器，用于利用3A算法根据M路原始图像数据的3A统计信息、N路融合图像数据的色调映射处理参数，得到N路融合图像数据的AWB校准参数和CCM参数，并根据色调映射处理参数对AWB增益参数进行校准；其中，第二图像信号处理器具体用于利用校准后的AWB增益参数和CCM参数对M路融合图像数据进行自动白平衡校准和色彩校准。
在某些实施方式中，第二中央处理器在根据色调映射处理参数对AWB增益参数进行校准时，具体用于：对色调映射处理后的融合图像数据进行反向色调映射处理；根据如下公式计算AWB增益校准参数：R Gain校准＝R Gain/Cr/Cg；B Gain校准＝B Gain/Cb/Cg；其中，R Gain校准为校准后的R增益，B Gain校准为校准后的B增益，R Gain为校准前的R增益，Cr/Cg为R的相对G增益，Cb/Cg为B的相对G增益，Cr＝Rsum/Rsum_untonemapping，Cg＝Gsum/Gsum_untonemapping，Cb＝Bsum/Bsum_untonemapping，Rsum、Gsum、Bsum分别为色调映射处理后的融合图像的R、G、B分量总值，Rsum_untonemapping、Gsum_untonemapping、Bsum_untonemapping分别为反向色调映射处理后的融合图像的R、G、B分量总值。
本申请的图像处理方法包括获取M路原始图像数据；对M路原始图像数据进行融合处理，以得到N路融合图像数据；对N路融合图像数据进行校准处理。
在某些实施方式中，对M路原始图像数据进行融合处理，包括：将M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数，且2≤m≤M；根据如下公式对各组中的m路原始图像数据进行融合处理：Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)，其中，Pixel_Value_j_Fusioned表示N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像中第i路原始图像的像素值，ki表示m路原始图像的曝光时间中的最长曝光时间与第i路原始图像的曝光时间的比值，i为整数，且1＜i≤m。
在某些实施方式中，图像处理方法还包括：对各路融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数。
在本申请的一个实施例中，如图1所示，当可采集多路原始图像数据的电子设备进行图像采集时，需要不停地将摄像头的图像传感器获取的多路原始RAW数据依次传输给图像处理芯片和应用处理芯片进行处理。若将多路原始图像数据传输到应用处理芯片进行处理，会造成传输的数据量较大，对带宽要求较高，功耗也较高。并且，参见图1，若采用MIPI(Mobile Industry Processor Interface，移动行业处理器接口)进行数据传输，则受限于硬件和成本，难以实现太多路的数据传输。
具体地，作为一个示例，电子设备在平滑变焦等模式下拍摄图像时，多个摄像头同时拍摄，需依次向图像处理芯片和向应用处理芯片传输多张原始图像数据，以及每一张原始图像的3A统计信息(3A stats)，其中，3A统计信息包括自动曝光统计信息、自动白平衡统计信息和自动对焦统计信息，数据传输量较大，对传输带宽的要求高，传输数据的功耗高。
作为另一个示例，电子设备在DOL(Digital overlap，数字重叠)模式下拍摄图像时， 需依次向图像处理芯片和向应用处理芯片传输摄像头的图像传感器输出的多路曝光图像，以及每路曝光图像的3A统计信息和PD。以两个摄像头为例，需要统计3路*2颗*3A，至少需要使用和传输18种类别的统计数据，加上(3路Raw图像+3路PD)*2(颗)数据，一共有30路数据，受限于硬件和成本，MIPI(Mobile Industry Processor Interface，移动行业处理器接口)硬件的数据通路数量不能满足需求，其中，PD为Phase Data(相位信息)，PD用于对焦。
为此，本申请提出一种图像处理芯片、应用处理芯片、电子设备和图像处理方法，旨在解决数据量大，MIPI硬件的数据通路数量少，不能满足数据传输需求的问题。下面将结合说明书附图2-11以及具体的实施方式对本申请实施例的图像处理芯片、应用处理芯片、电子设备和图像处理方法进行详细地说明。
图2是本申请实施例的图像处理芯片的结构示意图。
图2所示，图像处理芯片2包括第一图像信号处理器21。其中，第一图像信号处理器21用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N；图像处理芯片2还用于将融合图像数据发送给应用处理芯片3。
具体地，参见图2，M路原始图像数据可以是一个或多个图像传感器获得的，例如，M路原始图像数据可以是图像传感器通过在数字重叠DOL模式下获得的，若图像传感器的数量为2两个，则可得到M＝2*3＝6路原始图像数据。第一图像信号处理器21将M路(如6路)原始图像数据融合处理成N(N＜M，如M＝6时，N＝2)路融合图像数据后，图像处理芯片2将N路融合图像数据传输给应用处理芯片3。由此，可降低图像处理芯片2向应用处理芯片3回传数据时对传输带宽的要求，降低回传数据时功耗。
其中，图像传感器可采用CMOS(Complementary Metal Oxide Semiconductor，互补金属氧化物半导体)、CCD(Charge-coupled Device，感光元件或者电荷耦合元件)等感光元件。
在该实施例中，原始图像数据，为图像传感器采集得到的未经加工的图像数据，是CMOS(Complementary Metal Oxide Semiconductor，互补金属氧化物半导体)、CCD(Charge-coupled Device，感光元件或者电荷耦合元件)等感光元件将捕捉到的光源信号转化为数字信号的原始数据。原始图像数据记录有图像传感器的原始信息，同时还记录由摄像头拍摄所产生的一些元数据，如ISO的设置、快门速度、光圈值、白平衡等。若图像传感器可工作在数字重叠DOL模式下，则每个图像传感器获得的原始图像数据包括多张曝光图像。例如，当在3DOL模式下获取原始图像数据时，获取的原始图像数据会包括3路曝光图像，如长曝光图像、中间曝光图像和短曝光图像。
在本申请的实施例中，图像传感器的数量可为一个或多个(大于等于两个)，用于获取M路原始图像数据。当图像传感器在DOL模式下获取原始图像数据时，每个图像传感器获取到的多路原始图像数据为多路曝光图像数据。
作为一个可行的实施方式，图像处理芯片2可用在具有摄像头的电子设备中，为更好的支持ZSL(Zero Shutter Lang，零延时拍照)拍照，需不停地将摄像头的图像传感器采集的M路原始图像数据输入至图像处理芯片2，通过第一图像信号处理器21将M路原始图像数据融合处理成N(N＜M)路融合图像数据后，图像处理芯片2将N路融合图像数据传输给应用处理芯片3。由此，可降低图像处理芯片2向应用处理芯片3回传数据时对 传输带宽的要求，降低回传数据时功耗，有利于零延迟拍照技术在低端平台落地。
在本申请的一个实施例中，第一图像信号处理器21具体用于将M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数，且2≤m≤M；根据如下公式对各组中的m路原始图像数据进行融合处理：
Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)(1)
其中，Pixel_Value_j_Fusioned表示N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像数据中第i路原始图像数据的像素值，ki表示m路原始图像数据的曝光时间中的最长曝光时间与第i路原始图像数据的曝光时间的比值，i为整数，且1＜i≤m。
作为一具体的实施例，参见图3，第一图像信号处理器21可包括第一ISP(Image Signal Processing，图像信号处理)模块、融合模块，其中，第一ISP模块、融合模块的数量可以是一个，也可以是N个，若是N个，则第一ISP模块、融合模块与每组中的m路原始图像数据一一对应，此时m路原始图像数据依次输入至对应的第一ISP模块、融合模块进行处理；若是一个，则第一ISP模块和融合模块可对N组原始图像数据进行并行处理。由此，可保证图像处理效率。参见图3，图像处理芯片2还可包括神经网络处理器，记为NPU(Neural-network Processing Unit，神经网络处理单元)模块。
在该实施例中，N个第一ISP模块用于接收M路原始图像数据，并对接收到的原始图像数据进行预处理，获取该路的预览图像。
具体地，第一ISP模块处理图像传感器传输过来的原始图像数据，以匹配不同型号的图像传感器。同时，第一ISP模块通过一系列数字图像处理算法完成对原始图像数据的效果处理，主要可包括3A(自动白平衡、自动对焦、自动曝光)、坏点校正、去噪、强光抑制、背光补偿、色彩增强、镜头阴影校正等处理，得到预览图像。
NPU模块用于利用AI算法分别对各预览图像进行处理。
具体地，NPU模块利用AI算法完成对各预览图像的Demosaic(反马赛克)差值算法、自动白平衡、色彩校正、降噪、HDR(High-Dynamic Range，高动态范围图像)、超分辨率等处理。
融合模块用于对经AI算法处理后的对应的预览图像进行融合处理，得到N路融合图像。
具体地，图像传感器传输的原始图像数据，虽然经过第一ISP模块和NPU模块处理，但是其数据量并没有减小。通过融合模块对经过第一ISP模块和NPU模块处理后的图像进行融合处理，将M路原始图像数据转为成N路融合图像，可降低数据传输带宽，节省功耗。
作为一个具体实施例，参见图4，当在3DOL模式下获取原始图像数据时，每个图像传感器获取的原始图像数据包括3路曝光图像(长曝光图像、中间曝光图像和短曝光图像)，因此在对长曝光图像、中间曝光图像和短曝光图像进行融合处理时，可根据如下公式对原始图像进行融合处理：
Pixel_Value_Fusioned＝Pixel_Value_长+Pixel_Value_中*4+Pixel_Value_短*16，
其中，Pixel_Value_Fusioned表示融合图像的像素值，Pixel_Value_长表示长曝光图像的像素值，Pixel_Value_中表示中间曝光图像的像素值，Pixel_Value_短表示短曝光图像的 像素值。
在该实施例中，长曝光图像的曝光时间t长、中间曝光图像的曝光时间t中和短曝光图像的曝光时间t短呈四倍关系：t长＝4*t中＝16*t短。
在该实施例中，融合模块在对预览图像中的曝光图像进行处理时，会对曝光图像进行重排。作为一个示例，如图4所示，融合模块可将3张10bits的曝光图像融合成一路30bits的融合图像。
在本申请的实施例中，第一图像信号处理器21还用于对各路融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数；其中，图像处理芯片2还用于将N路色调映射处理后的融合图像数据及其对应的色调映射处理参数发送给应用处理芯片3。
作为一具体的实施例，第一图像信号处理器21可包括色调映射模块。色调映射模块可与第一ISP模块、融合模块一一对应，即色调映射模块的数量与第一ISP模块的数量、融合模块的数量相同，第一ISP模块的数量、融合模块的数量均为N时，色调映射模块的数量也是N，第一ISP模块的数量、融合模块的数量均为1时，色调映射模块的数量也是1，以实现经第一ISP模块、融合模块处理后的融合图像，可传输至对应的色调映射模块进行处理，保证数据处理的可靠性。色调映射模块用于对融合图像进行色调映射处理，得到色调映射处理后的融合图像和色调映射处理参数。具体地，色调映射模块可采用色调映射算法(tone mapping)，对融合处理得到的高位宽融合图像进行色调映射处理。如图5所示，融合处理后得到的30bits的融合图像，经过色调映射处理后，可以得到10bits的图像。
在本申请的实施例中，第一图像信号处理器21在对融合图像数据进行色调映射处理时，具体用于：确定融合图像数据的感兴趣区域；基于感兴趣区域进行直方图均衡化处理，得到直方图均衡化映射关系，其中，直方图均衡化映射关系为色调映射处理参数；将直方图均衡化映射关系映射到融合图像数据的全图。
具体地，确定融合图像的感兴趣区域，以针对性地对图像中的某一部分进行增强，对于划定感兴趣区域的方法可以为用户输入的方式。对于划定的感兴趣图像的个数，可以为一个或多个。对于获取的感兴趣图像的形状可以为多边形、椭圆形等。直方图均衡化就是将图像进行非线性的拉伸，重新将图像的像素值进行分配，以达到一定灰度范围内的像素数量大致相同，使得给定的直方图分布变换为一种均匀的直方图分布，从而获得最大的对比度。基于感兴趣区域进行直方图均衡化处理时，记录直方图均衡化映射关系。基于该直方图均衡化映射关系，映射到融合图像的全图，以对融合图像的全图进行直方图均衡化处理，保证ROI区域的信息保真度最高。
作为一个示例，在得到ROI区域后，可进一步得到扩展区域，该扩展区域的尺寸可以是(ROI区域的宽高*1.25)，如ROI区域为矩形区域，扩展区域特为矩形区域，且扩展区域的长为ROI区域长*1.5，扩展区域的宽为ROI区域宽*1.5，两区域的中心重合。基于扩展区域进行直方图均衡化处理，得到直方图均衡化映射关系。
需要说明的是，直方图均衡化对于背景和前景都过亮或者过暗的图像都非常有用，能更好地显示曝光过度或者曝光不足照片中的细节。这种方法的一个主要优势在于其相当直观且是可逆操作，如果已知均衡化函数，那么就可以恢复原始的直方图，并且计算量不大。
在本申请的实施例中，第一图像信号处理器21还用于统计得到M路原始图像数据的 3A统计信息，其中，3A统计信息包括自动曝光统计信息、自动白平衡统计信息和自动对焦统计信息；其中，图像处理芯片2还用于将3A统计信息发送给应用处理芯片3。
具体地，第一图像信号处理器21可利用第一ISP模块统计得到M路原始图像数据的3A统计信息。其中，3A统计信息包括自动曝光统计信息(AE，Auto Exposure)、自动白平衡统计信息(AWB，Auto White Balance)和自动对焦统计信息(AF，Auto Focus)。
在本申请的实施例中，图像处理芯片2还用于对3A统计信息、色调映射处理后的融合图像数据、色调映射处理参数和PD数据进行编码，以得到编码信息，并将编码信息发送给应用处理芯片3。
作为一具体的实施例，参见图3，图像处理芯片2可包括MIPI-TX编码子模块，其中，MIPI-TX编码子模块可与上述的色调映射模块一一对应，即MIPI-TX编码子模块数量与色调映射模块的数量相同，可以是一个，也可以是N个。MIPI-TX编码子模块接收原始图像数据的3A统计信息、色调映射处理后的融合图像、色调映射处理参数和PD数据进行编码，以对原始图像数据的3A统计信息、色调映射处理后的融合图像、色调映射处理参数和PD数据进行编码，并将编码后的信息通过MIPI协议传输给应用处理芯片3。
本申请提供的图像处理芯片，对M路原始图像数据进行融合处理，得到N路融合图像数据，并对N路融合图像数据进行色调映射处理，大大减小了数据传输量，降低了数据传输过程中对带宽的要求，并且还具有降低功耗的作用，有助于零延迟拍照技术在低端平台应用。
本申请提供了一种应用处理芯片。
图6是本申请实施例的应用处理芯片的结构示意图。在本申请的实施例中，参见图2、图6，应用处理芯片3用于从图像处理芯片2处获得N路融合图像数据。
如图6所示，应用处理芯片3包括第二图像信号处理器31。其中，第二图像信号处理器31用于对N路融合图像数据进行校准处理；其中，N路融合图像是对M路原始图像数据进行融合处理而得到，其中M和N均为正整数，且M>N。
具体地，原始图像数据在经过图像处理芯片2融合或者融合且色调映射处理后，数据量大大减小。但是，图像处理芯片2在对融合图像进行色调映射处理后，影响了图像的3A准确性，因此需要对色调映射处理后的融合图像进行校准处理。作为一个示例，可获取色调映射处理后的融合图像，以及3A统计信息、色调映射处理参数和PD数据，以对融合图像数据进行校准处理进行校准，获得目标图像。
作为一个可行的实施方式，参见图7，应用处理芯片3可包括MIPI-RX解码子模块，第二图像信号处理器31可包括第二ISP模块。其中，MIPI-RX解码子模块、第二ISP模块的数量可以是一个，也可以是N个，具体可与图像处理芯片2中的MIPI-TX编码子模块的数量相同。
在该实施例中，MIPI-RX解码子模块用于接收对应MIPI-TX编码子模块的编码信息，并对编码信息进行解码，得到3A统计信息、色调映射处理后的融合图像、色调映射处理参数和PD数据，进而将色调映射处理后的融合图像传送给第二ISP模块。第二ISP模块用于在接收到对应的色调映射处理后的融合图像后，利用数字图像处理算法对色调映射处理后的融合图像进行预处理。第二ISP模块对色调映射处理后的融合图像进行的预处理与第一ISP模块进行的预处理相同，在此不在赘述。
在本申请的实施例中，参见图6、图7，应用处理芯片3还包括第二中央处理器32，第二中央处理器32的数量可以是一个，也可以是N个，具体可与MIPI-RX解码子模块、第二ISP模块的数量相同。第二中央处理器32用于利用3A算法根据M路原始图像数据的3A统计信息、N路融合图像数据的色调映射处理参数，得到N路融合图像数据的AWB校准参数和CCM参数，并根据色调映射处理参数对AWB增益参数进行校准；其中，第二图像信号处理器31具体用于利用校准后的AWB增益参数和CCM参数对M路融合图像数据进行自动白平衡校准和色彩校准。
具体地，第二中央处理器32用于在接收到对应的3A统计信息、色调映射处理参数和PD数据后，利用3A算法根据3A统计信息、色调映射处理参数和PD数据得到AWB校准参数和CCM(Color Correct Matrix，色彩校正)参数，并根据色调映射处理参数对AWB增益参数进行校准。
作为一个示例，参见图8，第二中央处理器32可利用图像融合压缩前的3A统计信息和图像融压缩后的3A统计信息进行对比，以对应用处理芯片3端接收到的RAW图像颜色进行校准，通过融合压缩前后RGB统计对比得到比值系数，使用该比值在应用处理芯片端对AWB算法的结果(RGB Gain)进行校正，使用校正之后的3A算法结果对应用处理芯片3的Raw图进行颜色校准。
在本申请的实施例中，第二中央处理器32在根据色调映射处理参数对AWB增益参数进行校准时，具体可用于：
对色调映射处理后的融合图像数据进行反向色调映射处理；
根据如下公式计算AWB增益校准参数：
RGain校准＝RGain/Cr/Cg；
BGain校准＝BGain/Cb/Cg；
其中，RGain校准为校准后的R增益，BGain校准为校准后的B增益，RGain为校准前的R增益，Cr/Cg为R的相对G增益，Cb/Cg为B的相对G增益，Cr＝Rsum/Rsum_untonemapping，Cg＝Gsum/Gsum_untonemapping，Cb＝Bsum/Bsum_untonemapping，Rsum、Gsum、Bsum分别为色调映射处理后的融合图像的R、G、B分量总值，Rsum_untonemapping、Gsum_untonemapping、Bsum_untonemapping分别为反向色调映射处理后的融合图像的R、G、B分量总值。
进一步地，利用校准后的AWB增益参数和CCM参数对色调映射处理后的融合图像进行自动白平衡校准和色彩校准。
综上，本申请实施例的应用处理芯片，通过对由M路原始图像数据融合得到的N路融合图像数据进行校准处理，可保证图像的显示效果。
本申请还提供了一种电子设备。
参见图9和图10，电子设备10包括图像处理芯片2和应用处理芯片3。
在该实施例中，图像处理芯片2用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N。
具体地，图像处理芯片2具体用于：将M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数，且2≤m≤M；根据如下公式对各组中的m路原始图像数据进行融合处理：
Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)，
其中，Pixel_Value_j_Fusioned表示N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像数据中第i路原始图像数据的像素值，ki表示m路原始图像数据的曝光时间中的最长曝光时间与第i路原始图像数据的曝光时间的比值，i为整数，且1＜i≤m。
在本申请的一个实施例中，图像处理芯片2还用于：对各路融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数，并将N路色调映射处理后的融合图像数据及其对应的色调映射处理参数发送给应用处理芯片3。
应用处理芯片3用于从图像处理芯片处获得N路融合图像数据，并对N路融合图像数据进行校准处理。
本申请实施例的电子设备可以是移动终端，如智能手机、平板电脑等。
需要说明的是，本申请实施例的电子设备10中的图像处理芯片2和应用处理芯片3的其他具体实施方式可参见本申请上述实施例的图像处理芯片2和应用处理芯片3的具体实施方式。
另外，参见图9，图像处理芯片2还可包括CPU、存储器、计算机视觉引擎，其中，CPU可负责图像处理芯片2的控制，如上下电、固件加载，运行时控制等；存储器可用于存储图像数据处理过程中需存储的数据；计算机视觉引擎可以被配置来处理场景，产生表征所观察活动的信息流，并将信息流通过系统总线传输给其他模块，以学习相应场景的对象行为。应用处理器芯片3还可包括存储器，用以存储图像数据处理过程中需存储的数据。
本申请实施例提供的电子设备，通过图像处理芯片将图像传感器传输过来的原始图像进行融合或融合且色调映射处理，并将压缩的融合图像发送给应用处理芯片，大大减小了数据传输量，降低了数据传输过程中对带宽的要求，并且还具有降低功耗的作用。本申请实施例提供的电子设备可应用到多摄像头(如两摄像头，分别为主副摄像头)场景中，主副摄像头同步使用该方法降低带宽，对主副摄对融合时的色调映射的参数进行同步和综合，使色调映射更准确。
本申请还提供了一种图像处理方法。
图11是本申请实施例的图像处理方法的流程示意图。如图11所示，图像处理方法包括：
S1，获取M路原始图像数据。
具体地，可利用图像传感器获取M路原始图像数据，其中，原始图像为在数字重叠DOL模式下获得。其中，图像传感器为一种感光元件，是利用光电器件的光电转换功能将感光面上的光像转换为与光像成相应比例关系的电信号。图像传感器可采用CMOS、CCD等感光元件。
具体而言，CMOS图像传感器本质是一块芯片，主要包括：感光区阵列(Bayer阵列)、时序控制、模拟信号处理以及模数转换等模块。主要作用把光信号转化成电信号，然后通过ADC(Analog-to-digital converter，模拟数字转换器)转化成数字信号。
S2，对M路原始图像数据进行融合处理，以得到N路融合图像数据。
作为一可行地实施方式，对M路原始图像数据进行融合处理，可包括：
将M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数， 且2≤m≤M；
根据如下公式对各组中的m路原始图像数据进行融合处理：
Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)，
其中，Pixel_Value_j_Fusioned表示N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像数据中第i路原始图像数据的像素值，ki表示m路原始图像数据的曝光时间中的最长曝光时间与第i路原始图像数据的曝光时间的比值，i为整数，且1＜i≤m。
在本申请的实施例中，图像处理方法还包括：对各路融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数。
S3，对N路融合图像数据进行校准处理。
需要说明的是，本申请实施例的图像处理方法的其他具体实施方式可参见本申请上述实施例的图像处理芯片和应用处理芯片的具体实施方式。
本申请实施例提供的图像处理方法，对M路原始图像进行融合或融合且色调映射处理，并对色调映射处理后融合图像进行校正，大大减小了数据传输量，降低了数据传输过程中对带宽的要求，并且还具有降低功耗的作用。另外，本申请实施例提供的图像处理方法可应用到多摄像头(如两摄像头，分别为主副摄像头)场景中，主副摄像头同步使用该方法降低带宽，对主副摄对融合时的色调映射的参数进行同步和综合，使色调映射更准确。
需要说明的是，在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，“计算机可读介质”可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(电子装置)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一 定指的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。
在本申请的描述中，需要理解的是，术语“中心”、“纵向”、“横向”、“长度”、“宽度”、“厚度”、“上”、“下”、“前”、“后”、“左”、“右”、“竖直”、“水平”、“顶”、“底”“内”、“外”、“顺时针”、“逆时针”、“轴向”、“径向”、“周向”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本申请和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本申请的限制。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本申请的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。
在本申请中，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”、“固定”等术语应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或成一体；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通或两个元件的相互作用关系，除非另有明确的限定。对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本申请中的具体含义。
在本申请中，除非另有明确的规定和限定，第一特征在第二特征“上”或“下”可以是第一和第二特征直接接触，或第一和第二特征通过中间媒介间接接触。而且，第一特征在第二特征“之上”、“上方”和“上面”可是第一特征在第二特征正上方或斜上方，或仅仅表示第一特征水平高度高于第二特征。第一特征在第二特征“之下”、“下方”和“下面”可以是第一特征在第二特征正下方或斜下方，或仅仅表示第一特征水平高度小于第二特征。
尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。

Claims (20)

  1. 一种图像处理芯片，其特征在于，所述图像处理芯片包括：
    第一图像信号处理器，用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N；
    所述图像处理芯片还用于将所述融合图像数据发送给应用处理芯片。
  2. 根据权利要求1所述的图像处理芯片，其特征在于，所述第一图像信号处理器具体用于：
    将所述M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数，且2≤m≤M；
    根据如下公式对各组中的m路原始图像数据进行融合处理：
    Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)，
    其中，Pixel_Value_j_Fusioned表示所述N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像中第i路原始图像的像素值，ki表示所述m路原始图像的曝光时间中的最长曝光时间与第i路原始图像的曝光时间的比值，i为整数，且1＜i≤m。
  3. 根据权利要求1或2所述的图像处理芯片，其特征在于，所述第一图像信号处理器还用于：
    对各路所述融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数；
    其中，所述图像处理芯片还用于将N路色调映射处理后的融合图像数据及其对应的色调映射处理参数发送给所述应用处理芯片。
  4. 根据权利要求1所述的图像处理芯片，其特征在于，所述第一图像信号处理器在对所述融合图像数据进行色调映射处理时，具体用于：
    确定所述融合图像数据的感兴趣区域；
    基于所述感兴趣区域进行直方图均衡化处理，得到直方图均衡化映射关系，其中，所述直方图均衡化映射关系为所述色调映射处理参数；
    将所述直方图均衡化映射关系映射到所述融合图像数据的全图。
  5. 根据权利要求3所述的图像处理芯片，其特征在于，所述第一图像信号处理器还用于：
    统计得到所述M路原始图像数据的3A统计信息，其中，所述3A统计信息包括自动曝光统计信息、自动白平衡统计信息和自动对焦统计信息；
    其中，所述图像处理芯片还用于将所述3A统计信息发送给所述应用处理芯片。
  6. 根据权利要求5所述的图像处理芯片，其特征在于，所述图像处理芯片还用于：
    对所述3A统计信息、所述色调映射处理后的融合图像数据和所述色调映射处理参数进行编码，以得到编码信息，并将所述编码信息发送给所述应用处理芯片。
  7. 一种应用处理芯片，其特征在于，所述应用处理芯片用于从图像处理芯片处获得N路融合图像数据，所述应用处理芯片包括：
    第二图像信号处理器，用于对所述N路融合图像数据进行校准处理；
    其中，所述N路融合图像是对M路原始图像数据进行融合处理而得到，其中M和N均为正整数，且M>N。
  8. 根据权利要求7所述的应用处理芯片，其特征在于，所述应用处理芯片还包括：
    第二中央处理器，用于利用3A算法根据所述M路原始图像数据的3A统计信息、所述N路融合图像数据的色调映射处理参数，得到所述N路融合图像数据的AWB校准参数和CCM参数，并根据所述色调映射处理参数对所述AWB增益参数进行校准；
    其中，所述第二图像信号处理器具体用于利用校准后的AWB增益参数和所述CCM参数对所述M路融合图像数据进行自动白平衡校准和色彩校准。
  9. 根据权利要求8所述的应用处理芯片，其特征在于，所述第二中央处理器在根据所述色调映射处理参数对所述AWB增益参数进行校准时，具体用于：
    对所述色调映射处理后的融合图像数据进行反向色调映射处理；
    根据如下公式计算所述AWB增益校准参数：
    R Gain校准＝R Gain/Cr/Cg；
    B Gain校准＝B Gain/Cb/Cg；
    其中，R Gain校准为校准后的R增益，B Gain校准为校准后的B增益，R Gain为校准前的R增益，Cr/Cg为R的相对G增益，Cb/Cg为B的相对G增益，Cr＝Rsum/Rsum_untonemapping，Cg＝Gsum/Gsum_untonemapping，Cb＝Bsum/Bsum_untonemapping，Rsum、Gsum、Bsum分别为所述色调映射处理后的融合图像的R、G、B分量总值，Rsum_untonemapping、Gsum_untonemapping、Bsum_untonemapping分别为反向色调映射处理后的融合图像的R、G、B分量总值。
  10. 一种电子设备，其特征在于，包括：
    图像处理芯片，用于对M路原始图像数据进行融合处理，以得到N路融合图像数据，其中，M、N均为正整数，且M＞N；
    应用处理芯片，用于从所述图像处理芯片处获得N路融合图像数据，并对所述N路融合图像数据进行校准处理。
  11. 如权利要求10所述的电子设备，其特征在于，所述图像处理芯片具体用于：
    将所述M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数，且2≤m≤M；
    根据如下公式对各组中的m路原始图像数据进行融合处理：
    Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)，
    其中，Pixel_Value_j_Fusioned表示所述N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像中第i路原始图像的像素值，ki表示所述m路原始图像的曝光时间中的最长曝光时间与第i路原始图像的曝光时间的比值，i为整数，且1＜i≤m。
  12. 根据权利要求10或11所述的电子设备，其特征在于，所述图像处理芯片还用于：
    对各路所述融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数，并将N路色调映射处理后的融合图像数据及其对应的色调映射处理参数发送给所述应用处理芯片。
  13. 根据权利要求10所述的电子设备，其特征在于，所述第一图像信号处理器在对所述融合图像数据进行色调映射处理时，具体用于：确定所述融合图像数据的感兴趣区域；基于所述感兴趣区域进行直方图均衡化处理，得到直方图均衡化映射关系，其中，所述直方图均衡化映射关系为所述色调映射处理参数；将所述直方图均衡化映射关系映射到所述 融合图像数据的全图。
  14. 根据权利要求12所述的电子设备，其特征在于，所述第一图像信号处理器还用于：统计得到所述M路原始图像数据的3A统计信息，其中，所述3A统计信息包括自动曝光统计信息、自动白平衡统计信息和自动对焦统计信息；其中，所述图像处理芯片还用于将所述3A统计信息发送给所述应用处理芯片。
  15. 根据权利要求14所述的电子设备，其特征在于，所述图像处理芯片还用于：
    对所述3A统计信息、所述色调映射处理后的融合图像数据和所述色调映射处理参数进行编码，以得到编码信息，并将所述编码信息发送给所述应用处理芯片。
  16. 根据权利要求10所述的电子设备，其特征在于，所述应用处理芯片还包括：
    第二中央处理器，用于利用3A算法根据所述M路原始图像数据的3A统计信息、所述N路融合图像数据的色调映射处理参数，得到所述N路融合图像数据的AWB校准参数和CCM参数，并根据所述色调映射处理参数对所述AWB增益参数进行校准；
    其中，所述第二图像信号处理器具体用于利用校准后的AWB增益参数和所述CCM参数对所述M路融合图像数据进行自动白平衡校准和色彩校准。
  17. 根据权利要求16所述的电子设备，其特征在于，所述第二中央处理器在根据所述色调映射处理参数对所述AWB增益参数进行校准时，具体用于：
    对所述色调映射处理后的融合图像数据进行反向色调映射处理；
    根据如下公式计算所述AWB增益校准参数：
    R Gain校准＝R Gain/Cr/Cg；
    B Gain校准＝B Gain/Cb/Cg；
    其中，R Gain校准为校准后的R增益，B Gain校准为校准后的B增益，R Gain为校准前的R增益，Cr/Cg为R的相对G增益，Cb/Cg为B的相对G增益，Cr＝Rsum/Rsum_untonemapping，Cg＝Gsum/Gsum_untonemapping，Cb＝Bsum/Bsum_untonemapping，Rsum、Gsum、Bsum分别为所述色调映射处理后的融合图像的R、G、B分量总值，Rsum_untonemapping、Gsum_untonemapping、Bsum_untonemapping分别为反向色调映射处理后的融合图像的R、G、B分量总值。
  18. 一种图像处理方法，其特征在于，所述方法包括：
    获取M路原始图像数据；
    对所述M路原始图像数据进行融合处理，以得到N路融合图像数据；
    对所述N路融合图像数据进行校准处理。
  19. 根据权利要求18所述的图像处理方法，其特征在于，所述对所述M路原始图像数据进行融合处理，包括：
    将所述M路原始图像数据划分为N组，其中，每组包括m路原始图像数据，m为整数，且2≤m≤M；
    根据如下公式对各组中的m路原始图像数据进行融合处理：
    Pixel_Value_j_Fusioned＝(Pixel_Value_i*ki)，
    其中，Pixel_Value_j_Fusioned表示所述N路融合图像中第j路融合图像的像素值，Pixel_Value_i表示m路原始图像中第i路原始图像的像素值，ki表示所述m路原始图像的曝光时间中的最长曝光时间与第i路原始图像的曝光时间的比值，i为整数，且1＜i≤m。
  20. 根据权利要求18所述的图像处理方法，其特征在于，所述方法还包括：
    对各路所述融合图像数据进行色调映射处理，以得到色调映射处理后的融合图像数据和色调映射处理参数。
PCT/CN2022/112534 2021-09-15 2022-08-15 图像处理芯片、应用处理芯片、电子设备和图像处理方法 WO2023040540A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111081125.8A CN115835011A (zh) 2021-09-15 2021-09-15 图像处理芯片、应用处理芯片、电子设备和图像处理方法
CN202111081125.8 2021-09-15

Publications (1)

Publication Number Publication Date
WO2023040540A1 true WO2023040540A1 (zh) 2023-03-23

Family

ID=85514896

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/112534 WO2023040540A1 (zh) 2021-09-15 2022-08-15 图像处理芯片、应用处理芯片、电子设备和图像处理方法

Country Status (2)

Country Link
CN (1) CN115835011A (zh)
WO (1) WO2023040540A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102257531A (zh) * 2008-12-19 2011-11-23 高通股份有限公司 高动态范围图像组合
CN107948544A (zh) * 2017-11-28 2018-04-20 长沙全度影像科技有限公司 一种基于fpga的多路视频拼接系统及方法
US20190035048A1 (en) * 2017-07-26 2019-01-31 Altek Semiconductor Corp. Image processing chip and image processing system
CN112669241A (zh) * 2021-01-29 2021-04-16 成都国科微电子有限公司 一种图像处理方法、装置、设备及介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102257531A (zh) * 2008-12-19 2011-11-23 高通股份有限公司 高动态范围图像组合
US20190035048A1 (en) * 2017-07-26 2019-01-31 Altek Semiconductor Corp. Image processing chip and image processing system
CN107948544A (zh) * 2017-11-28 2018-04-20 长沙全度影像科技有限公司 一种基于fpga的多路视频拼接系统及方法
CN112669241A (zh) * 2021-01-29 2021-04-16 成都国科微电子有限公司 一种图像处理方法、装置、设备及介质

Also Published As

Publication number Publication date
CN115835011A (zh) 2023-03-21

Similar Documents

Publication Publication Date Title
US8508619B2 (en) High dynamic range image generating apparatus and method
US8797421B2 (en) System and method to selectively combine images
US8111300B2 (en) System and method to selectively combine video frame image data
US6982756B2 (en) Digital camera, image signal processing method and recording medium for the same
US10325354B2 (en) Depth assisted auto white balance
US20230107373A1 (en) Imaging unit, imaging apparatus, and computer-readable medium having stored thereon a control program
US20070166020A1 (en) Hand jitter reduction system for cameras
US10600170B2 (en) Method and device for producing a digital image
EP4218226A1 (en) Image signal processing in multi-camera system
CN110830789A (zh) 一种过曝检测方法、装置及一种过曝抑制方法、装置
WO2023036034A1 (zh) 图像处理方法及其相关设备
US20110187877A1 (en) Image Correction For Image Capturing With an Optical Image Stabilizer
JP2013085176A (ja) 撮像装置
KR102285756B1 (ko) 전자 시스템 및 영상 처리 방법
EP2446414B1 (en) Lens roll-off correction operation using values corrected based on brightness information
WO2023040540A1 (zh) 图像处理芯片、应用处理芯片、电子设备和图像处理方法
CN116567432A (zh) 拍摄方法和电子设备
WO2022073364A1 (zh) 图像获取方法及装置、终端和计算机可读存储介质
CN113873142B (zh) 多媒体处理芯片、电子设备和动态图像处理方法
CN115225828B (zh) 前置图像信号处理装置及图像信息传输方法
JP4356651B2 (ja) 画像処理装置、撮像装置、および画像処理方法
JP2015119436A (ja) 撮像装置
JP7447947B2 (ja) 電子機器
US20230017498A1 (en) Flexible region of interest color processing for cameras
KR20230015161A (ko) 촬영 장치

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE