WO2021196553A1

WO2021196553A1 - High-dynamic-range image processing system and method, electronic device and readable storage medium

Info

Publication number: WO2021196553A1
Application number: PCT/CN2020/119959
Authority: WO
Inventors: 杨鑫
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-04-03
Filing date: 2020-10-09
Publication date: 2021-10-07
Also published as: CN111479071B; CN111479071A

Abstract

A high-dynamic-range image processing system (100), a high-dynamic-range image processing method, an electronic device (1000) and a computer-readable storage medium. The high-dynamic-range image processing system (100) comprises an image sensor (10), an image fusion module (20) and a high-dynamic-range image processing module (30). A pixel array (11) in the image sensor (10) is exposed. The image fusion module (20) and the high-dynamic-range image processing module (30) are used for performing high-dynamic-range processing and fusion algorithm processing on a first color original image, a second color original image and a full color original image, so as to obtain a first high-dynamic-range image.

Description

High dynamic range image processing system and method, electronic equipment and readable storage medium

Priority information

This application requests the priority and rights of the patent application with the patent application number 202010259292.6 filed with the State Intellectual Property Office of China on April 3, 2020, and the full text is incorporated herein by reference.

Technical field

This application relates to the field of image processing technology, and in particular to a high dynamic range image processing system, a high dynamic range image processing method, electronic equipment, and a non-volatile computer-readable storage medium.

Background technique

A camera may be provided in an electronic device such as a mobile phone to realize a photographing function. An image sensor for receiving light can be set in the camera. The image sensor may be provided with a filter array.

Summary of the invention

The embodiments of the present application provide a high dynamic range image processing system, a high dynamic range image processing method, electronic equipment, and a non-volatile computer-readable storage medium.

The embodiment of the present application provides a high dynamic range image processing system. The high dynamic range image processing system includes an image sensor, an image fusion module and a high dynamic range image processing module. The image sensor includes a pixel array, the pixel array includes a plurality of panchromatic photosensitive pixels and a plurality of color photosensitive pixels, the color photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixels, and the pixel array includes The smallest repeating unit, each of the smallest repeating units includes a plurality of sub-units, each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels, and the pixel array in the image sensor is exposed to light. Wherein, for a plurality of photosensitive pixels in the same subunit, at least one of the single-color photosensitive pixels is exposed for a first exposure time, and at least one of the single-color photosensitive pixels is exposed for a second exposure that is less than the first exposure time Time exposure, at least one of the full-color photosensitive pixels is exposed at a third exposure time that is less than the first exposure time. Wherein, the first color information generated by the single-color photosensitive pixel exposed at the first exposure time obtains a first color original image, and the second color generated by the single-color photosensitive pixel exposed at the second exposure time The information obtains a second color original image, and the panchromatic photosensitive pixels exposed at the third exposure time generate a first panchromatic original image. The image fusion module and the high dynamic range image processing module are used to perform fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image To get the first high dynamic range image. The first high dynamic range image includes a plurality of color image pixels, and the plurality of color image pixels are arranged in a Bayer array. The first high dynamic range image is processed by an image processor to obtain a second high dynamic range image.

The embodiments of the present application provide a high dynamic range image processing method. The high dynamic range image processing method is used in a high dynamic range image processing system. The high dynamic range image processing system includes an image sensor, the image sensor includes a pixel array, the pixel array includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels, and the color photosensitive pixels have a higher sensitivity than the full-color photosensitive pixels. The pixel has a narrower spectral response. The pixel array includes a minimum repeating unit, each of the minimum repeating units includes a plurality of sub-units, and each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. The high dynamic range image processing method includes: controlling the exposure of the pixel array, wherein, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed at a first exposure time, and at least one The single-color photosensitive pixel is exposed at a second exposure time that is less than the first exposure time, and at least one full-color photosensitive pixel is exposed at a third exposure time that is less than the first exposure time. Wherein, the first color information generated by the single-color photosensitive pixel exposed at the first exposure time obtains a first color original image, and the second color generated by the single-color photosensitive pixel exposed at the second exposure time The information obtains a second color original image, and the panchromatic photosensitive pixels exposed at the third exposure time generate a first panchromatic original image. And performing fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain a first high dynamic range image. The first high dynamic range image includes a plurality of color image pixels, and the plurality of color image pixels are arranged in a Bayer array. The first high dynamic range image is processed by an image processor to obtain a second high dynamic range image.

The embodiment of the present application provides an electronic device. The electronic device includes a lens, a housing, and the above-mentioned high dynamic range image processing system. The lens and the high dynamic range image processing system are combined with the housing, and the lens cooperates with the image sensor of the high dynamic range image processing system for imaging.

The embodiments of the present application provide a non-volatile computer-readable storage medium containing a computer program. When the computer program is executed by the processor, the processor is caused to execute the above-mentioned high dynamic range image processing method.

The additional aspects and advantages of the embodiments of the present application will be partly given in the following description, and part of them will become obvious from the following description, or be understood through the practice of the present application.

Description of the drawings

The above-mentioned and/or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, in which:

FIG. 1 is a schematic diagram of a high dynamic range image processing system according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a pixel array according to an embodiment of the present application;

3 is a schematic cross-sectional view of a photosensitive pixel according to an embodiment of the present application;

FIG. 4 is a pixel circuit diagram of a photosensitive pixel according to an embodiment of the present application;

FIG. 5 is a schematic diagram of the arrangement of the smallest repeating unit in a pixel array according to an embodiment of the present application;

6 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application;

FIG. 7 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application;

FIG. 8 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application;

FIG. 9 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application;

FIG. 10 is a schematic diagram of the arrangement of the smallest repeating unit in another pixel array according to an embodiment of the present application;

FIG. 11 is a schematic diagram of an original image output by an image sensor according to an embodiment of the present application;

FIG. 12 is a schematic diagram of an image fusion processing principle according to an embodiment of the present application;

FIG. 13 is a schematic diagram of another image fusion processing principle according to an embodiment of the present application;

FIG. 14 is a schematic diagram of brightness alignment processing according to an embodiment of the present application;

FIG. 15 is a schematic diagram of a high dynamic range processing principle according to an embodiment of the present application;

16 is a schematic diagram of another high dynamic range image processing system according to an embodiment of the present application;

FIG. 17 is a schematic diagram of lens shading correction processing according to an embodiment of the present application;

FIG. 18 is a schematic diagram of another high dynamic range image processing system according to an embodiment of the present application;

FIG. 19 is a schematic diagram of another high dynamic range processing principle according to an embodiment of the present application;

FIG. 20 is a schematic diagram of an original image output by another image sensor according to an embodiment of the present application;

FIG. 21 is a schematic diagram of another high dynamic range processing principle according to an embodiment of the present application;

FIG. 22 is a schematic diagram of another image fusion processing principle according to an embodiment of the present application;

FIG. 23 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

FIG. 24 is a schematic flowchart of a method for acquiring a high dynamic range image according to an embodiment of the present application;

FIG. 25 is a schematic diagram of interaction between a non-volatile computer-readable storage medium and a processor according to an embodiment of the present application.

Detailed ways

The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions throughout. The following embodiments described with reference to the drawings are exemplary, and are only used to explain the embodiments of the present application, and should not be understood as limitations on the embodiments of the present application.

Please refer to FIG. 1, an embodiment of the present application provides a high dynamic range image processing system 100. The high dynamic range image processing system 100 includes an image sensor 10, an image fusion module 20 and a high dynamic range image processing module 30. The image sensor 10 includes a pixel array 11. The pixel array 11 includes a plurality of panchromatic photosensitive pixels and a plurality of color photosensitive pixels, and the color photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixels. The pixel array 11 includes a minimum repeating unit, each minimum repeating unit includes a plurality of sub-units, and each sub-unit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. The pixel array 11 in the image sensor 10 is exposed to light, wherein, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed at a first exposure time, and at least one single-color photosensitive pixel is exposed at a first exposure time that is less than the first exposure time. Exposure at a second exposure time, at least one full-color photosensitive pixel is exposed at a third exposure time that is less than the first exposure time. Wherein, the first color information generated by the single-color photosensitive pixels exposed at the first exposure time obtains the first color original image, and the second color information generated by the single-color photosensitive pixels exposed at the second exposure time obtains the second color original image, The full-color photosensitive pixels exposed at the third exposure time generate a first full-color original image. The image fusion module 20 and the high dynamic range image processing module 30 are used to perform fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain the first high dynamic range image . The first high dynamic range image includes a plurality of color image pixels, and the plurality of color image pixels are arranged in a Bayer array. The first high dynamic range image is processed by the image processor 40 to obtain a second high dynamic range image.

Referring to FIG. 1, in some embodiments, part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time, and the fourth exposure time is less than or equal to the first exposure time. The exposure time is greater than the third exposure time. The image fusion module 20 is used for fusing the first color original image and the second panchromatic original image into a first intermediate image, and fusing the second color original image and the first panchromatic original image into a second intermediate image, with a fourth exposure The second panchromatic information generated by the time-exposed single-color photosensitive pixels obtains the second panchromatic original image. The high dynamic range image processing module 30 is used to fuse the first intermediate image and the second intermediate image into a first high dynamic range image.

Referring to FIG. 16, in some embodiments, the high dynamic range image processing module 30 includes a high dynamic range image processing unit 31 and a brightness mapping unit 33. The high dynamic range image processing unit 31 is used for fusing the first intermediate image and the second intermediate image into a third high dynamic range image. The brightness mapping unit 33 is configured to perform brightness mapping on the third high dynamic range image to obtain the first high dynamic range image.

Referring to FIG. 16, in some embodiments, the high dynamic range image processing module 30 includes a high dynamic range image processing unit 31, a lens shading correction unit 37 and a statistical unit 35. The high dynamic range image processing unit 31 is used for fusing the first intermediate image and the second intermediate image into a third high dynamic range image; the lens shading correction unit 37 is used for correcting the third high dynamic range image to obtain a high dynamic range corrected image; The statistical unit 35 is used to process the high dynamic range correction image to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.

Referring to FIG. 16, in some embodiments, the high dynamic range image processing module 30 includes a statistical unit 35, the statistical unit 35 is used to process the first intermediate image and the second intermediate image to obtain statistical data, the statistical data is provided to the image processing The device 40 performs automatic exposure processing and/or automatic white balance processing.

Referring to FIG. 18, in some embodiments, part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time, and the fourth exposure time is less than or equal to the first exposure time. The exposure time is greater than the third exposure time. The high dynamic range image processing module 30 is used for fusing the first color original image and the second color original image into a first high dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into a first high Dynamic panchromatic original image. The image fusion module 20 is used for fusing the first high dynamic color original image and the first high dynamic full color original image into a first high dynamic range image.

Referring to FIG. 18, in some embodiments, the high dynamic range image processing module 30 further includes a high dynamic range image processing unit 31 and a brightness mapping unit 33. The high dynamic range image processing unit 31 is used for fusing the first color original image and the second color original image into a second high dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into the second high Dynamic full-color original image; the brightness mapping unit 33 is used to perform brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image, and perform brightness mapping on the second high dynamic full color original image to obtain the first high Dynamic panchromatic original image.

Referring to FIG. 18, in some embodiments, the high dynamic range image processing module 30 further includes a high dynamic range image processing unit 31, a lens shading correction unit 37 and a statistics unit 35. The high dynamic range image processing unit 31 is used for fusing the first color original image and the second color original image into a second high dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into the second high Dynamic panchromatic original image; the lens shading correction unit 37 is used to correct the second highly dynamic color original image to obtain a highly dynamic color corrected image, and correct the second highly dynamic panchromatic original image to obtain a highly dynamic panchromatic corrected image; the statistical unit 35 It is used to process the high dynamic color correction image and the high dynamic full color correction image to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.

Referring to FIG. 18, in some embodiments, all panchromatic photosensitive pixels in the same subunit are exposed at the third exposure time; the high dynamic range image processing module 30 is used to combine the first color original image with the second color original image Fusion into a first high dynamic color original image; the image fusion module 20 is used for fusing the first high dynamic color original image and the first panchromatic original image into a first high dynamic range image.

Referring to FIG. 18, in some embodiments, the high dynamic range image processing module 30 further includes a high dynamic range image processing unit 31 and a brightness mapping unit 33. The high dynamic range image processing unit 31 is used to fuse the first color original image and the second color original image into a second high dynamic color original image; the brightness mapping unit 33 is used to perform brightness mapping on the second high dynamic color original image to obtain The first high-dynamic color original image.

Referring to FIG. 18, in some embodiments, the high dynamic range image processing module 30 further includes a high dynamic range image processing unit 31, a lens shading correction unit 37 and a statistics unit 35. The high dynamic range image processing unit 31 is used for fusing the first color original image and the second color original image into a second high dynamic color original image. The lens shading correction unit 37 is used to correct the second high dynamic color original image to obtain a high dynamic color corrected image. The statistical unit 35 is used to process the high dynamic color correction image and the first full-color original image to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.

In some embodiments, both the image fusion module 20 and the high dynamic range image processing module 30 are integrated in the image sensor.

Please refer to FIG. 24. This application also provides a high dynamic range image processing method. The high dynamic range image processing method of the embodiment of the present application is used in the high dynamic range image processing system 100. The high dynamic range image processing system 100 includes an image sensor 10. The image sensor 10 includes a pixel array 11. The pixel array 11 includes a plurality of full-color photosensitive pixels and a plurality of color photosensitive pixels. Color photosensitive pixels have a narrower spectral response than full-color photosensitive pixels. The pixel array 11 includes a minimum repeating unit, and each minimum repeating unit includes a plurality of sub-units. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels. High dynamic range image processing methods include:

01: Exposure of the pixel array 11, where, for multiple photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed with a first exposure time, and at least one single-color photosensitive pixel is exposed with a second exposure time less than the first exposure time Exposure, at least one full-color photosensitive pixel is exposed at a third exposure time that is less than the first exposure time; wherein, the first color information generated by the single-color photosensitive pixel exposed at the first exposure time obtains the first color original image, and the second The second color information generated by the single-color photosensitive pixels exposed at the exposure time obtains the second color original image, and the panchromatic photosensitive pixels exposed at the third exposure time generate the first full-color original image; and

02: Perform fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain the first high dynamic range image, the first high dynamic range image includes multiple color images Pixels, a plurality of color image pixels are arranged in a Bayer array, and the first high dynamic range image is processed by the image processor to obtain the second high dynamic range image.

In some embodiments, part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time. The fourth exposure time is less than or equal to the first exposure time and greater than The third exposure time, and the second panchromatic information generated by the single-color photosensitive pixels exposed at the fourth exposure time to obtain the second panchromatic original image; for the first color original image, the second color original image and the first panchromatic original Performing fusion algorithm processing and high dynamic range processing on the image to obtain the first high dynamic range image includes: fusing the first color original image and the second panchromatic original image into a first intermediate image, and combining the second color original image with the first full color original image. The color original image is fused into a second intermediate image; and the first intermediate image and the second intermediate image are fused into a first high dynamic range image.

In some embodiments, fusing the first intermediate image and the second intermediate image into a first high dynamic range image includes: fusing the first intermediate image and the second intermediate image into a third high dynamic range image; and The three high dynamic range images are subjected to brightness mapping to obtain the first high dynamic range image.

In some embodiments, the high dynamic range image processing method further includes: fusing the first intermediate image and the second intermediate image into a third high dynamic range image; obtaining a high dynamic range corrected image from the third high dynamic range image; And processing the high dynamic range correction image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.

In some embodiments, the high dynamic range image processing method further includes: processing the first intermediate image and the second intermediate image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing .

In some embodiments, part of the panchromatic photosensitive pixels in the same subunit are exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the third exposure time. The fourth exposure time is less than or equal to the first exposure time and greater than The third exposure time, and the second panchromatic information generated by the single-color photosensitive pixels exposed at the fourth exposure time to obtain the second panchromatic original image. Performing fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain the first high dynamic range image includes: combining the first color original image and the second color original image Fusion into a first high-dynamic color original image, fusing the first full-color original image and the second full-color original image into a first high-dynamic full-color original image; and combining the first high-dynamic color original image with the first high-dynamic full-color original image The color original image is fused into the first high dynamic range image.

In some embodiments, the first color original image and the second color original image are fused into a first high dynamic color original image, and the first panchromatic original image and the second panchromatic original image are fused into a first high dynamic color original image. The color original image includes: fusing the first color original image and the second color original image into a second high-dynamic color original image, and fusing the first panchromatic original image and the second panchromatic original image into a second high-dynamic panchromatic original image Image; and brightness mapping is performed on the second high dynamic color original image to obtain the first high dynamic color original image, and brightness mapping is performed on the second high dynamic panchromatic original image to obtain the first high dynamic full color original image.

In some embodiments, the high dynamic range image processing method includes: fusing a first color original image and a second color original image into a second high dynamic color original image, and combining the first panchromatic original image with the second panchromatic original image. Image fusion into the second high dynamic panchromatic original image; correct the second high dynamic color original image to obtain a high dynamic color corrected image, correct the second high dynamic panchromatic original image to obtain a high dynamic panchromatic corrected image; and process high dynamics The color correction image and the high dynamic full color correction image are used to obtain statistical data, and the statistical data is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.

In some embodiments, all panchromatic photosensitive pixels in the same subunit are exposed at the third exposure time; the first color original image, the second color original image, and the first panchromatic original image are processed by a fusion algorithm and are highly dynamic The range processing to obtain the first high dynamic range image includes: fusing the first color original image and the second color original image into a first high dynamic color original image; and combining the first high dynamic color original image and the first panchromatic original image Fusion is the first high dynamic range image.

In some embodiments, fusing the first color original image and the second color original image into a first high dynamic color original image includes: fusing the first color original image and the second color original image into a second high dynamic color original image And performing brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image.

In some embodiments, the high dynamic range image processing method includes: fusing a first color original image and a second color original image into a second high dynamic color original image; correcting the second high dynamic color original image to obtain a high dynamic color Correcting the image; and processing the high dynamic color correction image and the first full-color original image to obtain statistical data, which is provided to the image processor 40 for automatic exposure processing and/or automatic white balance processing.

Please refer to FIG. 23. The present application also provides an electronic device 1000. The electronic device 1000 of the embodiment of the present application includes a lens 300, a housing 200, and the high dynamic range image processing system 100 described in any one of the above embodiments.

Please refer to 25. This application also provides a non-volatile computer-readable storage medium 400 containing a computer program. When the computer program is executed by the processor 60, the processor 60 is caused to execute the high dynamic range image processing method described in any one of the foregoing embodiments.

The high dynamic range image processing system 100 of the embodiment of the present application performs fusion algorithm processing and high dynamic range processing on the panchromatic original image and color original image output by the image sensor 10 through the image fusion module 20 and the high dynamic range image processing module 30. In order to obtain the first high dynamic range image in which the image pixels are arranged in a Bayer array, the first high dynamic range image is input into the image processor to complete subsequent processing, thereby solving the problem that the image processor 40 cannot directly arrange the image pixels in a non-Bayer array. The problem of processing cloth images.

The application will be further explained below in conjunction with the drawings.

FIG. 2 is a schematic diagram of the image sensor 10 in the embodiment of the present application. The image sensor 10 includes a pixel array 11, a vertical driving unit 12, a control unit 13, a column processing unit 14 and a horizontal driving unit 15.

For example, the image sensor 10 may adopt a complementary metal oxide semiconductor (CMOS, Complementary Metal Oxide Semiconductor) photosensitive element or a charge-coupled device (CCD, Charge-coupled Device) photosensitive element.

For example, the pixel array 11 includes a plurality of photosensitive pixels 110 (shown in FIG. 3) arranged two-dimensionally in an array (ie, arranged in a two-dimensional matrix), and each photosensitive pixel 110 includes a photoelectric conversion element 1111 (shown in FIG. 4) . Each photosensitive pixel 110 converts light into electric charge according to the intensity of light incident thereon.

For example, the vertical driving unit 12 includes a shift register and an address decoder. The vertical drive unit 12 includes readout scanning and reset scanning functions. The readout scan refers to sequentially scanning the unit photosensitive pixels 110 line by line, and reading signals from these unit photosensitive pixels 110 line by line. For example, the signal output by each photosensitive pixel 110 in the selected and scanned photosensitive pixel row is transmitted to the column processing unit 14. The reset scan is used to reset the charge, and the photocharge of the photoelectric conversion element is discarded, so that the accumulation of new photocharge can be started.

For example, the signal processing performed by the column processing unit 14 is correlated double sampling (CDS) processing. In the CDS process, the reset level and the signal level output from each photosensitive pixel 110 in the selected photosensitive pixel row are taken out, and the level difference is calculated. Thus, the signals of the photosensitive pixels 110 in a row are obtained. The column processing unit 14 may have an analog-to-digital (A/D) conversion function for converting analog pixel signals into a digital format.

For example, the horizontal driving unit 15 includes a shift register and an address decoder. The horizontal driving unit 15 sequentially scans the pixel array 11 column by column. Through the selection scanning operation performed by the horizontal driving unit 15, each photosensitive pixel column is sequentially processed by the column processing unit 14, and is sequentially output.

For example, the control unit 13 configures timing signals according to the operation mode, and uses various timing signals to control the vertical driving unit 12, the column processing unit 14 and the horizontal driving unit 15 to work together.

FIG. 3 is a schematic diagram of a photosensitive pixel 110 in an embodiment of the present application. The photosensitive pixel 110 includes a pixel circuit 111, a filter 112, and a micro lens 113. Along the light-receiving direction of the photosensitive pixel 110, the microlens 113, the filter 112, and the pixel circuit 111 are arranged in sequence. The microlens 113 is used for condensing light, and the filter 112 is used for passing light of a certain waveband and filtering out the light of other wavebands. The pixel circuit 111 is used to convert the received light into electrical signals, and provide the generated electrical signals to the column processing unit 14 shown in FIG. 2.

FIG. 4 is a schematic diagram of a pixel circuit 111 of a photosensitive pixel 110 in an embodiment of the present application. The pixel circuit 111 in FIG. 4 can be applied to each photosensitive pixel 110 (shown in FIG. 3) in the pixel array 11 shown in FIG. The working principle of the pixel circuit 111 will be described below with reference to FIGS. 2 to 4.

As shown in FIG. 4, the pixel circuit 111 includes a photoelectric conversion element 1111 (for example, a photodiode), an exposure control circuit (for example, a transfer transistor 1112), a reset circuit (for example, a reset transistor 1113), and an amplification circuit (for example, an amplification transistor 1114). ) And a selection circuit (for example, a selection transistor 1115). In the embodiment of the present application, the transfer transistor 1112, the reset transistor 1113, the amplifying transistor 1114, and the selection transistor 1115 are, for example, MOS transistors, but are not limited thereto.

For example, the photoelectric conversion element 1111 includes a photodiode, and the anode of the photodiode is connected to the ground, for example. The photodiode converts the received light into electric charge. The cathode of the photodiode is connected to the floating diffusion unit FD via an exposure control circuit (for example, a transfer transistor 1112). The floating diffusion unit FD is connected to the gate of the amplification transistor 1114 and the source of the reset transistor 1113.

For example, the exposure control circuit is a transfer transistor 1112, and the control terminal TG of the exposure control circuit is the gate of the transfer transistor 1112. When a pulse of an active level (for example, VPIX level) is transmitted to the gate of the transfer transistor 1112 through the exposure control line, the transfer transistor 1112 is turned on. The transfer transistor 1112 transfers the charge photoelectrically converted by the photodiode to the floating diffusion unit FD.

For example, the drain of the reset transistor 1113 is connected to the pixel power supply VPIX. The source of the reset transistor 113 is connected to the floating diffusion unit FD. Before the charge is transferred from the photodiode to the floating diffusion unit FD, a pulse of an effective reset level is transmitted to the gate of the reset transistor 113 via the reset line, and the reset transistor 113 is turned on. The reset transistor 113 resets the floating diffusion unit FD to the pixel power supply VPIX.

For example, the gate of the amplifying transistor 1114 is connected to the floating diffusion unit FD. The drain of the amplifying transistor 1114 is connected to the pixel power supply VPIX. After the floating diffusion unit FD is reset by the reset transistor 1113, the amplifying transistor 1114 outputs the reset level through the output terminal OUT via the selection transistor 1115. After the charge of the photodiode is transferred by the transfer transistor 1112, the amplifying transistor 1114 outputs a signal level through the output terminal OUT via the selection transistor 1115.

For example, the drain of the selection transistor 1115 is connected to the source of the amplification transistor 1114. The source of the selection transistor 1115 is connected to the column processing unit 14 in FIG. 2 through the output terminal OUT. When the pulse of the active level is transmitted to the gate of the selection transistor 1115 through the selection line, the selection transistor 1115 is turned on. The signal output by the amplifying transistor 1114 is transmitted to the column processing unit 14 through the selection transistor 1115.

It should be noted that the pixel structure of the pixel circuit 111 in the embodiment of the present application is not limited to the structure shown in FIG. 4. For example, the pixel circuit 111 may also have a three-transistor pixel structure, in which the functions of the amplifying transistor 1114 and the selecting transistor 1115 are performed by one transistor. For example, the exposure control circuit is not limited to the way of a single transfer transistor 1112, and other electronic devices or structures with the function of controlling the conduction of the control terminal can be used as the exposure control circuit in the embodiment of the present application. The implementation of the transistor 1112 is simple, low in cost, and easy to control.

5 to 10 are schematic diagrams of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the pixel array 11 (shown in FIG. 2) according to some embodiments of the present application. The photosensitive pixels 110 include two types, one is a full-color photosensitive pixel W, and the other is a color photosensitive pixel. 5 to 10 only show the arrangement of a plurality of photosensitive pixels 110 in a minimum repeating unit. The smallest repeating unit shown in FIGS. 5 to 10 is copied multiple times in rows and columns to form the pixel array 11. Each minimum repeating unit is composed of multiple full-color photosensitive pixels W and multiple color photosensitive pixels. Each minimum repeating unit includes multiple subunits. Each subunit includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels W. Among them, in the smallest repeating unit shown in FIGS. 5 to 8, the full-color photosensitive pixel W and the color photosensitive pixel in each sub-unit are alternately arranged. In the smallest repeating unit shown in FIGS. 9 and 10, in each sub-unit, multiple photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category; or, multiple photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category 110.

Specifically, for example, FIG. 5 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit of an embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

W represents the full-color photosensitive pixel; A represents the first color photosensitive pixel among the multiple color photosensitive pixels; B represents the second color photosensitive pixel among the multiple color photosensitive pixels; C represents the third color photosensitive pixel among the multiple color photosensitive pixels Pixels.

For example, as shown in FIG. 5, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

For example, as shown in Figure 5, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, a first type subunit UA and a third type subunit UC are arranged in the first diagonal direction D1 (for example, the direction connecting the upper left corner and the lower right corner in FIG. 5), and two second type subunits UB are arranged In the second diagonal direction D2 (for example, the direction where the upper right corner and the lower left corner are connected in FIG. 5). The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

It should be noted that in other embodiments, the first diagonal direction D1 may also be a direction connecting the upper right corner and the lower left corner, and the second diagonal direction D2 may also be a direction connecting the upper left corner and the lower right corner. In addition, the "direction" here is not a single direction, but can be understood as the concept of a "straight line" indicating the arrangement, and there may be two-way directions at both ends of the straight line. The explanation of the first diagonal direction D1 and the second diagonal direction D2 in FIGS. 6 to 10 is the same as here.

For another example, FIG. 6 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in a minimum repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 36 photosensitive pixels 110 in 6 rows and 6 columns, and the sub-units are 9 photosensitive pixels 110 in 3 rows and 3 columns. The arrangement method is:

For example, as shown in FIG. 6, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

For example, as shown in Figure 6, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1, and two second type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

For another example, FIG. 7 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the minimum repeating unit is 8 rows and 8 columns and 64 photosensitive pixels 110, and the sub-units are 4 rows and 4 columns and 16 photosensitive pixels 110. The arrangement method is:

For example, as shown in FIG. 7, for each sub-unit, full-color photosensitive pixels W and single-color photosensitive pixels are alternately arranged.

For example, as shown in Figure 7, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1, and two second type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

Specifically, for example, FIG. 8 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

The arrangement of the photosensitive pixels 110 in the smallest repeating unit shown in FIG. 8 is roughly the same as the arrangement of the photosensitive pixels 110 in the smallest repeating unit shown in FIG. The alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the subunit UB is inconsistent with the alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the second type of subunit UB in the lower left corner of FIG. 5, and , The alternating sequence of the full-color photosensitive pixel W and the single-color photosensitive pixel in the third type subunit UC in FIG. 8 is the same as the full-color photosensitive pixel W and the single-color photosensitive pixel W in the third type subunit UC in the lower right corner of FIG. 5 The alternating sequence of photosensitive pixels is also inconsistent. Specifically, in the second type subunit UB in the lower left corner of FIG. 5, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (ie, second-color photosensitive pixels B), and The alternating sequence of the two rows of photosensitive pixels 110 is single-color photosensitive pixels (ie, second-color photosensitive pixels B) and full-color photosensitive pixels W; and in the second-type subunit UB in the lower left corner of FIG. 8, the first row The alternating sequence of photosensitive pixels 110 is single-color photosensitive pixels (ie, second-color photosensitive pixels B), full-color photosensitive pixels W, and the alternating sequence of photosensitive pixels 110 in the second row is full-color photosensitive pixels W, single-color photosensitive pixels (ie The second color photosensitive pixel B). In the third type of subunit UC in the lower right corner of FIG. 5, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, third-color photosensitive pixels C), and the second row The alternating sequence of the photosensitive pixels 110 is a single-color photosensitive pixel (that is, a third-color photosensitive pixel C) and a full-color photosensitive pixel W; and in the third type subunit UC in the lower right corner of FIG. 8, the photosensitive pixels 110 in the first row The alternating sequence of the single-color photosensitive pixel (ie the third color photosensitive pixel C), the full-color photosensitive pixel W, the alternating sequence of the photosensitive pixel 110 in the second row is the full-color photosensitive pixel W, the single-color photosensitive pixel (ie the third color Photosensitive pixel C).

As shown in FIG. 8, the alternating sequence of full-color photosensitive pixels W and single-color photosensitive pixels in the first type subunit UA in FIG. The alternating sequence of pixels is not consistent. Specifically, in the first type of sub-unit CA shown in FIG. 8, the alternating sequence of the photosensitive pixels 110 in the first row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, first-color photosensitive pixels A), and the second row The alternating sequence of the photosensitive pixels 110 is a single-color photosensitive pixel (that is, the first color photosensitive pixel A), a full-color photosensitive pixel W; and in the third type of subunit CC shown in FIG. 8, the photosensitive pixels 110 in the first row The alternating sequence is single-color photosensitive pixels (that is, third-color photosensitive pixels C), full-color photosensitive pixels W, and the alternating sequence of photosensitive pixels 110 in the second row is full-color photosensitive pixels W, single-color photosensitive pixels (that is, third-color photosensitive pixels). Pixel C). That is to say, in the same minimum repeating unit, the alternating sequence of full-color photosensitive pixels W and color photosensitive pixels in different subunits can be the same (as shown in Figure 5) or inconsistent (as shown in Figure 8). Show).

For another example, FIG. 9 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

For example, as shown in FIG. 9, for each sub-unit, multiple photosensitive pixels 110 in the same row are photosensitive pixels 110 of the same category. The photosensitive pixels 110 of the same category include: (1) all full-color photosensitive pixels W; (2) all first-color photosensitive pixels A; (3) all second-color photosensitive pixels B; (4) all The third color photosensitive pixel C.

For example, as shown in Figure 9, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, one first type subunit UA and one third type subunit UC are arranged in the first diagonal direction D1, and two second type subunits UB are arranged in the second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

For another example, FIG. 10 is a schematic diagram of the arrangement of photosensitive pixels 110 (shown in FIG. 3) in the smallest repeating unit according to another embodiment of the application. Among them, the smallest repeating unit is 4 rows, 4 columns and 16 photosensitive pixels 110, and the sub-units are 2 rows, 2 columns and 4 photosensitive pixels 110. The arrangement method is:

For example, as shown in FIG. 10, for each sub-unit, a plurality of photosensitive pixels 110 in the same column are photosensitive pixels 110 of the same category. The photosensitive pixels 110 of the same category include: (1) all full-color photosensitive pixels W; (2) all first-color photosensitive pixels A; (3) all second-color photosensitive pixels B; (4) all The third color photosensitive pixel C.

For example, as shown in Figure 10, the categories of subunits include three categories. Wherein, the first type subunit UA includes a plurality of full-color photosensitive pixels W and a plurality of first color photosensitive pixels A; the second type of subunit UB includes a plurality of panchromatic photosensitive pixels W and a plurality of second-color photosensitive pixels B; The third type of subunit UC includes a plurality of full-color photosensitive pixels W and a plurality of third-color photosensitive pixels C. Each minimum repeating unit includes four subunits, which are one subunit of the first type UA, two subunits of the second type UB, and one subunit of the third type UC. Among them, a first type subunit UA and a third type subunit UC are arranged in a first diagonal direction D1, and two second type subunits UB are arranged in a second diagonal direction D2. The first diagonal direction D1 is different from the second diagonal direction D2. For example, the first diagonal line and the second diagonal line are perpendicular.

For example, in other embodiments, in the same minimum repeating unit, multiple photosensitive pixels 110 in the same row in some sub-units may be photosensitive pixels 110 of the same category, and multiple photosensitive pixels 110 in the same column in the remaining sub-units The pixels 110 are photosensitive pixels 110 of the same type.

For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a red photosensitive pixel R; the second color photosensitive pixel B may be a green photosensitive pixel G; and the third color photosensitive pixel C may be Blue photosensitive pixel Bu.

For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a red photosensitive pixel R; the second color photosensitive pixel B may be a yellow photosensitive pixel Y; and the third color photosensitive pixel C may be Blue photosensitive pixel Bu.

For example, in the smallest repeating unit shown in FIGS. 5 to 10, the first color photosensitive pixel A may be a magenta photosensitive pixel M; the second color photosensitive pixel B may be a cyan photosensitive pixel Cy; and the third color photosensitive pixel C may It is the yellow photosensitive pixel Y.

It should be noted that, in some embodiments, the response band of the full-color photosensitive pixel W may be the visible light band (for example, 400 nm-760 nm). For example, the full-color photosensitive pixel W is provided with an infrared filter to filter out infrared light. In other embodiments, the response wavelength bands of the full-color photosensitive pixel W are visible light and near-infrared wavelengths (for example, 400nm-1000nm), and the photoelectric conversion element 1111 (shown in FIG. 4) in the image sensor 10 (shown in FIG. 1) (Shown) to match the response band. For example, the full-color photosensitive pixel W may not be provided with a filter or a filter that can pass light of all wavelength bands. The response band of the full-color photosensitive pixel W is determined by the response band of the photoelectric conversion element 1111, that is, the two match. . The embodiments of the present application include, but are not limited to, the above-mentioned waveband range.

Referring to FIGS. 1 to 3, 5 and 11, in some embodiments, the control unit 13 controls the pixel array 11 to expose. Wherein, for a plurality of photosensitive pixels 110 in the same subunit, at least one single-color photosensitive pixel is exposed with a first exposure time, at least one single-color photosensitive pixel is exposed with a second exposure time less than the first exposure time, and at least one full-color photosensitive pixel is exposed The photosensitive pixel W is exposed at a third exposure time that is less than or equal to the first exposure time. The plurality of single-color photosensitive pixels exposed at the first exposure time in the pixel array 11 may generate first color information, and the plurality of single-color photosensitive pixels exposed at the second exposure time may generate second color information, which are exposed at the third exposure time. A plurality of panchromatic photosensitive pixels W can generate panchromatic information. The first color information may form a first color original image. The second color information can form a second color original image. Panchromatic information can generate a panchromatic original image.

In some embodiments, part of the panchromatic photosensitive pixels W in the same subunit is exposed at the fourth exposure time, and the remaining panchromatic photosensitive pixels W are exposed at the third exposure time. Wherein, the fourth exposure time is less than or equal to the first exposure time and greater than the third exposure time.

Wherein, the fourth exposure time is less than or equal to the first exposure time and greater than the third exposure time. Specifically, for the photosensitive pixels 110 (shown in FIG. 3) (shown in FIG. 3) in each sub-unit (four in FIG. 11), a single-color photosensitive pixel takes the first exposure time (for example, the long exposure time L shown in FIG. 11) Exposure, a single-color photosensitive pixel is exposed for the second exposure time (for example, the short exposure time S shown in FIG. 11), and a full-color photosensitive pixel W is exposed for the third exposure time (for example, the short exposure time S shown in FIG. 11) , One full-color photosensitive pixel W is exposed for the fourth exposure time (for example, the long exposure time L shown in FIG. 11).

It should be noted that, in some embodiments, the exposure process of the pixel array 11 may be: (1) the photosensitive pixels 110 exposed at the first exposure time, the photosensitive pixels 110 exposed at the second exposure time, and the third exposure time. The time-exposed photosensitive pixel 110 and the photosensitive pixel 110 exposed at the fourth exposure time are sequentially exposed (the exposure sequence of the four is not limited), and the exposure time of the four does not overlap; (2) the first exposure time The exposed photosensitive pixels 110, the photosensitive pixels 110 exposed at the second exposure time, the photosensitive pixels 110 exposed at the third exposure time, and the photosensitive pixels 110 exposed at the fourth exposure time are sequentially exposed (the exposure order of the four is not limited ), and there is a partial overlap in the exposure time of the four; (3) The exposure time of all the photosensitive pixels 110 exposed with a shorter exposure time is within the exposure time of the photosensitive pixels 110 exposed with the longest exposure time For example, the exposure time of all the single-color photosensitive pixels exposed at the second exposure time is within the exposure time of all the single-color photosensitive pixels exposed at the first exposure time, and all the full-color photosensitive pixels exposed at the third exposure time The exposure time of the pixel W is within the exposure time of all single-color photosensitive pixels exposed at the first exposure time, and the exposure time of all the full-color photosensitive pixels W exposed at the fourth exposure time is within the first exposure time The exposure time of all single-color photosensitive pixels exposed. In the specific embodiment of the present application, the image sensor 10 adopts the (3) exposure method, which can shorten the overall exposure time required by the pixel array 11, which is beneficial to increase the frame rate of the image.

After the pixel array 11 is exposed, the image sensor 10 can output four original images, which are: (1) The first color original image, which is generated by multiple single-color photosensitive pixels exposed with a long exposure time L (first exposure time) (2) The second color original image is composed of the second color information generated by multiple single-color photosensitive pixels exposed with a short exposure time S (second exposure time); (3) The first full The color original image is composed of the first panchromatic information generated by multiple panchromatic photosensitive pixels W (third exposure time) exposed with a short exposure time S; (4) The second panchromatic original image is composed of a long exposure time L (Fourth Exposure Time) It is composed of second panchromatic information generated by a plurality of panchromatic photosensitive pixels W exposed.

Referring to FIGS. 11 and 16, after the image sensor 10 obtains the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image, it will transmit these four images to the image fusion Module 20. The image fusion module 20 performs fusion processing on the first color original image and the second panchromatic original image to obtain a first intermediate image, and performs fusion processing on the second color original image and the first panchromatic original image to obtain the first intermediate image. Two intermediate images.

Taking the first color original image as an example, as shown in FIG. 12 and FIG. 16, the image fusion module 20 first separates the color and brightness of the first color original image to obtain a color-brightness separated image. LIT stands for brightness and CLR stands for color. Specifically, assuming that the single-color photosensitive pixel A is a red photosensitive pixel R, the single-color photosensitive pixel B is a green photosensitive pixel G, and the single-color photosensitive pixel C is a blue photosensitive pixel Bu, then: (1) The image fusion module 20 can combine RGB The first color original image in the space is converted into a color-brightness separated image in YCrCb space. At this time, Y in YCrCb is the brightness LIT in the color-brightness separated image, and Cr and Cb in YCrCb are the color CLR in the color-brightness separated image. (2) The image fusion module 20 can also convert the first color original image of RGB into a color-light-separated image in Lab space. At this time, L in Lab is the brightness LIT in the color-light-separated image, and a and a in Lab b is the color CLR in the color-light-separated image. It should be noted that the LIT+CLR in the color-light separation image shown in FIG. 12 does not mean that the pixel value of each pixel is formed by adding L and CLR, but only that the pixel value of each pixel is composed of LIT and CLR.

Subsequently, the image fusion module 20 fuses the brightness of the color-brightness separated image and the brightness of the second full-color original image. For example, the pixel value of each panchromatic pixel W in the second panchromatic original image is the brightness value of each panchromatic pixel, and the image fusion module 20 can separate the LIT of each pixel in the color-brightness image with the panchromatic pixel. The W of the panchromatic pixel at the corresponding position in the intermediate image is added to obtain the pixel value after brightness correction. The image fusion module 20 forms a brightness-corrected color-brightness separated image according to a plurality of brightness-corrected pixel values, and then uses color space conversion to convert the brightness-corrected color-brightness separated image into a first intermediate image.

Similarly, referring to Figs. 13 and 16, the image fusion module 20 performs fusion processing on the second color original image and the first panchromatic original image to obtain a second intermediate image. The acquisition process of the second intermediate image is the same as the acquisition process of the first intermediate image, and will not be repeated here. Of course, the image fusion module 20 can also use other methods to perform fusion processing, which is not limited here. The fusion processing of the color original image and the full-color original image by the image fusion module 20 can increase the brightness of the intermediate image obtained after fusion.

It should be noted that since the first color original image is composed of first color information generated by multiple single-color photosensitive pixels exposed with a long exposure time L, and the second full-color original image is also exposed with a long exposure time L The second panchromatic information generated by a plurality of panchromatic photosensitive pixels W is composed, so the exposure time corresponding to all the image pixels in the first intermediate image obtained by the fusion process of the first color original image and the second panchromatic original image is Long exposure time L. Similarly, since the second color original image is composed of second color information generated by a plurality of single-color photosensitive pixels exposed with a short exposure time S, and the first full-color original image is also composed of a plurality of single-color photosensitive pixels exposed with a short exposure time S The first panchromatic information generated by the panchromatic photosensitive pixel W is composed, so the exposure time corresponding to all image pixels in the second intermediate image obtained by the fusion process of the second color original image and the first panchromatic original image is short exposure Time S.

After the image fusion module 20 obtains the first intermediate image and the second intermediate image, the two images are transmitted to the high dynamic range image processing module 30 for high dynamic processing to obtain the first high dynamic range image. For example, referring to FIG. 16, the high dynamic range image processing module 30 includes a high dynamic range image processing unit 31 and a brightness mapping unit 33. The high dynamic range image processing unit 31 is used to fuse the first intermediate image and the second intermediate image into a third high dynamic range image; the brightness mapping unit 33 is used to perform brightness mapping on the third high dynamic range image to obtain the first high dynamic range image Range image.

Specifically, referring to FIG. 16, the process of the high dynamic range image processing unit 31 fusing the first intermediate image and the second intermediate image may include brightness alignment processing. The high dynamic range image processing unit 31 performing brightness alignment processing on the first intermediate image and the second intermediate image includes the following steps: (1) identifying overexposed image pixels in the first intermediate image with pixel values greater than a first preset threshold; (2) ) For each overexposed image pixel, expand the predetermined area with the overexposed image pixel as the center; (3) Find the intermediate image pixel with the pixel value less than the first preset threshold in the predetermined area; (4) Use the intermediate image pixel and The second intermediate image corrects the pixel values of the pixels of the overexposed image; (5) the first intermediate image is updated with the corrected pixel values of the pixels of the overexposed image to obtain the first intermediate image with the brightness aligned. Specifically, please refer to Figure 14, assuming that the pixel value V1 of the image pixel P12 (the image pixel marked with a dashed circle in the first intermediate image in Figure 14) is greater than the first preset threshold V0, that is, the image pixel P12 is an overexposed image pixel P12, the high dynamic range image processing unit 31 expands a predetermined area with the overexposed image pixel P12 as the center, for example, the 3*3 area shown in FIG. 14. Of course, in other embodiments, it may also be a 4*4 area, a 5*5 area, a 10*10 area, etc., which is not limited here. Subsequently, the high dynamic range image processing unit 31 searches for an intermediate image pixel with a pixel value less than the first preset threshold V0 in a predetermined area of 3*3, such as image pixel P21 in FIG. 14 (marked in the first intermediate image in FIG. 14). If the pixel value V2 of the image pixel with a dotted circle is less than the first preset threshold V0, the image pixel P21 is the intermediate image pixel P21. Subsequently, the high dynamic range image processing unit 31 searches the second intermediate image for image pixels corresponding to the overexposed image pixel P12 and the intermediate image pixel P21 respectively, that is, the image pixel P1'2' (marked in the second intermediate image in FIG. 14). The image pixel with a dotted circle) and the image pixel P2'1' (the image pixel marked with a dotted circle in the second intermediate image in Figure 14), where the image pixel P1'2' corresponds to the overexposed image pixel P12, the image The pixel P2'1' corresponds to the intermediate image pixel P21, the pixel value of the image pixel P1'2' is V3, and the pixel value of the image pixel P2'1' is V4. Then, V1' is calculated according to V1'/V3=V2/V4, and the value of V1' is used to replace the value of V1. Thus, the actual pixel value of the overexposed image pixel P12 can be calculated. The high dynamic range image processing unit 31 performs this brightness alignment process on each overexposed image pixel in the first intermediate image to obtain the first intermediate image after brightness alignment. Since the pixel value of the overexposed image pixel in the first intermediate image after brightness alignment is corrected, the pixel value of each image pixel in the first intermediate image after brightness alignment is relatively accurate.

In the high dynamic range processing process, after acquiring the first intermediate image and the second intermediate image with the brightness aligned, the high dynamic range image processing unit 31 may fuse the first intermediate image and the second intermediate image with the brightness aligned to obtain The third high dynamic color image. Referring to FIG. 15, specifically, the high dynamic range image processing unit 31 first performs motion detection on the first intermediate image after brightness alignment to identify whether there is a motion blur area in the first intermediate image after brightness alignment. If there is no motion blur area in the first intermediate image after brightness alignment, the first intermediate image and the second intermediate image after brightness alignment are directly merged to obtain the first high dynamic range image. If there is a motion blur area in the first intermediate image after brightness alignment, the motion blur area in the first intermediate image is eliminated, and only all areas of the second intermediate image are merged and the first intermediate image after brightness alignment is removed. The area outside the area to obtain the first high dynamic range image. Specifically, when fusing the first intermediate image and the second intermediate image after brightness alignment, if there is no motion blur area in the first intermediate image after brightness alignment, the fusion of the two intermediate images at this time follows the following principles: ( 1) In the first intermediate image after brightness alignment, the pixel value of the image pixel in the overexposed area is directly replaced with the pixel value of the image pixel corresponding to the overexposed area in the second intermediate image; (2) the first intermediate image after brightness alignment In an intermediate image, the pixel value of the image pixel in the under-exposed area is: the long-exposure pixel value divided by the ratio of the long-short pixel value; (3) the first intermediate image after brightness alignment, the image in the area that is neither under-exposed nor over-exposed The pixel value of a pixel: the long-exposure pixel value divided by the ratio of the long-short pixel value. If there is a motion blur area in the first intermediate image after brightness alignment, the fusion of the two intermediate images at this time must follow the above three principles, and also need to follow the (4) principle: the first intermediate image after brightness alignment , The pixel value of the image pixel in the motion blur area is directly replaced with the pixel value of the image pixel in the second intermediate image corresponding to the motion blur area. It should be noted that for under-exposed areas and areas that are neither under-exposed nor over-exposed, the pixel value of the image pixels in these areas is the long-exposure pixel value divided by the ratio of the long-short pixel value, that is, VL/(VL/VS) =VS', where VL represents the long-exposure pixel value, VS represents the short-exposure pixel value, and VS' represents the calculated pixel value of the image pixel in the under-exposed area and the neither under-exposed nor over-exposed area. The signal-to-noise ratio of VS’ will be greater than the signal-to-noise ratio of VS.

By performing high dynamic range processing on the intermediate image by the high dynamic range image processing unit 31, the dynamic range of the obtained image can be increased, and the imaging effect of the image can be improved.

Of course, the high dynamic range image processing unit 31 may also use other methods to fuse the first intermediate image and the second intermediate image with the brightness aligned to obtain the third high dynamic color image. For example, the high dynamic range image processing unit 31 may also perform motion blur detection on the first intermediate image and the second intermediate image after the brightness is aligned, and perform motion blur detection on the detected first intermediate image and the second intermediate image. The motion blur is eliminated to obtain the first intermediate image after the motion blur is eliminated and the second intermediate image after the motion blur is eliminated. After the high dynamic range image processing unit 31 obtains the first intermediate image after removing the motion blur and the second intermediate image after removing the motion blur, the first intermediate image after removing the motion blur and the second intermediate image after removing the motion blur are processed again. The intermediate images are fused to obtain the third high dynamic range image with high dynamic range, which is not limited here.

After obtaining the third high dynamic range image, the high dynamic range image processing unit 31 transmits the third high dynamic range image to the brightness mapping unit 33. The brightness mapping unit 33 subjects the third high dynamic range image to brightness mapping processing to obtain the first high dynamic range image. Wherein, the bit width of the data of each image pixel in the first high dynamic range image is smaller than the bit width of the data of each image pixel in the third high dynamic range image.

For example, after the first intermediate image and the second intermediate image with a data bit width of 10 bits undergo high dynamic range processing by the high dynamic range image processing unit 31, a third high dynamic range image with a bit width of 16 bits can be obtained. The brightness mapping unit 33 may perform brightness mapping processing on the third high dynamic range image with a bit width of 16 bits to obtain the first high dynamic range image with a bit width of 10 bits. Of course, in some embodiments, the third high dynamic range image with a bit width of 16 bits may also be subjected to brightness mapping processing to obtain the first high dynamic range image with a bit width of 12 bits, which is not limited here. In this way, the data volume of the high dynamic range image is reduced through the brightness mapping process, thereby avoiding the problem that the image processor 40 cannot process the high dynamic range image with excessive data volume, and is beneficial to improve the image processor 40 processing high dynamic range image. speed.

The high dynamic range image processing unit 31 can transmit the first high dynamic range image to the image processor 40 for subsequent processing such as black level, demosaicing, color conversion, lens shading correction, dead pixel compensation, global tone mapping, etc., to obtain the first high dynamic range image. 2. High dynamic range image. The multiple color image pixels in the first high dynamic range image are all arranged in a Bayer array, and the pixel value of each image pixel only contains the information of one color channel, while the pixels of each image pixel in the second high dynamic range image The values all contain the information of each color channel.

Referring to FIG. 16, the high dynamic range image processing module 30 further includes a statistical unit 35, which is used to process the first intermediate image and the second intermediate image to obtain statistical data. After obtaining the statistical data, the statistical unit 35 provides the statistical data to the image processor 40 for automatic exposure processing and/or automatic white balance processing. In other words, after receiving the statistical data, the image processor 40 can perform at least one of automatic exposure processing and automatic white balance processing according to the statistical data. For example, the image processor 40 performs automatic exposure processing based on statistical data; or, the image processor 40 performs automatic white balance processing based on statistical data; or, the image processor 40 performs automatic exposure processing and automatic white balance processing based on statistical data. In this way, the image processor 40 can perform automatic exposure and automatic white balance processing according to the statistical data, which is beneficial to improve the quality of the image finally output by the image processor 40.

Referring to FIG. 16, the high dynamic range image processing module 30 further includes a lens shading correction unit 37, which is used to correct the third high dynamic range image to obtain a high dynamic range corrected image. Specifically, after the high dynamic range image processing unit 31 fuses the first intermediate image and the second intermediate image into a third high dynamic range image, the lens shading correction unit 37 performs lens shading correction processing on the third high dynamic range image to obtain a high dynamic range image. Dynamic range correction image. The specific process of lens shading correction processing is shown in Figure 17. The lens shading correction unit 37 divides the third high dynamic range image into sixteen grids equally, and each of the sixteen grids has one The preset compensation coefficient. Then, the lens shading correction unit 37 performs shading correction on the image by the bilinear interpolation method according to the compensation effect of each grid area adjacent or itself and its vicinity. R2 is the pixel value in the dotted frame in the third high dynamic range image that has undergone lens shading correction processing, and R1 is the pixel value in the dotted frame in the first color original image shown in the figure. R2=R1*k1, k1 is obtained by bilinear interpolation of the compensation coefficients 1.10, 1.04, 1.05, and 1.09 of the grid adjacent to the R1 pixel. Suppose the coordinates of the image are (x, y), x is counted from the first pixel from the left to the right, y is counted from the first pixel on the top, and both x and y are natural numbers, as indicated by the logo on the edge of the image Show. For example, if the coordinates of R1 are (3,3), the coordinates of R1 in each grid compensation coefficient map should be (0.75,0.75). f(x, y) represents the compensation value of the coordinate (x, y) in each grid compensation coefficient graph. Then f(0.75,j0.75) is the compensation coefficient value corresponding to R1 in each grid compensation coefficient graph, then f(0.75,0.75)=(0.25)*(0.25)*f(0,0)+0.25 *0.75*f(0,1)+0.75*0.25*f(1,0)+0.75*0.75f(1,1)=0.0625*1.11+0.1875*1.10+0.1875*1.09+0.5625*1.03. The compensation coefficient of each grid is set in advance before the lens shading correction unit 37 performs lens shading correction processing.

After obtaining the high dynamic range correction image, the lens shading correction unit 37 transmits the high dynamic range correction image to the statistics unit 35. The statistical unit 35 is configured to process the high dynamic range correction image to obtain statistical data, and provide the statistical data to the image processor 40 for automatic exposure processing and/or automatic white balance processing, that is, the statistical data is provided to the image processor 40 To perform at least one of automatic exposure processing and automatic white balance processing.

Since the third high dynamic range image is first subjected to lens shading correction, and then the high dynamic range correction image after the shadow correction is processed to obtain statistical data, the influence of lens shadow is avoided, so that the image processor 40 performs automatic exposure based on the statistical data The quality of the image obtained by processing and/or automatic white balance processing is higher. It should be noted that both the image fusion module 20 and the high dynamic range processing module 30 are integrated in the image sensor 10.

In summary, the high dynamic range image processing system 100 shown in FIG. 16 first fuses the color original image and the panchromatic original image through the image fusion module 20 to obtain the first intermediate image and the second intermediate image, and then passes the high dynamic range image The processing module 30 performs high dynamic range processing on the first intermediate image and the second intermediate image to obtain the first high dynamic range image. Since the multiple color image pixels in the first high dynamic range image are arranged in a Bayer array, the first high dynamic range image can be directly processed by the image processor 40.

In other embodiments, referring to FIG. 18, when the image sensor 10 obtains the first color original image, the second color original image, the first full color original image, and the second full color original image, the four images The image is transmitted to the high dynamic range image processing module 30. The high dynamic range image processing module 30 fuses the first color original image and the second color original image into a first high dynamic color original image, and combines the first full color original image with the second full color original image. The color original image is fused into the first high dynamic panchromatic original image. Subsequently, the high dynamic range image processing module 30 transmits the first high dynamic full color original image and the first high dynamic color original image to the image fusion module 20 for fusion processing to finally obtain the first high dynamic range image.

Specifically, referring to FIGS. 1, 11, 18, and 19, after the image sensor 10 obtains the first color original image, the second color original image, the first panchromatic original image, and the second panchromatic original image, it will These four images are transmitted to the high dynamic range image processing module 30, and the high dynamic range image processing unit 31 in the high dynamic range image processing module 30 fuses the first color original image and the second color original image into a second high dynamic color original image Image, fusing the first panchromatic original image and the second panchromatic original image into a second high dynamic panchromatic original image. The specific fusion process is the same as the specific process of fusing the first intermediate image and the second intermediate image into the third high dynamic range image in the embodiment shown in FIG. 16, and will not be repeated here.

The brightness mapping unit 33 is configured to perform brightness mapping on the second high dynamic color original image to obtain a first high dynamic color original image with a small amount of data, and perform brightness mapping on the second high dynamic full color original image to obtain a small amount of data The first high dynamic panchromatic original image. The specific brightness mapping process is the same as the specific process of mapping the brightness of the third high dynamic range image to the first high dynamic range image in the embodiment shown in FIG. 16, and will not be repeated here.

The lens shading correction unit 37 is used to correct the second high dynamic color original image to obtain a high dynamic color corrected image, and correct the second high dynamic panchromatic original image to obtain a high dynamic full color corrected image. The specific correction process is the same as the process of performing lens shading correction on the third high dynamic range image in the embodiment shown in FIG. 16 and FIG. 17, and will not be repeated here.

The statistical unit 35 is used to process the high dynamic color correction image and the high dynamic full color correction image to obtain statistical data, and transmit the statistical data to the image processor 40, so that the image processor 40 can perform automatic exposure and automatic exposure based on the statistical information. At least one of the white balance processing. Of course, the statistical unit 35 can also directly process the first color original image, the second color original image, the first full-color original image, and the second full-color original image to obtain statistical data, and transmit the statistical data to the image processor 40 , So that the image processor 40 can perform at least one of automatic exposure and automatic white balance processing according to the statistical information.

After the high dynamic range image processing module 30 obtains the first high dynamic color original image and the first high dynamic full color original image, the two images are transmitted to the image fusion module 20 for fusion processing to obtain the first high dynamic range image. The specific process of the image fusion module 20 fusing the first high dynamic color original image and the first high dynamic panchromatic original image into the first high dynamic range image is the same as in the embodiment shown in FIG. 12, combining the first color original image and the second high dynamic range image. The specific fusion process of fusion of the panchromatic original image into the first intermediate image is the same, and will not be repeated here.

In summary, the high dynamic range image processing system 100 shown in FIG. 18 first uses the high dynamic range image processing module 30 to fuse the color original image and the full color original image to obtain the first high dynamic color original image and the first high dynamic range image processing module 30. Then, the first high-dynamic color original image and the first high-dynamic panchromatic original image are merged by the image fusion module 20 to obtain the first high-dynamic range image. Since the multiple color image pixels in the first high dynamic range image are arranged in a Bayer array, the first high dynamic range image can be directly processed by the image processor 40.

In still other embodiments, as shown in FIG. 20, all panchromatic photosensitive pixels W in the pixel array 11 are exposed at the third exposure time, and the third exposure time may be greater than the second exposure time, so that all panchromatic photosensitive pixels W All are exposed with the medium exposure time M; or, the third exposure time is equal to the first exposure time, so that all the panchromatic photosensitive pixels W are exposed with the long exposure time L. Of course, the third exposure time can also be equal to or less than the second exposure time, Exposure of the full-color photosensitive pixel W with a short exposure time is not limited here. In the following description, the third exposure time is greater than the second exposure time, that is, all the full-color photosensitive pixels W are exposed with the medium exposure time M as an example. Specifically, for a plurality of (4 shown in FIG. 20) photosensitive pixels 110 (shown in FIG. 3) in each subunit, a single-color photosensitive pixel has a first exposure time (for example, the long exposure time shown in FIG. 20). L) Exposure. One single-color photosensitive pixel is exposed for the second exposure time (for example, the short exposure time S shown in FIG. 20), and the two full-color photosensitive pixels W are both exposed for the third exposure time (for example, the medium exposure time shown in FIG. 20). Time M) Exposure.

It should be noted that, in some embodiments, the exposure process of the pixel array 11 may be: (1) the photosensitive pixel 110 exposed at the first exposure time, the photosensitive pixel 110 exposed at the second exposure time, and the third exposure The time-exposed photosensitive pixels 110 are sequentially exposed (the exposure sequence of the three is not limited), and the exposure time of the three does not overlap; (2) the photosensitive pixels 110 exposed at the first exposure time are exposed at the second exposure time The exposed photosensitive pixels 110 and the photosensitive pixels 110 exposed at the third exposure time are sequentially exposed (the exposure sequence of the three is not limited), and the exposure time of the three overlaps partially; (3) All exposures are shorter The exposure time of the time-exposed photosensitive pixels 110 are all within the exposure time of the photosensitive pixels 110 that are exposed with the longest exposure time. For example, the exposure time of all the single-color photosensitive pixels exposed at the second exposure time are all within the time The exposure time of all the single-color photosensitive pixels exposed at the first exposure time is within the exposure time of all the full-color photosensitive pixels W exposed at the third exposure time are within the exposure time of all the single-color photosensitive pixels exposed at the first exposure time Within time. In the specific embodiment of the present application, the pixel array 11 adopts the (3) exposure method, which can shorten the overall exposure time required by the pixel array 11, which is beneficial to increase the frame rate of the image.

After the pixel array 11 is exposed, the image sensor 10 can output three original images, which are: (1) The first color original image, which is generated by multiple single-color photosensitive pixels exposed with a long exposure time L (first exposure time) (2) The second color original image is composed of the second color information generated by multiple single-color photosensitive pixels exposed with a short exposure time S (second exposure time); (3) The first full The color original image is composed of first panchromatic information generated by a plurality of panchromatic photosensitive pixels W exposed at a medium exposure time M (third exposure time).

Referring to FIGS. 18 and 20, the image sensor 10 first transmits the first color original image and the second color original image to the high dynamic range image processing module 30 for high dynamic range processing to obtain the first high dynamic color original image, and then transfers the first color original image to the high dynamic range image processing module 30. A high dynamic color original image and a first panchromatic original image are transmitted to the image fusion module 20 for fusion algorithm processing to obtain the first high dynamic range image.

Specifically, referring to FIG. 21, the image sensor 10 transmits the first color original image, the second color original image, and the first panchromatic to the high dynamic range image processing module 30. The high dynamic range image in the high dynamic range image processing module 30 The processing unit 31 fuses the first color original image and the second color original image into a second high-dynamic color original image. The specific fusion process is the same as that of the first intermediate image and the second intermediate image in the embodiment shown in FIG. 15 The specific process of the three high dynamic range images is the same, so I won’t repeat them here.

The brightness mapping unit 33 is configured to perform brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image with a small amount of data. The specific process is the same as the specific process of mapping the brightness of the third high dynamic range image to the first high dynamic range image in the embodiment shown in FIG. 16, and will not be repeated here.

The lens shading correction unit 37 is used to correct the second high dynamic color original image to obtain a high dynamic color corrected image. The specific correction process is the same as the process of performing lens shading correction on the third high dynamic range image in the embodiment shown in FIGS. 16 and 17. I will not repeat them here.

The statistical unit 35 is used to process the high dynamic color correction image and the high dynamic full color correction image to obtain statistical data, and transmit the statistical data to the image processor 40, so that the image processor 40 can perform automatic exposure and automatic exposure based on the statistical information. At least one of the white balance processing. Of course, the statistical unit 35 can also directly process the first color original image, the second color original image, and the first full-color original image to obtain statistical data, and transmit the statistical data to the image processor 40 so that the image processor 40 can At least one of automatic exposure and automatic white balance processing is performed based on the statistical information.

After obtaining the first high dynamic color original image, the high dynamic range image processing module 30 transmits the first high dynamic color original image and the first panchromatic original image to the image fusion module 20 for fusion processing to obtain the first high dynamic range image. Specifically, referring to FIGS. 20 and 22, the first panchromatic original image obtained by the image sensor 10 includes a plurality of panchromatic image pixels W and a plurality of empty image pixels N (NULL), wherein the empty image pixels are neither complete Color image pixels are also not color image pixels. The position of the empty image pixel N in the first full-color original image can be regarded as there is no image pixel in that position, or the pixel value of the empty image pixel can be regarded as zero. Comparing the pixel array 11 with the full-color original image, it can be seen that for each sub-unit in the pixel array 11, the sub-unit includes two full-color image pixels W and two color image pixels (color image pixel A, color image pixel B, Or color image pixel C). The first full-color original image also has a sub-unit corresponding to each sub-unit in the pixel array 11, and the sub-unit of the first full-color original image includes two full-color image pixels W and two empty image pixels N, The positions of the two empty image pixels N correspond to the positions of the two color image pixels in the subunit of the pixel array 11.

The image fusion module 20 may further process the first full-color original image to obtain a full-color intermediate image. Illustratively, each sub-unit includes a plurality of empty image pixels N and a plurality of panchromatic image pixels. Specifically, some sub-units include two empty image pixels N and two panchromatic image pixels W. The image fusion module 20 may use the pixel values of all panchromatic image pixels in the subunit including the empty image pixel N and the panchromatic image pixel W as the panchromatic large pixel W in the subunit to obtain a panchromatic intermediate image. The resolution of the panchromatic intermediate image at this time is the same as the resolution of the first high dynamic color original image, so as to facilitate the fusion of the panchromatic intermediate image and the first high dynamic color original image. The specific fusion process of the panchromatic intermediate image and the first high dynamic color original image is the same as the specific fusion process of fusing the first color original image and the second panchromatic original image into the first intermediate image in the embodiment shown in FIG. 12. I will not repeat them here.

In summary, the high dynamic range image processing system 100 shown in FIG. 18 first fused the color original image and the panchromatic original image through the high dynamic range image processing module 30 to obtain the first high dynamic color original image, and then passed the image fusion module 20 The first high dynamic color original image and the first panchromatic original image are merged to obtain the first high dynamic range image. Since the multiple color image pixels in the first high dynamic range image are arranged in a Bayer array, the first high dynamic range image can be directly processed by the image processor 40.

Please refer to FIG. 1 and FIG. 23. The present application also provides an electronic device 1000. The electronic device 1000 of the embodiment of the present application includes a lens 300, a housing 200, and the high dynamic range image processing system 100 described in any one of the above embodiments. The lens 300 and the high dynamic range image processing system 100 are combined with the housing 200. The lens 300 cooperates with the image sensor 10 of the high dynamic range image processing system 100 for imaging.

The electronic device 1000 may be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, a smart glasses, a smart helmet), a drone, a head-mounted display device, etc., which are not limited here.

The electronic device 1000 of the embodiment of the present application performs fusion algorithm processing and high dynamic range processing on the full-color original image and the color original image output by the image sensor 10 through the image fusion module 20 and the high dynamic range image processing module 30 to obtain image pixels. The first high dynamic range image arranged in a Bayer array, and then the first high dynamic range image is input to the image processor for subsequent processing, thereby solving the problem that the image processor 40 cannot directly perform processing on the image with image pixels arranged in a non-Bayer array. Handling issues.

The specific implementation process of the high dynamic range image processing method of any one of the foregoing embodiments is the same as the specific implementation process of the aforementioned high dynamic range image processing system 100 to obtain a high dynamic range image, and will not be further described here.

Please refer to FIG. 25. The present application also provides a non-volatile computer-readable storage medium 400 containing a computer program. When the computer program is executed by the processor 60, the processor 60 is caused to execute the high dynamic range image processing method described in any one of the foregoing embodiments.

For example, referring to FIG. 1, FIG. 3, FIG. 11, and FIG. 25, when the computer program is executed by the processor 60, the processor 60 is caused to perform the following steps:

The pixel array 11 is exposed, wherein, for a plurality of photosensitive pixels in the same subunit, at least one single-color photosensitive pixel is exposed with a first exposure time, and at least one single-color photosensitive pixel is exposed with a second exposure time that is less than the first exposure time, At least one full-color photosensitive pixel is exposed at a third exposure time that is less than the first exposure time; wherein, the first color information generated by the single-color photosensitive pixel exposed at the first exposure time obtains the first color original image, and the first color original image is obtained at the second exposure time. The second color information generated by the exposed single-color photosensitive pixels obtains a second color original image, and the panchromatic photosensitive pixels exposed at the third exposure time generate a first panchromatic original image; and

Perform fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain a first high dynamic range image, the first high dynamic range image including a plurality of color image pixels, A plurality of color image pixels are arranged in a Bayer array, and the first high dynamic range image is processed by the image processor to obtain a second high dynamic range image.

For another example, please refer to FIG. 25. When the computer program is executed by the processor 60, the processor 60 is caused to perform the following steps:

Fusing the first color original image and the second color original image into a first high dynamic color original image; and

The first high dynamic color original image and the first panchromatic original image are fused into a first high dynamic range image.

In the description of this specification, the description with reference to the terms “one embodiment”, “some embodiments”, “exemplary embodiments”, “examples”, “specific examples” or “some examples” etc. means to combine the described implementations The specific features, structures, materials, or characteristics described by the manners or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above-mentioned terms does not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples and the features of the different embodiments or examples described in this specification without contradicting each other.

Any process or method description described in the flowchart or described in other ways herein can be understood as a module, segment, or part of code that includes one or more executable instructions for implementing specific logical functions or steps of the process , And the scope of the preferred embodiments of the present application includes additional implementations, which may not be in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. This should It is understood by those skilled in the art to which the embodiments of the present application belong.

Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application. Those of ordinary skill in the art can comment on the above within the scope of the present application. The implementation mode undergoes changes, modifications, replacements and modifications.

Claims

A high dynamic range image processing system, characterized in that it includes an image sensor, an image fusion module, and a high dynamic range image processing module;

The image sensor includes a pixel array, the pixel array includes a plurality of panchromatic photosensitive pixels and a plurality of color photosensitive pixels, the color photosensitive pixels have a narrower spectral response than the panchromatic photosensitive pixels, and the pixel array includes The smallest repeating unit, each of the smallest repeating units includes a plurality of sub-units, each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels, the pixel array in the image sensor is exposed, wherein, for In the multiple photosensitive pixels in the same subunit, at least one of the single-color photosensitive pixels is exposed with a first exposure time, and at least one of the single-color photosensitive pixels is exposed with a second exposure time that is less than the first exposure time, At least one of the full-color photosensitive pixels is exposed at a third exposure time that is less than the first exposure time; wherein, the first color information generated by the single-color photosensitive pixels exposed at the first exposure time obtains a first color The original image, the second color information generated by the single-color photosensitive pixels exposed at the second exposure time to obtain a second color original image, and the full-color photosensitive pixels exposed at the third exposure time generate a first full Color original image;

The image fusion module and the high dynamic range image processing module are used to perform fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image To obtain a first high dynamic range image, the first high dynamic range image includes a plurality of color image pixels, the plurality of color image pixels are arranged in a Bayer array, and the first high dynamic range image is processed by an image processor To get the second high dynamic range image.
The high dynamic range image processing system according to claim 1, wherein some of the full-color photosensitive pixels in the same subunit are exposed with a fourth exposure time, and the remaining full-color photosensitive pixels are exposed with the first Three exposure time exposure, the fourth exposure time is less than or equal to the first exposure time and greater than the third exposure time;

The image fusion module is used for fusing the first color original image and the second full-color original image into a first intermediate image, and fusing the second color original image and the first full-color original image into a second An intermediate image, the second panchromatic information generated by the single-color photosensitive pixels exposed at the fourth exposure time to obtain the second panchromatic original image;

The high dynamic range image processing module is used for fusing the first intermediate image and the second intermediate image into the first high dynamic range image.
The high dynamic range image processing system according to claim 2, wherein the high dynamic range image processing module comprises a high dynamic range image processing unit and a brightness mapping unit;

The high dynamic range image processing unit is configured to merge the first intermediate image and the second intermediate image into a third high dynamic range image;

The brightness mapping unit is configured to perform brightness mapping on the third high dynamic range image to obtain the first high dynamic range image.
The high dynamic range image processing system according to claim 2, wherein the high dynamic range image processing module includes a high dynamic range image processing unit, a lens shading correction unit, and a statistical unit;

The high dynamic range image processing unit is configured to merge the first intermediate image and the second intermediate image into a third high dynamic range image;

The lens shading correction unit is used to correct the third high dynamic range image to obtain a high dynamic range corrected image;

The statistical unit is configured to process the high dynamic range correction image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.
The high dynamic range image processing system according to claim 2, wherein the high dynamic range image processing module comprises a statistical unit, and the statistical unit is used to process the first intermediate image and the second intermediate image To obtain statistical data, the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.
The high dynamic range image processing system according to claim 1, wherein some of the full-color photosensitive pixels in the same subunit are exposed with a fourth exposure time, and the remaining full-color photosensitive pixels are exposed with the first Three exposure time exposure, the fourth exposure time is less than or equal to the first exposure time and greater than the third exposure time;

The high dynamic range image processing module is used to fuse the first color original image and the second color original image into a first high dynamic color original image, and combine the first full-color original image with the second color original image. The panchromatic original image is fused into the first high dynamic panchromatic original image;

The image fusion module is used for fusing the first high dynamic color original image and the first high dynamic full color original image into the first high dynamic range image.
The high dynamic range image processing system according to claim 6, wherein the high dynamic range image processing module further comprises a high dynamic range image processing unit and a brightness mapping unit;

The high dynamic range image processing unit is configured to merge the first color original image and the second color original image into a second high dynamic color original image, and combine the first full-color original image with the second color original image. The panchromatic original image is fused into the second high dynamic panchromatic original image;

The brightness mapping unit is configured to perform brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image, and perform brightness mapping on the second high dynamic full color original image to obtain the The first high dynamic panchromatic original image.
The high dynamic range image processing system according to claim 6, wherein the high dynamic range image processing module further comprises a high dynamic range image processing unit, a lens shading correction unit, and a statistical unit;

The high dynamic range image processing unit is configured to merge the first color original image and the second color original image into a second high dynamic color original image, and combine the first full-color original image with the second color original image. The panchromatic original image is fused into the second high dynamic panchromatic original image;

The lens shading correction unit is configured to correct the second high dynamic color original image to obtain a high dynamic color correction image, and correct the second high dynamic panchromatic original image to obtain a high dynamic panchromatic correction image;

The statistical unit is configured to process the high dynamic color correction image and the high dynamic panchromatic correction image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance deal with.
The high dynamic range image processing system according to claim 1, wherein all the panchromatic photosensitive pixels in the same subunit are exposed at a third exposure time;

The high dynamic range image processing module is configured to fuse the first color original image and the second color original image into a first high dynamic color original image;

The image fusion module is used for fusing the first high dynamic color original image and the first panchromatic original image into the first high dynamic range image.
The high dynamic range image processing system according to claim 9, wherein the high dynamic range image processing module further comprises a high dynamic range image processing unit and a brightness mapping unit;

The high dynamic range image processing unit is configured to fuse the first color original image and the second color original image into a second high dynamic color original image;

The brightness mapping unit is configured to perform brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image.
The high dynamic range image processing system according to claim 9, wherein the high dynamic range image processing module further comprises a high dynamic range image processing unit, a lens shading correction unit, and a statistical unit;

The high dynamic range image processing unit is configured to fuse the first color original image and the second color original image into a second high dynamic color original image;

The lens shading correction unit is used to correct the second high dynamic color original image to obtain a high dynamic color correction image;

The statistical unit is used to process the high dynamic color correction image and the first full-color original image to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance deal with.
The high dynamic range image processing system according to claim 1, wherein the image fusion module and the high dynamic range image processing module are integrated in the image sensor.
A high dynamic range image processing method for a high dynamic range image processing system, wherein the high dynamic range image processing system includes an image sensor, the image sensor includes a pixel array, and the pixel array includes a plurality of pixels. A color photosensitive pixel and a plurality of color photosensitive pixels, the color photosensitive pixel has a narrower spectral response than the full-color photosensitive pixel, the pixel array includes a minimum repeating unit, and each minimum repeating unit includes a plurality of subunits, Each of the sub-units includes a plurality of single-color photosensitive pixels and a plurality of full-color photosensitive pixels; the high dynamic range image processing method includes:

The pixel array is exposed, wherein, for a plurality of photosensitive pixels in the same subunit, at least one of the single-color photosensitive pixels is exposed with a first exposure time, and at least one of the single-color photosensitive pixels is less than the first exposure time. The second exposure time of the exposure time is exposed, and at least one of the full-color photosensitive pixels is exposed at a third exposure time that is less than the first exposure time; wherein, the single-color photosensitive pixels exposed at the first exposure time are generated The first color information to obtain a first color original image, the second color information generated by the single-color photosensitive pixel exposed at the second exposure time to obtain a second color original image, and the all exposed at the third exposure time Said full-color photosensitive pixels generate a first full-color original image; and

Perform fusion algorithm processing and high dynamic range processing on the first color original image, the second color original image, and the first panchromatic original image to obtain a first high dynamic range image, the first high dynamic range The image includes a plurality of color image pixels, the plurality of color image pixels are arranged in a Bayer array, and the first high dynamic range image is processed by an image processor to obtain a second high dynamic range image.
The high dynamic range image processing method according to claim 13, wherein a part of the panchromatic photosensitive pixels in the same subunit are exposed with a fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed with the first exposure time. Three exposure time exposure, the fourth exposure time is less than or equal to the first exposure time and greater than the third exposure time, and the second exposure generated by the single-color photosensitive pixel exposed at the fourth exposure time Panchromatic information obtains the second panchromatic original image; the fusion algorithm processing and high dynamic range processing are performed on the first color primitive image, the second color primitive image, and the first panchromatic primitive image to Obtain the first high dynamic range image, including:

Fusing the first color original image and the second panchromatic original image into a first intermediate image, and fusing the second color original image and the first panchromatic original image into a second intermediate image; and

Fusing the first intermediate image and the second intermediate image into the first high dynamic range image.
The high dynamic range image processing method according to claim 14, wherein the fusing the first intermediate image and the second intermediate image into the first high dynamic range image comprises:

Fusing the first intermediate image and the second intermediate image into a third high dynamic range image; and

Perform brightness mapping on the third high dynamic range image to obtain the first high dynamic range image.
The high dynamic range image processing method according to claim 14, wherein the high dynamic range image processing method further comprises:

Fusing the first intermediate image and the second intermediate image into a third high dynamic range image;

Obtaining a high dynamic range corrected image on the third high dynamic range image; and

The high dynamic range correction image is processed to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.
The high dynamic range image processing method according to claim 14, wherein the high dynamic range image processing method further comprises:

The first intermediate image and the second intermediate image are processed to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.
The high dynamic range image processing method according to claim 13, wherein a part of the panchromatic photosensitive pixels in the same subunit are exposed at a fourth exposure time, and the remaining panchromatic photosensitive pixels are exposed at the first exposure time. Three exposure time exposure, the fourth exposure time is less than or equal to the first exposure time and greater than the third exposure time, and the second exposure generated by the single-color photosensitive pixel exposed at the fourth exposure time The panchromatic information obtains the second panchromatic original image; the fusion algorithm processing and high dynamic range processing are performed on the first color primitive image, the second color primitive image, and the first panchromatic primitive image to Obtain the first high dynamic range image, including:

The first color original image and the second color original image are merged into a first high dynamic color original image, and the first panchromatic original image and the second panchromatic original image are merged into a first high dynamic image Full-color original image; and

The first high dynamic color original image and the first high dynamic panchromatic original image are fused into the first high dynamic range image.
The high dynamic range image processing method according to claim 18, wherein the first color original image and the second color original image are merged into a first high dynamic color original image, and the first color original image is combined with the second color original image. The fusion of a panchromatic original image and the second panchromatic original image into a first high dynamic panchromatic original image includes:

The first color original image and the second color original image are merged into a second high-dynamic color original image, and the first panchromatic original image and the second panchromatic original image are merged into a second high-dynamic Full-color original image; and

Perform brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image, and perform brightness mapping on the second high dynamic panchromatic original image to obtain the first high dynamic full color original image image.
18. The high dynamic range image processing method of claim 18, wherein the high dynamic range image processing method further comprises:

The first color original image and the second color original image are merged into a second high-dynamic color original image, and the first panchromatic original image and the second panchromatic original image are merged into a second high-dynamic Full-color original image;

Correcting the second high-dynamic color original image to obtain a high-dynamic color corrected image, and correcting the second high-dynamic panchromatic original image to obtain a high-dynamic panchromatic correction image; and

The high dynamic color correction image and the high dynamic full color correction image are processed to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.
The high dynamic range image processing method according to claim 13, wherein all the panchromatic photosensitive pixels in the same sub-unit are exposed with a third exposure time; the pair of the first color original image, Performing fusion algorithm processing and high dynamic range processing on the second color original image and the first panchromatic original image to obtain a first high dynamic range image includes:

Fusing the first color original image and the second color original image into a first high dynamic color original image; and

The first high dynamic color original image and the first panchromatic original image are fused into the first high dynamic range image.
The high dynamic range image processing method according to claim 21, wherein said fusing the first color original image and the second color original image into a first high dynamic color original image comprises:

Fusing the first color original image and the second color original image into a second high dynamic color original image; and

Performing brightness mapping on the second high dynamic color original image to obtain the first high dynamic color original image.
The high dynamic range image processing method according to claim 21, wherein the high dynamic range image processing method further comprises:

Fusing the first color original image and the second color original image into a second high dynamic color original image;

Correcting the second high dynamic color original image to obtain a high dynamic color correction image; and

The high dynamic color correction image and the first full-color original image are processed to obtain statistical data, and the statistical data is provided to the image processor for automatic exposure processing and/or automatic white balance processing.
An electronic device, characterized in that it comprises:

Lens

Shell; and

The high dynamic range image processing system according to any one of claims 1 to 12, wherein the lens, the high dynamic range image processing system are combined with the housing, and the lens is connected to the high dynamic range image processing system. The image sensor cooperates with imaging.
A non-volatile computer-readable storage medium containing a computer program, wherein when the computer program is executed by a processor, the processor executes the high dynamic range of any one of claims 13 to 23 Image processing method.