CN114125318B

CN114125318B - Image sensor, camera module, electronic device, image generation method and device

Info

Publication number: CN114125318B
Application number: CN202111339984.2A
Authority: CN
Inventors: 李小涛
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-11-12
Filing date: 2021-11-12
Publication date: 2024-08-02
Anticipated expiration: 2041-11-12
Also published as: WO2023082766A1; CN114125318A

Abstract

The application relates to an image sensor and image generation method, the image sensor 21 includes the filter array 23 and pixel array 24, the filter array 23 includes the minimum repeating unit 230, the minimum repeating unit 230 includes a plurality of filter sets, each filter set includes the panchromatic filter 233 and color filter 234, the light incoming quantity that the panchromatic filter 233 transmits is greater than the light incoming quantity that the color filter 234 transmits, the color filter 234 and panchromatic filter 233 each include 4 sub-filters, the plurality of filter sets includes at least the first filter set 231, include the first color sub-filter 2341 and the second color sub-filter 2342 in at least one color filter 234 in the first filter set 231; the pixel array 24 includes a plurality of pixels, the pixels of the pixel array 24 being disposed corresponding to the sub-filters of the filter array 23, the pixel array 24 being configured to receive light passing through the filter array 23 to generate an electrical signal. The image generated by the image sensor has higher definition and better quality.

Description

Image sensor, camera module, electronic device, image generation method and device

Technical Field

The present application relates to the field of image technology, and in particular, to an image sensor, an image capturing module, an electronic device, an image generating method, an image generating apparatus, a computer readable storage medium, and a computer program product.

Background

With the development of computer technology, most of electronic devices such as mobile phones are configured with cameras so as to realize a photographing function through the cameras. An image sensor is arranged in the camera, and color images are acquired through the image sensor. In order to realize color image acquisition, an optical filter array arranged in a Bayer (Bayer) array is generally disposed in an image sensor, so that a plurality of pixels in the image sensor can receive light passing through corresponding optical filters, thereby generating pixel signals with different color channels and further generating an image.

However, the image sharpness generated by the conventional image sensor is low.

Disclosure of Invention

The embodiment of the application provides an image sensor, an image generation method, an image generation device, electronic equipment and a computer readable storage medium, which can improve imaging definition.

An image sensor comprising a filter array and a pixel array, the filter array comprising a minimal repeating unit, the minimal repeating unit comprising a plurality of filter sets, each filter set comprising a color filter and a panchromatic filter, the panchromatic filter transmitting a greater amount of light than the color filter, the color filter and the panchromatic filter each comprising 4 sub-filters, the plurality of filter sets comprising at least a first filter set, at least one of the color filters comprising a first color sub-filter and a second color sub-filter; the pixel array includes a plurality of pixels, the pixels of the pixel array being disposed corresponding to sub-filters of the filter array, the pixel array being configured to receive light passing through the filter array to generate an electrical signal.

A camera module comprising a lens and an image sensor; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays; the image sensor comprises a filter array and a pixel array, wherein the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, each filter set comprises a color filter and a full-color filter, the light incoming quantity transmitted by the full-color filter is larger than the light incoming quantity transmitted by the color filter, the color filter and the full-color filter comprise 4 sub-filters, the filter sets at least comprise a first filter set, and at least one color filter in the first filter set comprises a first color sub-filter and a second color sub-filter; the pixel array includes a plurality of pixels, the pixels of the pixel array being disposed corresponding to sub-filters of the filter array, the pixel array being configured to receive light passing through the filter array to generate an electrical signal.

An electronic device comprises a camera module and a shell, wherein the camera module is arranged on the shell; the camera module comprises a lens and an image sensor; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays; the image sensor comprises a filter array and a pixel array, wherein the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, each filter set comprises a color filter and a full-color filter, the light incoming quantity transmitted by the full-color filter is larger than the light incoming quantity transmitted by the color filter, the color filter and the full-color filter comprise 4 sub-filters, the filter sets at least comprise a first filter set, and at least one color filter in the first filter set comprises a first color sub-filter and a second color sub-filter; the pixel array includes a plurality of pixels, the pixels of the pixel array being disposed corresponding to sub-filters of the filter array, the pixel array being configured to receive light passing through the filter array to generate an electrical signal.

The image sensor comprises the filter array and the pixel array, wherein the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, the filter sets comprise a full-color filter and a color filter, the light inlet quantity transmitted by the full-color filter is larger than the light inlet quantity transmitted by the color filter, and more light quantity can be obtained during shooting, so that shooting parameters are not required to be regulated, and the definition of imaging under dark light is improved under the condition of not affecting the shooting stability. When imaging under the dark light, the stability and the definition can be considered, and the stability and the definition of imaging under the dark light are both higher. And each full-color filter comprises 4 sub-filters, each color filter comprises 4 sub-filters, the plurality of filter sets at least comprises a first filter set, at least one color filter in the first filter set comprises a first color sub-filter and a second color sub-filter, the pixel array comprises a plurality of full-color pixels and a plurality of color pixels, each full-color pixel corresponds to one sub-filter of the full-color filter, each first color photosensitive pixel corresponds to one first color sub-filter of the color filter, and each second color photosensitive pixel corresponds to one second color sub-filter of the color filter, so that different color photosensitive pixels can be mixed and arranged in the same color filter, the resolving power of a color channel is effectively improved, and the problem of false color in a generated image is reduced.

An image generation method is applied to an image sensor, the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, each filter set comprises a color filter and a full-color filter, the light inlet amount of the full-color filter transmitted by the full-color filter is larger than the light inlet amount of the color filter transmitted by the full-color filter, the color filter and the full-color filter both comprise 4 sub-filters, the filter sets comprise a first filter set and a second filter set, and at least one color filter in the first filter set comprises a first color sub-filter and a second color sub-filter; the color filter in the second filter set at least comprises a third color sub-filter; the pixel array comprises a plurality of pixels, a first color photosensitive pixel of the pixel array is arranged corresponding to a first color sub-filter of the filter array, a second color photosensitive pixel of the pixel array is arranged corresponding to a second color sub-filter of the filter array, a third color photosensitive pixel of the pixel array is arranged corresponding to a third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electric signal;

The method comprises the following steps:

In a first resolution mode, according to a first pixel value which is read out in a merging way by a plurality of full-color pixels corresponding to the full-color filter in the filter set, a second pixel value which is read out in a merging way by a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value which is read out in a merging way by a plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained;

Combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image;

and obtaining a first target image based on the first combined image and the second combined image.

An image generating device is applied to an image sensor, the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, each filter set comprises a color filter and a full-color filter, the light inlet amount of the full-color filter transmitted by the full-color filter is larger than the light inlet amount of the color filter transmitted by the full-color filter, the color filter and the full-color filter both comprise 4 sub-filters, the filter sets comprise a first filter set and a second filter set, and at least one color filter in the first filter set comprises a first color sub-filter and a second color sub-filter; the color filter in the second filter set at least comprises a third color sub-filter; the pixel array comprises a plurality of pixels, a first color photosensitive pixel of the pixel array is arranged corresponding to a first color sub-filter of the filter array, a second color photosensitive pixel of the pixel array is arranged corresponding to a second color sub-filter of the filter array, a third color photosensitive pixel of the pixel array is arranged corresponding to a third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electric signal;

The device comprises:

The first merging module is used for merging the read first pixel values according to a plurality of full-color pixels corresponding to the full-color filters in the filter set, merging the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the color filters, and merging the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the color filters in a first resolution mode to obtain a first merged image;

The second merging module is used for merging the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second merged image;

and the image generation module is used for obtaining a first target image based on the first combined image and the second combined image.

An electronic device comprising a memory, a processor and an image sensor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of:

in a first resolution mode, according to a first pixel value which is read out in a merging way by a plurality of full-color pixels corresponding to the full-color filter in the filter set, a second pixel value which is read out in a merging way by a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value which is read out in a merging way by a plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained; combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image; and obtaining a first target image based on the first combined image and the second combined image.

A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

A computer program product comprising a computer program which, when executed by a processor, performs the steps of:

In the image generating method, the device, the electronic equipment, the computer readable storage medium and the computer program product, in the first resolution mode, the full-color channel information can be fused into the image, the integral light inlet quantity is improved, and the first combined image with more information and clearer detail analysis can be generated by combining the read first pixel values through a plurality of full-color pixels corresponding to the full-color filters in the filter set, combining the read second pixel values through a plurality of first color photosensitive pixels corresponding to the color filters and combining the read third pixel values through a plurality of third color photosensitive pixels corresponding to the color filters. According to the fourth pixel value read by combining the plurality of second color photosensitive pixels corresponding to the color filter, the second color photosensitive pixels can be separated to form a second combined image, so that the positions of the second color photosensitive pixels in the second combined image and the positions of the first color photosensitive pixels in the first combined image are kept consistent. Based on the first combined image and the second combined image, a first target image is obtained, and the first color photosensitive pixels in the first combined image and the second color photosensitive pixels in the second combined image can be arranged in a mixed mode, so that the generated first target image is clearer in color. And the pixel binning readout results in a reduced size of the generated first target image, low power consumption required to generate the image.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an electronic device in one embodiment;

FIG. 2 is an exploded view of an image sensor in one embodiment;

FIG. 3 is a schematic diagram of the connection of a pixel array and readout circuitry in one embodiment;

FIG. 4a is a schematic diagram of a first diagonal direction and a second diagonal direction in one embodiment;

FIG. 4b is a schematic diagram of a third diagonal and a fourth diagonal in one embodiment;

FIG. 5 is a schematic diagram of an arrangement of minimal repeating units in one embodiment;

FIG. 6 is a schematic diagram of an arrangement of minimal repeating units in one embodiment;

FIG. 7 is a schematic diagram of an arrangement of minimal repeating units in another embodiment;

FIG. 8 is a schematic diagram of an arrangement of minimal repeating units in another embodiment;

FIG. 9 is a flow chart of an image generation method in one embodiment;

FIG. 10 is a schematic diagram of obtaining a first target image in a first resolution mode in one embodiment;

FIG. 11 is a schematic diagram of obtaining a first target image in a first resolution mode according to another embodiment;

FIG. 12 is a schematic diagram of obtaining a second target image in a second resolution mode in one embodiment;

FIG. 13 is a schematic diagram of obtaining a second target image in a second resolution mode according to another embodiment;

FIG. 14 is a schematic diagram of an image generation method in a second resolution mode in another embodiment;

FIG. 15 is a block diagram showing the structure of an image generating apparatus in one embodiment;

fig. 16 is an internal block diagram of an electronic device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

It will be understood that the terms first, second, etc. as used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another element. For example, a first target image may be referred to as a second target image, and similarly, a second target image may be referred to as a first target image, without departing from the scope of the application. Both the first and second are target images, but they are not the same target image. The plurality refers to at least two, such as 2,4, but not limited thereto.

In one embodiment, the electronic device 100 includes a mobile phone, a tablet computer, a notebook computer, an atm, a gate, a smart watch, a head display device, etc., and it is understood that the electronic device 100 may be any other device with an image processing function. The electronic device 100 includes a camera module 20, a processor 30, and a housing 40. The camera module 20 and the processor 30 are both disposed in the housing 40, and the housing 40 can be further used for mounting functional modules such as a power supply device and a communication device of the electronic device 100, so that the housing 40 provides protection such as dust protection, drop protection, water protection, etc. for the functional modules.

The camera module 20 may be a front camera module, a rear camera module, a side camera module, an under-screen camera module, etc., which is not limited herein. The image capturing module 20 includes a lens and an image sensor 21, and when the image capturing module 20 captures an image, light passes through the lens and reaches the image sensor 21, and the image sensor 21 is configured to convert an optical signal applied to the image sensor 21 into an electrical signal.

In one embodiment, as shown in FIG. 2, the image sensor 21 includes a microlens array 22, a filter array 23, and a pixel array 24.

The microlens array 22 includes a plurality of microlenses 221, the sub-filters in the filter array 23, and the pixels in the pixel array 24 are arranged in a one-to-one correspondence, the microlenses 221 are configured to collect incident light, and the collected light passes through the corresponding sub-filters and then is projected onto the pixels, received by the corresponding pixels, and the pixels convert the received light into electrical signals.

The filter array 23 includes a plurality of minimal repeating units 230. The minimal repeating unit 230 may include a plurality of filter sets. Each filter set includes a full-color filter 233 and a color filter 234, and the full-color filter 233 transmits a greater amount of light than the color filter 234. Each of the full color filters 233 includes 4 sub-filters 2331, each of the color filters 234 includes 4 sub-filters, the plurality of filter sets includes at least a first filter set 231, and at least one of the color filters 234 in the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. Different color filters 234 are also included in different filter sets.

In one embodiment, the plurality of filter sets further includes a second filter set 232, and the color filters 234 in the second filter set 232 include at least a third color sub-filter 2343.

Colors corresponding to the wavelength bands of light transmitted by the color filters 234 of the filter set in the minimal repeating unit 230 include color a, color b, and/or color c. The colors corresponding to the wavelength bands of the transmitted light of the color filters 234 of the filter set 232 include color a, color b, and color c, or color a, color b, or color c, or color a and color b, or color b and color c, or color a and color c. Here, the color a is red, the color b is green, and the color c is blue, or, for example, the color a is magenta, the color b is cyan, the color c is yellow, or the like, without limitation.

In one embodiment, the width of the band of light transmitted by the color filter 234 is smaller than the width of the band of light transmitted by the panchromatic filter 233, for example, the band of light transmitted by the color filter 234 may correspond to the band of red, green, or blue light, the band of light transmitted by the panchromatic filter 233 is all the bands of visible light, that is, the color filter 234 allows only light of a particular color to pass, and the panchromatic filter 233 may pass light of all colors. Of course, the wavelength band of the transmitted light of the color filter 234 may also correspond to the wavelength band of other color light, such as magenta, violet, cyan, yellow, etc., which is not limited herein.

In one embodiment, the ratio of the number of color filters 234 to the number of panchromatic filters 233 in the filter set may be 1:3, 1:1, or 3:1. For example, if the ratio of the number of the color filters 234 to the number of the full-color filters 233 is 1:3, the number of the color filters 234 is 1, and the number of the full-color filters 233 is 3, and at this time, the number of the full-color filters 233 is larger, so that the imaging quality under dark light is better; or the ratio of the number of the color filters 234 to the number of the full-color filters 233 is 1:1, the number of the color filters 234 is 2, and the number of the full-color filters 233 is 2, so that better color expression can be obtained, and the imaging quality under dark light is better at the same time; or the ratio of the number of the color filters 234 to the number of the full-color filters 233 is 3:1, the number of the color filters 234 is 3, and the number of the full-color filters 233 is 1, so that better color performance can be obtained and the imaging quality under dark light can be improved.

The pixel array 24 includes a plurality of pixels, and the pixels of the pixel array 24 are disposed corresponding to the sub-filters of the filter array 23. The pixel array 24 is configured to receive light passing through the filter array 23 to generate an electrical signal.

Wherein pixel array 24 is configured to receive light rays passing through filter array 23 to generate electrical signals, refers to pixel array 24 being configured to photoelectrically convert light rays of a scene of a given set of subjects passing through filter array 23 to generate electrical signals. Rays of a scene of a given set of objects are used to generate image data. For example, a subject is a building, and a scene of a given set of subjects refers to a scene in which the building is located, and other subjects may be included in the scene.

In one embodiment, pixel array 24 includes a plurality of minimal repeating units 240, where minimal repeating units 240 include a plurality of pixel groups including panchromatic pixel group 243 and color pixel group 244. Each of the full-color pixel groups 243 includes 4 full-color pixels 2431, each of the color pixel groups 244 includes 4 color pixels, the plurality of pixel groups includes at least a first pixel group 241, and at least one of the color pixel groups 244 of the first pixel group 241 includes a first color photosensitive pixel 2441 and a second color photosensitive pixel 2442. Each panchromatic pixel 2431 corresponds to one of the sub-filters 2331 in the panchromatic filter 233, and the panchromatic pixel 2431 receives the light passing through the corresponding sub-filter 2331 to generate an electrical signal. Each first color sensitive pixel 2441 corresponds to one first color sub-filter 2341 of the color filter 234, and the first color sensitive pixel 2441 receives light passing through the corresponding first color sub-filter 2341 to generate an electrical signal. Each second color sensitive pixel 2442 corresponds to one second color sub-filter 2342 of the color filter 234, and the second color sensitive pixel 2442 receives light passing through the corresponding second color sub-filter 2342 to generate an electrical signal.

In one embodiment, the plurality of pixel groups further includes a second pixel group 242, and the color pixel group 244 in the second pixel group 242 includes at least a third color sensitive pixel 2443. Each third color sensitive pixel 2443 corresponds to one third color sub-filter 2343 of the color filter 234, and the third color sensitive pixel 2443 receives light passing through the corresponding third color sub-filter 2343 to generate an electrical signal.

The image sensor 21 in this embodiment includes the filter array 23 and the pixel array 24, the filter array 23 includes the minimum repeating unit 230, the minimum repeating unit 230 includes a plurality of filter sets, each filter set includes the panchromatic filter 233 and the color filter 234, the color filter 234 has a narrower spectral response than the panchromatic filter 233, the panchromatic filter 233 transmits a greater amount of light than the color filter 234, and a greater amount of light can be obtained during shooting, so that the shooting parameters do not need to be adjusted, and the definition of imaging under dark light is improved without affecting the stability of shooting. When imaging under the dark light, the stability and the definition can be considered, and the stability and the definition of imaging under the dark light are both higher. In addition, each full-color filter 233 includes 4 sub-filters 2331, each color filter 234 includes 4 sub-filters, the plurality of filter sets includes at least a first filter set 231, at least one color filter 234 in the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342, the pixel array 24 includes a plurality of full-color pixels 2431 and a plurality of color pixels 2441, each full-color pixel 2431 corresponds to one sub-filter 2331 of the full-color filter 233, each first color sub-filter 2441 corresponds to one first color sub-filter 2341 of the color filter 234, and each second color sub-filter 2442 corresponds to one second color sub-filter 2342 of the color filter 234.

In one embodiment, as shown in FIG. 2, the minimal repeating unit 230 in the filter array 23 comprises 4 filter sets, and the 4 filter sets comprise 2 first filter sets 231 and 2 second filter sets 232,2 first filter sets 231 and 2 second filter sets 232 arranged in a matrix.

Each of the first filter sets 231 includes a full-color filter 233 and a color filter 234, each of the full-color filter 233 and the color filter 234 has 4 sub-filters, and at least one of the color filters 234 includes a first color sub-filter 2341 and a second color sub-filter 2342, and then the second filter set 232 includes 16 sub-filters in total.

Each second filter set 232 includes a full color filter 233 and a color filter 234, each full color filter 233 and each color filter 234 has 4 sub-filters, and at least a third color sub-filter 2343 is included in the color filter 234, and the second filter set 232 includes 16 sub-filters in total.

Similarly, the pixel array 24 includes a plurality of minimum repeating units 240, which are in one-to-one correspondence with the plurality of minimum repeating units 230 of the filter array 23. Each minimal repeating unit 240 includes 4 pixel groups, and the 4 pixel groups include 2 first pixel groups 241 and 2 second pixel groups 242, and the 2 first pixel groups 241 and 2 second pixel groups 242 are arranged in a matrix. Each first pixel group 241 corresponds to one first filter group 231, and each second pixel group 242 corresponds to one second filter group 232.

As shown in fig. 3, the readout circuit 25 is electrically connected to the pixel array 24, and is used for controlling the exposure of the pixel array 24 and the reading and outputting of the pixel values of the pixel points. The readout circuit 25 includes a vertical driving unit 251, a control unit 252, a column processing unit 253, and a horizontal driving unit 254. The vertical driving unit 251 includes a shift register and an address decoder. The vertical driving unit 251 includes a readout scan and a reset scan function. The control unit 252 configures timing signals according to an operation mode, and controls the vertical driving unit 251, the column processing unit 253, and the horizontal driving unit 254 to operate cooperatively using various timing signals. The column processing unit 253 may have an analog-to-digital (a/D) conversion function for converting analog pixel signals into a digital format. The horizontal driving unit 254 includes a shift register and an address decoder. The horizontal driving unit 254 sequentially scans the pixel array 24 column by column.

In one embodiment, as shown in fig. 4a, the first filter set 231 is disposed in a first diagonal D1 direction and the second filter set 232 is disposed in a second diagonal D2 direction in the minimal repeating unit 230, and the first diagonal D1 direction is different from the second diagonal D2 direction. The first diagonal line D1 direction and the second diagonal line D2 direction are different, and color expression and dim light imaging quality can be considered.

The first diagonal D1 direction is different from the second diagonal D2 direction, specifically, the first diagonal D1 direction is not parallel to the second diagonal D2 direction, or the first diagonal D1 direction is perpendicular to the second diagonal D2 direction, or the like.

In one embodiment, as shown in FIG. 4b, each filter set includes color filters 234 and panchromatic filters 233, the color filters 234 and panchromatic filters 233 of each filter set being arranged in a matrix. Each of the full color filters 233 in the filter set is disposed in the third diagonal direction D3, and each of the color filters 234 in the filter set is disposed in the fourth diagonal direction D4. The third diagonal line D3 direction and the fourth diagonal line D4 direction are different, and color expression and dim light imaging quality can be considered.

The third diagonal D3 direction is different from the fourth diagonal D4 direction, and specifically, the third diagonal D3 direction is not parallel to the fourth diagonal D4 direction, or the third diagonal D3 direction is perpendicular to the fourth diagonal D4 direction, or the like.

In other embodiments, one color filter 234 and one full-color filter 233 may be located at the third diagonal D3, and the other color filter 234 and the other full-color filter 233 may be located at the fourth diagonal D4.

In one embodiment, the color filter 234 may be disposed in a direction parallel to the third diagonal D3, or the color filter 234 may be disposed in a direction parallel to the fourth diagonal D4.

In one embodiment, at least one color filter 234 in the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The first color sub-filter 2341 is disposed in one direction of the fifth diagonal direction or the sixth diagonal direction, and the second color sub-filter 2342 is disposed in the other direction of the fifth diagonal direction or the sixth diagonal direction. The sub-filters 2331 of the full-color filter 233 in the first filter group 231 are disposed in the fifth diagonal direction and the sixth diagonal direction.

In one embodiment, the minimal repeating unit 230 includes 2 first filter sets 231 and 2 second filter sets 232,2 first filter sets 231 and 2 second filter sets 232 arranged in a matrix.

Of the 2 first filter sets 231, a first color sub-filter 2341 of the color filters 234 of one of the first filter sets 231 is disposed in a fifth diagonal direction, and a second color sub-filter 2342 is disposed in a sixth diagonal direction; also, a first color sub-filter 2341 of the color filters 234 of the other first filter set 231 is disposed in the sixth diagonal direction, and a second color sub-filter 2342 is disposed in the fifth diagonal direction.

It is understood that the first color sub-filter 2341 is disposed in the fifth diagonal direction, and the first color sub-filter 2341 may be disposed on the fifth diagonal, or the first color sub-filter 2341 may be disposed in a direction parallel to the fifth diagonal direction, and the other diagonal directions may be similar.

In one embodiment, as shown in fig. 5, the plurality of filter sets includes a first filter set 231 and a second filter set 232, and at least one color filter 234 of the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The color filters 234 in the second filter set 232 include at least a third color sub-filter 2343. The third color sub-filters 2343 in the second filter set 232 are disposed in a seventh diagonal D7 direction and an eighth diagonal D8 direction, the seventh diagonal D7 direction being different from the eighth diagonal D8 direction.

The seventh diagonal D7 direction is different from the eighth diagonal D8 direction, and specifically, the seventh diagonal D7 direction is not parallel to the eighth diagonal D8 direction, or the seventh diagonal D7 direction is perpendicular to the eighth diagonal D8 direction, or the like.

In one embodiment, as shown in FIG. 5, the minimal repeating unit 230 in the filter array 23 comprises 4 filter sets, and the 4 filter sets are arranged in a matrix. The 4 filter sets include a first filter set 231 and a second filter set 232. Each of the first filter sets 231 includes 2 full-color filters 233 and 2 color filters 234, and each of the second filter sets 232 includes 2 full-color filters 233 and 2 color filters 234. The full color filter 233 includes 4 sub-filters 2331, the same color filter 234 of the first filter set 231 includes 2 first color sub-filters 2341 and 2 second color sub-filters 2342, the same color filter 234 of the second filter set 232 includes 4 third color sub-filters 2343, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, which are arranged in the following manner:

Where w represents the full color sub-filter 2331, and a, b, and c each represent a color sub-filter. The color sub-filters include a first color sub-filter 2341, a second color sub-filter 2342, and a third color sub-filter 2343.

A. b and c represent one of the first color sub-filter 2341, the second color sub-filter 2342, and the third color sub-filter 2343, respectively. The full sub-filter 2331 may be a sub-filter that filters out all light outside the visible light band, and the first, second and third color sub-filters 2341, 2342 and 2343 may be red, green, blue, magenta, cyan and yellow sub-filters. The red sub-filter is a sub-filter for filtering all light except red light, the green sub-filter is a sub-filter for filtering all light except green light, the blue sub-filter is a sub-filter for filtering all light except blue light, the magenta sub-filter is a sub-filter for filtering all light except magenta light, the cyan sub-filter is a sub-filter for filtering all light except cyan light, and the yellow sub-filter is a sub-filter for filtering all light except yellow light.

A may be a red, green, blue, magenta, cyan, or yellow sub-filter, b may be a red, green, blue, magenta, cyan, or yellow sub-filter, and c may be a red, green, blue, magenta, cyan, or yellow sub-filter. For example, b is a red sub-filter, a is a green sub-filter, and c is a blue sub-filter; or c is a red sub-filter, a is a green sub-filter, b is a blue sub-filter; for another example, c is a red sub-filter, a is a green sub-filter, and b is a blue sub-filter; or a is a red sub-filter, b is a blue sub-filter, c is a green sub-filter, etc., without limitation; for another example, b is a magenta sub-filter, a is a cyan sub-filter, b is a yellow sub-filter, and the like. In other embodiments, the color filter may further include sub-filters of other colors, such as orange sub-filters, violet sub-filters, etc., without limitation.

In one embodiment, as shown in FIG. 6, the minimal repeating unit 230 in the filter array 23 comprises 4 filter sets, and the 4 filter sets are arranged in a matrix. The 4 filter sets include a first filter set 231 and a second filter set 232. Each of the first filter sets 231 includes 2 full-color filters 233 and 2 color filters 234, and each of the second filter sets 232 includes 2 full-color filters 233 and 2 color filters 234. The full color filter 233 includes 4 sub-filters 2331, the same color filter 234 of the first filter set 231 includes 2 first color sub-filters 2341 and 2 second color sub-filters 2342, the same color filter 234 of the second filter set 232 includes 4 third color sub-filters 2343, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, which are arranged in the following manner:

In this embodiment, the photosensitive pixels with different colors are arranged in a mixed manner, for example, red pixels and blue pixels are arranged in a mixed manner, so that each row and each column of the minimum repeating unit have RGB pixels, the resolving power of a color channel can be effectively improved, and the problem that a generated image has a false color is reduced. And, full-color pixels are introduced, so that the light quantity is effectively improved, and the definition of an image can be improved. Meanwhile, the image sensor in the embodiment also has the advantage of two-level pixel merging and reading, each row and each column of the minimum repeating unit of the image sensor have RGB pixels, and pixel merging of the same pixels is not needed to be carried out across different minimum repeating units, so that the power consumption of processing can be effectively reduced.

In one embodiment, as shown in fig. 7, the plurality of filter sets includes a first filter set 231 and a second filter set 232, and at least one color filter 234 of the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The color filters 234 in the second filter set 232 include a third color sub-filter 2343, and the color filters 234 in the second filter set 232 further include a first color sub-filter 2341 or a second color sub-filter 2342.

Specifically, the same color filter 234 in the second filter set 232 includes a third color sub-filter 2343 and a first color sub-filter 2341, or the same color filter 234 in the second filter set 232 includes a third color sub-filter 2343 and a second color sub-filter 2342.

In one embodiment, as shown in fig. 7, the second filter set 232 includes 2 color filters 234, and the sub-filters of the same color in each color filter 234 are diagonally arranged, and the arrangement directions of the sub-filters of the same color in the two color filters 234 are opposite.

Specifically, the second filter set 232 includes two color filters 234, and the same color filter 234 includes a third color sub-filter 2343 and a first color sub-filter 2341. The third color sub-filters 2343 in the same color filter 234 are diagonally arranged, and the first color sub-filters 2341 in the same color filter 234 are diagonally arranged. The arrangement directions of the first color sub-filters 2341 in the 2 color filters 234 are opposite, and the arrangement directions of the third color sub-filters 2343 in the 2 color filters 234 are opposite. For example, the third color sub-filters 2343 are disposed in the diagonal A1 direction and the diagonal A2 direction, respectively, among the 2 color filters 234.

Or the second filter set 232 includes two color filters 234, and the same color filter 234 includes a third color sub-filter 2343 and a second color sub-filter 2342. The third color sub-filters 2343 in the same color filter 234 are diagonally arranged, and the second color sub-filters 2342 in the same color filter 234 are diagonally arranged. The arrangement direction of the second color sub-filters 2342 in the 2 color filters 234 is opposite, and the arrangement direction of the third color sub-filters 2343 in the 2 color filters 234 is opposite.

In one embodiment, as shown in FIG. 7, the minimal repeating unit 230 in the filter array 23 comprises 4 filter sets, and the 4 filter sets are arranged in a matrix. Each filter set contains 2 full color filters 233 and 2 color filters 234. The full color filter 233 includes 4 sub-filters 2331, the same color filter 234 of the first filter set 231 includes 2 first color sub-filters 2341 and 2 second color sub-filters 2342, the same color filter 234 where the second filter set 232 exists includes 2 first color sub-filters 2341 and 2 third color sub-filters 2343, and the same color filter 234 where the second filter set 232 exists includes 2 second color sub-filters 2342 and 2 third color sub-filters 2343, the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is:

In one embodiment, as shown in FIG. 8, the minimal repeating unit 230 in the filter array 23 comprises 4 filter sets, and the 4 filter sets are arranged in a matrix. Each filter set contains 2 full color filters 233 and 2 color filters 234. The full color filter 233 includes 4 sub-filters 2331, the same color filter 234 of the first filter set 231 includes 2 first color sub-filters 2341 and 2 second color sub-filters 2342, the same color filter 234 where the second filter set 232 exists includes 2 first color sub-filters 2341 and 2 third color sub-filters 2343, and the same color filter 234 where the second filter set 232 exists includes 2 second color sub-filters 2342 and 2 third color sub-filters 2343, the smallest repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is:

In one embodiment, an image generation method is provided for application to an image sensor 21 as shown in fig. 2, the image sensor 21 comprising a filter array 23 and a pixel array 24, the filter array 23 comprising a minimal repeating unit 230, the minimal repeating unit 230 comprising a plurality of filter sets, each filter set comprising a color filter 234 and a panchromatic filter 233, the panchromatic filter 233 transmitting an amount of light greater than the color filter 234, the color filter 234 having a narrower spectral response than the panchromatic filter 233, 4 sub-filters 2331 included in each panchromatic filter 233, 4 sub-filters included in each color filter 234, the plurality of filter sets comprising a first filter set 231 and a second filter set 232, at least one color filter 234 in the first filter set 231 comprising a first color sub-filter 2341 and a second color sub-filter 2342. The color filters 234 of the second filter set 232 include at least a third color sub-filter 2343.

The pixel array 24 includes a plurality of pixels, a first color sensitive pixel 2441 of the pixel array 24 is disposed corresponding to a first color sub-filter 2341 of the filter array 23, a second color sensitive pixel 2442 of the pixel array 24 is disposed corresponding to a second color sub-filter 2342 of the filter array 23, a third color sensitive pixel 2443 of the pixel array 24 is disposed corresponding to a third color sub-filter 2343 of the filter array 23, and the pixel array 24 is configured to receive light passing through the filter array 23 to generate an electrical signal;

In one embodiment, the pixel array 24 includes a plurality of panchromatic pixels 2431 and a plurality of color pixels 2441, each panchromatic pixel 2431 corresponding to one sub-filter 2331 of the panchromatic filter 233, each first color-sensitive pixel 2441 corresponding to one first color sub-filter 2341 of the color filter 234, each second color-sensitive pixel 2442 corresponding to one second color sub-filter 2342 of the color filter 234, each third color-sensitive pixel 2443 corresponding to one third color sub-filter 2343 of the color filter 234;

as shown in fig. 9, the image generation method includes:

In step 902, in the first resolution mode, the first combined image is obtained according to the first pixel values read out by combining the plurality of full-color pixels corresponding to the full-color filters in the filter set, the second pixel values read out by combining the plurality of first color photosensitive pixels corresponding to the color filters, and the third pixel values read out by combining the plurality of third color photosensitive pixels corresponding to the color filters.

The first resolution mode refers to a primary pixel merging read-out mode in which resolution, power consumption, signal-to-noise ratio and frame rate are relatively balanced. The first resolution mode may be a default mode for image or video capturing. The merged readout refers to summing pixel values of a plurality of pixels or calculating a mean value of pixel values of a plurality of pixels. For example, the first pixel value read out by combining the plurality of full-color pixels may be a first pixel value obtained by summing the pixel values of the plurality of full-color pixels; or calculating an average of pixel values of a plurality of full-color pixels as the first pixel value.

The color filter 234 has a narrower spectral response than the full-color filter 233, so that the amount of light entering through the full-color filter 233 is greater than the amount of light entering through the color filter 234, i.e., the band width of light passing through the color filter 234 is smaller than the band width of light passing through the full-color filter 233, the full-color filter 233 transmits more light, and the corresponding full-color pixel 2431, which contains more information and can resolve more texture details, is obtained through the full-color filter 233 with a higher signal-to-noise ratio. Where the signal-to-noise ratio refers to the ratio between the normal signal and the noise signal. The higher the signal-to-noise ratio of a pixel, the higher the proportion of normal signals contained in that pixel, and the more information is parsed from that pixel.

The color pixels may be G (Green), R (Red), B (Blue), etc., but are not limited thereto.

In the case of receiving a photographing instruction, whether a user selects a resolution mode to be used is detected, and when it is detected that the user selects the first resolution mode to be used, or when the user does not select the resolution mode to be used, preview photographing is not used, and the current environment is not night scene mode, the first resolution mode is used in response to the photographing instruction.

In the first resolution mode, the light transmitted through the sub-filter 2331 in the full-color filter 233 is projected onto the corresponding full-color pixel 2431, and the full-color pixel 2431 receives the light transmitted through the sub-filter 2331 to generate an electrical signal. The light transmitted by the first color sub-filter 2341 in the color filter 234 is projected onto the corresponding first color photosensitive pixel 2441, and the first color photosensitive pixel 2441 passes through the light of the corresponding first color sub-filter 2341 to generate an electrical signal. The light transmitted through the second color sub-filter 2342 in the color filter 234 is projected onto the corresponding second color photosensitive pixel 2442, and the second color photosensitive pixel 2442 passes through the light of the corresponding second color sub-filter 2342 to generate an electrical signal. The light transmitted through the third color sub-filter 2343 in the color filter 234 is projected onto the corresponding third color photosensitive pixel 2443, and the third color photosensitive pixel 2443 passes through the light of the corresponding third color sub-filter 2343 to generate an electrical signal.

At least one color filter 234 in the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342, and then at least one color filter 234 in the first filter set 231 corresponds to the first color sensitive pixel 2441 and the second color sensitive pixel 2442.

The electronics combine the plurality of panchromatic pixels 2431 corresponding to the same panchromatic filter 233 to read out the first pixel value. The same color filter 234 may include a plurality of first color sub-filters 2341 and a plurality of second color sub-filters 2342, and the electronic device combines the first color sensitive pixels 2441 corresponding to the plurality of first color sub-filters 2341 in the same color filter 234 to read out the second pixel value. The same color filter 234 may include a plurality of third color sub-filters 2343, and the electronic device combines the third color sensitive pixels 2443 corresponding to the plurality of third color sub-filters 2343 in the same color filter 234 to read out a third pixel value.

The electronic device obtains a first combined image according to each first pixel value, each second pixel value and each third pixel value.

Step 904, merging the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second merged image.

For the color filter 234 including the second color sub-filters 2342, the electronic device reads out the second color sensitive pixels 2442 corresponding to each of the second color sub-filters 2342 in the same color filter 234 to combine and read out the fourth pixel values, and generates a second combined image based on each of the fourth pixel values.

In one embodiment, the first combined image is an image comprised of first, second and third pixel values, and the second combined image is an image comprised of fourth and empty pixels. Wherein a null pixel is a pixel without any information.

In one embodiment, the same full color filter 233 corresponds to 4 full color pixels 2431, and in different filter sets, the same color filter 234 may simultaneously correspond to 2 first color sensitive pixels 2441 and 2 second color sensitive pixels 2442, and the color filter 234 may correspond to 4 third color sensitive pixels 2443.

Step 906, obtaining a first target image based on the first combined image and the second combined image.

The electronic device may read pixel values from the first combined image and the second combined image according to a preset pixel reading manner to generate a first target image. The preset pixel reading mode is a preset pixel reading mode.

In this embodiment, in the first resolution mode, the first pixel values read out by combining the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by combining the plurality of first color photosensitive pixels corresponding to the color filter, and the third pixel values read out by combining the plurality of third color photosensitive pixels corresponding to the color filter can fuse the full-color channel information into the image, so that the overall light incoming amount is improved, and a first combined image with more information and clearer detail analysis can be generated. According to the fourth pixel value read by combining the plurality of second color photosensitive pixels corresponding to the color filter, the second color photosensitive pixels can be separated to form a second combined image, so that the positions of the second color photosensitive pixels in the second combined image and the positions of the first color photosensitive pixels in the first combined image are kept consistent. Based on the first combined image and the second combined image, a first target image is obtained, and the first color photosensitive pixels in the first combined image and the second color photosensitive pixels in the second combined image can be arranged in a mixed mode, so that the generated first target image is clearer in color. And the pixel binning readout results in a reduced size of the generated first target image, low power consumption required to generate the image.

In one embodiment, obtaining a first target image based on the first combined image and the second combined image includes:

and traversing the pixel values in the first combined image, reading a fourth pixel value which is the same as the second pixel value from the second combined image under the condition that the traversed pixel value is the second pixel value, and adjusting the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain the first target image.

Specifically, the position of each second pixel value in the first combined image is the same as the position of each fourth pixel value in the second combined image. The electronic device may traverse each pixel value in the first combined image, in each traversal, determining whether the currently traversed pixel value is the second pixel value. In the case that the currently traversed pixel value is not the second pixel value, the next pixel value is continued to be traversed. And under the condition that the current traversed pixel value is the second pixel value, determining the position of the current traversed second pixel value, reading a fourth pixel value which is the same as the current traversed second pixel value from the second combined image, adjusting the read fourth pixel value to be adjacent to the second pixel value, and continuing to traverse the next pixel value until the pixel values in the first combined image are traversed, and generating the first target image.

In this embodiment, in the first target image, adjacent second pixel values and fourth pixel values indicate that the positions of the second pixel values and the fourth pixel values in the first target image are the same, that is, the coordinates of the second pixel values and the fourth pixel values in the first target image are the same.

In one embodiment, where the currently traversed pixel value is not the second pixel value, the next pixel value is continued to be traversed. And in the case that the currently traversed pixel value is the second pixel value, determining the position of the currently traversed second pixel value, reading a fourth pixel value which is the same as the position of the currently traversed second pixel value from the second combined image, and adjusting the read fourth pixel value to be horizontally adjacent to the second pixel value. The fourth pixel value is horizontally adjacent to the second pixel value, and may be an adjacent previous pixel value in the horizontal direction with the fourth pixel value as the second pixel value or an adjacent next pixel value in the horizontal direction with the fourth pixel value as the second pixel value.

In this embodiment, the pixel values in the first combined image are traversed, and in the case that the traversed pixel values are the second pixel values, the fourth pixel values with the same positions as the second pixel values are read from the second combined image, and the read fourth pixel values are adjusted to be adjacent to the second pixel values, so that the first color photosensitive pixels and the second color photosensitive pixels are arranged in a mixed manner, the color resolution is improved, and each row and each column of the generated first target image have the first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels, that is, each row and each column of the target image have the RGB pixels, so that the risk of false colors can be effectively reduced.

As shown in fig. 10, in the first resolution mode, an original image 1002 is obtained by an image sensor, locally identical pixels in the original image 1002 are averaged, and binning is performed, that is, locally identical 4 panchromatic pixels are combined to read out a first pixel value, locally identical 2 first color photosensitive pixels are combined to read out a second pixel value, and locally identical 4 third color photosensitive pixels are combined to read out a third pixel value, so as to obtain a first combined image 1004. And, the second combined image 1006 is obtained by combining and reading out the fourth pixel values from the 2 locally identical second color photosensitive pixels. To keep the second combined image 1006 and the first combined image 1004 the same size, the second combined image 1006 may be generated by a fourth pixel value and a null pixel.

For example, the full-color pixel is a w pixel, the first color photosensitive pixel is an R pixel, the third color photosensitive pixel is a G pixel, and the second color photosensitive pixel is a B pixel.

The pixel values are read from the first combined image 1004 and the second combined image 1006 such that each fourth pixel value is taken as a subsequent pixel value adjacent to the second pixel value, resulting in a first target image 1008.

In one embodiment, the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; in the first resolution mode, according to the first pixel values read by merging the plurality of full-color pixels corresponding to the full-color filters in the filter set, the second pixel values read by merging the plurality of first color photosensitive pixels corresponding to the color filters, and the third pixel values read by merging the plurality of third color photosensitive pixels corresponding to the color filters, a first merged image is obtained, which includes:

In a first resolution mode, according to first pixel values read out in a merging mode according to a plurality of full-color pixels corresponding to full-color filters in a filter set, according to second pixel values read out in a merging mode according to first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter set, according to third pixel values read out in a merging mode according to third color photosensitive pixels corresponding to a plurality of third color sub-filters in a second filter set containing the first color sub-filters, and according to fourth pixel values read out in a merging mode according to second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter set, a first merged image is obtained;

Combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image, wherein the second combined image comprises: and according to the fourth pixel value which is read out in a merging way by the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way by the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, and according to the third pixel value which is read out in a merging way by the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the second color sub-filters, obtaining a second merged image.

Specifically, the plurality of filter sets includes a first filter set 231 and a second filter set 232, and at least one color filter 234 in the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342. The color filters 234 of the second filter set 232 include a third color sub-filter 2343, and further include a first color sub-filter 2341 or a second color sub-filter 2342.

In the first resolution mode, the electronics combine the plurality of panchromatic pixels 2431 corresponding to the same panchromatic filter 233 to read out the first pixel value.

For the color filters 234 in the first filter set 231, the electronic device combines and reads the second pixel values from the first color sensitive pixels 2441 corresponding to the plurality of first color sub-filters 2341 of the same color filter 234, and combines and reads the fourth pixel values from the second color sensitive pixels 2442 corresponding to the plurality of second color sub-filters 2342 of the same color filter 234, and then the same color filter 234 corresponds to the second pixel values and the fourth pixel values.

In the case that the color filters 234 in the second filter set 232 include the first color sub-filters 2341 and the third color sub-filters 2343, the electronic device combines the first color sensitive pixels 2441 corresponding to the first color sub-filters 2341 of the same color filter 234 to read out the second pixel values, and combines the third color sensitive pixels 2443 corresponding to the third color sub-filters 2343 to read out the third pixel values, and then the same color filter 234 corresponds to the second pixel values and the third pixel values.

When the color filters 234 in the second filter set 232 include the second color sub-filters 2342 and the third color sub-filters 2343, the electronic device combines the second color sensitive pixels 2442 corresponding to the plurality of second color sub-filters 2342 of the same color filter 234 to read out the fourth pixel value, and combines the third color sensitive pixels 2443 corresponding to the plurality of third color sub-filters 2343 of the same color filter 234 to read out the third pixel value, and then the same color filter 234 corresponds to the fourth pixel value and the third pixel value.

For the color filters 234 corresponding to the second pixel value and the fourth pixel value, the electronic device may read the second pixel value; for the color filter 234 corresponding to the second pixel value and the third pixel value, the electronic device may read the third pixel value; for the color filter 234 corresponding to the fourth pixel value and the third pixel value, the electronic device may read the fourth pixel value; the electronic device generates a first combined image according to the read second pixel value, third pixel value, fourth pixel value and all first pixel values.

For the color filters 234 corresponding to the second pixel value and the fourth pixel value, the electronic device may obtain the fourth pixel value; for color filters 234 corresponding to the second pixel value and the third pixel value, the electronic device may obtain the second pixel value; for the color filter 234 corresponding to the fourth pixel value and the third pixel value, the electronic device may obtain the third pixel value; the electronic device generates a second combined image according to the acquired fourth pixel value, second pixel value, third pixel value and null pixel.

The electronic device reads pixel values from the first combined image and the second combined image, generating a first target image.

In the embodiment, the color filter 234 of the second filter set 232 includes a third color sub-filter 2343, and further includes a first color sub-filter 2341 or a second color sub-filter 2342, and in the first resolution mode, according to the first pixel values of the merging readout of the plurality of full-color pixels corresponding to the full-color filters in the filter set, the second pixel values of the merging readout of the plurality of first color photosensitive pixels corresponding to the plurality of first color sub-filters in the first filter set, the third pixel values of the merging readout of the plurality of third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set including the first color sub-filter, and the fourth pixel values of the merging readout of the plurality of second color photosensitive pixels corresponding to the plurality of second color sub-filters in the second filter set, the merging readout of the obtained first merged image includes all the pixels, so as to improve the resolution capability of the color channel. And obtaining a second combined image according to the second pixel values read by combining the plurality of first color photosensitive pixels corresponding to the color filter, the third pixel values read by combining the plurality of third color photosensitive pixels corresponding to the color filter and the fourth pixel values read by combining the plurality of second color photosensitive pixels corresponding to the color filter, so that the pixel distribution of each color in the first target image generated according to the first combined image and the second combined image is more uniform, and the definition of the image is effectively improved.

As shown in fig. 11, in the first resolution mode, an original image 1102 is obtained by an image sensor, partial 4 adjacent full-color pixels in the original image 1102 are combined and read out to first pixel values, 2 first color photosensitive pixels on a diagonal are combined and read out to second pixel values, 2 third color photosensitive pixels on a diagonal are combined and read out to third pixel values, and 2 second color photosensitive pixels on a diagonal are combined and read out to fourth pixel values; reading the second pixel value, the third pixel value, the fourth pixel value and all the first pixel values to generate a first combined image 1104; the remaining fourth pixel value, second pixel value, and third pixel value are read and combined with the empty pixel to generate a second combined image 1106. The electronic device may generate a first target image based on the first combined image 1104 and the second combined image 1106.

In one embodiment, as shown in fig. 12, the method further comprises:

Step 1202, in a second resolution mode, merging the read first pixel values according to a plurality of full-color pixels corresponding to the full-color filters in the filter set, merging the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the color filters, and merging the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the color filters, so as to obtain a first merged image; the second resolution mode corresponds to a resolution less than the resolution of the first resolution mode.

The second resolution mode is a mode used in a scene with lower resolution requirements, and is a two-level pixel merging read-out mode with low resolution, low power consumption, high signal-to-noise ratio and high frame rate. The resolution and power consumption corresponding to the second resolution mode are smaller than those corresponding to the first resolution mode. The signal-to-noise ratio and the frame rate corresponding to the second resolution mode are larger than those corresponding to the first resolution mode.

The second resolution mode may be, but not limited to, a preview mode at the time of image capturing, a preview mode at the time of video capturing, or a scene with low resolution requirements such as a night scene mode for image capturing and video capturing under night scenes. The preview mode of video capturing is, for example, 1080p video preview, application video preview, or the like.

In the case where a photographing instruction is received, it is determined whether the photographing instruction is a preview photographing. When the photographing instruction is a preview photographing, the second resolution mode is triggered. Or the electronic equipment detects whether the current environment is a night scene or not, and triggers the second resolution mode under the condition that the current environment is the night scene. Or when the user selects the second resolution mode, triggering a read-out mode corresponding to the second resolution mode.

In the second resolution mode, the light transmitted through the sub-filter 2331 in the full-color filter 233 is projected onto the corresponding full-color pixel 2431, and the full-color pixel 2431 receives the light transmitted through the sub-filter 2331 to generate an electrical signal. The light transmitted by the first color sub-filter 2341 in the color filter 234 is projected onto the corresponding first color photosensitive pixel 2441, and the first color photosensitive pixel 2441 passes through the light of the corresponding first color sub-filter 2341 to generate an electrical signal. The light transmitted through the second color sub-filter 2342 in the color filter 234 is projected onto the corresponding second color photosensitive pixel 2442, and the second color photosensitive pixel 2442 passes through the light of the corresponding second color sub-filter 2342 to generate an electrical signal. The light transmitted through the third color sub-filter 2343 in the color filter 234 is projected onto the corresponding third color photosensitive pixel 2443, and the third color photosensitive pixel 2443 passes through the light of the corresponding third color sub-filter 2343 to generate an electrical signal.

And step 1204, merging the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second merged image.

In step 1206, the readout fifth pixel value is combined based on the plurality of full-color pixels in the ninth diagonal direction in the first combined image, resulting in a full-color combined image.

The electronic device merges and reads out the fifth pixel values from the plurality of full-color pixels in the ninth diagonal direction in the first merged image, and generates a full-color merged image from each of the fifth pixel values. The full-color combined image is composed of full-color pixels.

Step 1208, obtaining a third combined image based on the sixth pixel values read out by combining the plurality of first color photosensitive pixels in the tenth diagonal direction and the seventh pixel values read out by combining the plurality of third color photosensitive pixels in the tenth diagonal direction, where the ninth diagonal direction is different from the tenth diagonal direction.

The electronic device merges and reads out a sixth pixel value from a plurality of first color photosensitive pixels in a tenth diagonal direction in the first merged image, merges and reads out a seventh pixel value from a plurality of third color photosensitive pixels in the tenth diagonal direction, and generates a third merged image according to the sixth pixel value and the seventh pixel value. The third combined image includes first color photosensitive pixels and third color photosensitive pixels. The ninth diagonal direction is different from the tenth diagonal direction, and specifically, the seventh diagonal direction is not parallel to the tenth diagonal direction, or the ninth diagonal direction is perpendicular to the tenth diagonal direction, or the like.

In step 1210, a fourth combined image is obtained based on the read eighth pixel values combined by the plurality of second color photosensitive pixels in the tenth diagonal direction in the second combined image.

The electronic device merges and reads out eighth pixel values from a plurality of second color photosensitive pixels in a tenth diagonal direction in the second merged image. A fourth combined image is generated based on each eighth pixel value.

In one embodiment, the electronics merge and read out the eighth pixel values from the 2 second color-sensitive pixels in the tenth diagonal direction in the second merged image, and generate a fourth merged image based on each of the eighth pixel values and the empty pixels.

Step 1212, obtaining a second target image based on the full-color combined image, the third combined image, and the fourth combined image.

The electronic device may generate the second target image based on reading pixel values from the full-color combined image, the third combined image, and the fourth combined image.

In this embodiment, in the second resolution mode, according to the first pixel values read by merging the plurality of full-color pixels corresponding to the full-color filters in the filter set, the second pixel values read by merging the plurality of color pixels corresponding to the color filters, and the third pixel values read by merging the plurality of third color photosensitive pixels corresponding to the color filters, a first merged image is obtained, and the fourth pixel values read by merging the plurality of second color photosensitive pixels corresponding to the color filters are obtained, so that the second merged image can be separated out as an independent image and the resolution of the image is reduced. The full-color combined image based on the fifth pixel value of the plurality of full-color pixels in the ninth diagonal direction in the first combined image is combined and read, and the full-color pixels can be separated from the first combined image while the size of the image is reduced to reduce power consumption. And obtaining a third combined image containing the first color photosensitive pixels and the third color photosensitive pixels based on the sixth pixel value read by combining the plurality of first color photosensitive pixels in the tenth diagonal direction and the seventh pixel value read by combining the plurality of third color photosensitive pixels in the tenth diagonal direction in the first combined image, wherein the pixel reading mode of all-in-one is adopted to reduce the generated image noise. The size of the fourth combined image is made to coincide with the size of the third combined image, full-color combined image, based on the eighth pixel value read out by combining the plurality of second-color photosensitive pixels in the tenth diagonal direction in the second combined image. Based on the full-color combined image, the third combined image and the fourth combined image, different first-color photosensitive pixels and second-color photosensitive pixels can be arranged in a mixed mode, so that RGB pixels in the generated second target image are distributed more uniformly, and the image quality is higher. In addition, the resolution of the obtained second target image is further reduced, the panchromatic pixels have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of the combined output of the two-stage pixels is achieved.

In one embodiment, obtaining the second target image based on the full-color combined image, the third combined image, and the fourth combined image includes:

Traversing pixels in the third combined image; in the case where the traversed pixel value is the sixth pixel value or the seventh pixel value, reading a fifth pixel value corresponding to the traversed pixel value from the full-color combined image, and adjusting the read fifth pixel value to be adjacent to the traversed pixel value; and under the condition that the traversed pixel value is the sixth pixel value, reading an eighth pixel value which is the same as the sixth pixel value from the fourth combined image, and adjusting the read eighth pixel value to be adjacent to the sixth pixel value until the pixel values in the full-color combined image, the third combined image and the fourth combined image are all read, so as to obtain a second target image.

Specifically, the position of each sixth pixel value in the third combined image is the same as the position of each eighth pixel value in the fourth combined image. The positions of the full-color pixels in the full-color combined image and the positions of the sixth pixel values in the third combined image and the positions of the eighth pixel values in the fourth combined image are the same.

The electronics can traverse the pixel values in the third combined image, with each traversal determining whether the currently traversed pixel value is a sixth pixel value or a seventh pixel value. In the case that the currently traversed pixel value is not the sixth pixel value and is not the seventh pixel value, the next pixel value is continued to be traversed.

In the case where the currently traversed pixel value is the sixth pixel value or the seventh pixel value, the position of the currently traversed pixel value is determined, a fifth pixel value corresponding to the traversed pixel value is read from the full-color combined image, and the read fifth pixel value is adjusted to be adjacent to the currently traversed pixel value. And when the pixel value traversed currently is the sixth pixel value, reading an eighth pixel value which is the same as the sixth pixel value from the fourth combined image, adjusting the read eighth pixel value to be adjacent to the sixth pixel value, and then continuing to traverse the next pixel value until the pixel values in the full-color combined image, the third combined image and the fourth combined image are all read, and obtaining a second target image.

In this embodiment, if the sixth pixel value and the eighth pixel value are adjacent to each other in the second target image, it indicates that the positions of the sixth pixel value and the eighth pixel value in the second target image are the same, that is, that the coordinates of the sixth pixel value and the eighth pixel value in the second target image are the same.

In one embodiment, the fifth pixel value is adjacent to the pixel value currently traversed, which may be the fifth pixel value horizontally adjacent to the pixel value currently traversed, specifically, may be the adjacent previous pixel value or the adjacent next pixel value horizontally adjacent to the fifth pixel value as the pixel value currently traversed.

The eighth pixel value is adjacent to the sixth pixel value, and may be horizontally adjacent to the sixth pixel value. The eighth pixel value is horizontally adjacent to the sixth pixel value, and may be an adjacent previous pixel value in the horizontal direction with the eighth pixel value as the sixth pixel value or an adjacent next pixel value in the horizontal direction with the eighth pixel value as the sixth pixel value.

In one embodiment, in the case where the fifth pixel value is an adjacent previous pixel value of the sixth pixel value in the horizontal direction, the eighth pixel value is an adjacent next pixel value of the sixth pixel value in the horizontal direction.

In this embodiment, the pixels in the third combined image are traversed, and in the case where the traversed pixel value is the sixth pixel value or the seventh pixel value, the fifth pixel value corresponding to the traversed pixel value is read from the full-color combined image, and the read fifth pixel value is adjusted to be adjacent to the traversed pixel value, so that full-color pixels are introduced into the image, and the light input amount is improved. In the case that the traversed pixel value is the sixth pixel value, an eighth pixel value which is the same as the sixth pixel value is read from the fourth combined image, and the read eighth pixel value is adjusted to be adjacent to the sixth pixel value, so that the first color photosensitive pixels and the second color photosensitive pixels are mixed and arranged, the color resolution capability is improved, each row and each column of the generated second target image has the first color photosensitive pixels, the second color photosensitive pixels and the second color photosensitive pixels, namely each row and each column of the target image has RGB pixels, and the risk of false colors can be effectively reduced.

As shown in fig. 13, in the second resolution mode, an original image 1302 is obtained by an image sensor, and the first combined image 1304 is obtained by combining and reading out the first pixel values of the locally identical 4 full-color pixels, the second pixel values of the locally identical 2 first color photosensitive pixels, and the third pixel values of the locally identical 4 third color photosensitive pixels in the original image 1302. And, the 2 second color photosensitive pixels which are locally the same are combined and read out the fourth pixel value, resulting in a second combined image 1306. To keep the second combined image 1306 and the first combined image 1304 the same size, the second combined image 1306 may be generated with fourth pixel values and empty pixels.

The fifth pixel value is read out by merging 2 panchromatic pixels in the ninth diagonal D9 direction in the first merged image 1304, resulting in a panchromatic merged image 1308. The sixth pixel value is read out by merging 2 first color photosensitive pixels in the direction of the tenth diagonal line D10 in the first merged image 1304, and the seventh pixel value is read out by merging 2 third color photosensitive pixels in the direction of the tenth diagonal line D10, resulting in a third merged image 1310.

The fourth combined image 1312 is obtained based on the read-out eighth pixel values of the 2 second-color photosensitive pixels in the direction of the tenth diagonal D10 in the second combined image 1306.

The pixel in the third combined image 1310 is traversed, and in the case where the traversed pixel value is the sixth pixel value or the seventh pixel value, a fifth pixel value corresponding to the traversed pixel value is read from the full-color combined image 1308 and the read fifth pixel value is taken as the previous pixel value adjacent to the traversed pixel value. And, when the traversed pixel value is the sixth pixel value, the eighth pixel value having the same position as the sixth pixel value is read from the fourth combined image 1312, and the read eighth pixel value is used as the next pixel value adjacent to the sixth pixel value, until all the pixel values in the third combined image are traversed, and then the second target image 1314 is obtained.

In one implementation, in the second resolution mode, the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; according to the first pixel values read out by merging a plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by merging a plurality of first-color photosensitive pixels corresponding to the color filter, and the third pixel values read out by merging a plurality of third-color photosensitive pixels corresponding to the color filter, a first merged image is obtained, which comprises:

In the second resolution mode, according to the first pixel values which are read out in a merging mode according to a plurality of full-color pixels corresponding to the full-color filters in the filter set, according to the second pixel values which are read out in a merging mode according to the first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter set, according to the third pixel values which are read out in a merging mode according to the third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter set containing the first color sub-filters, and according to the fourth pixel values which are read out in a merging mode according to the second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter set, a first merged image is obtained;

Combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image, wherein the second combined image comprises:

According to the fourth pixel value which is read out in a merging way and is corresponding to the second color photosensitive pixels of the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way and is corresponding to the first color photosensitive pixels of the plurality of first color sub-filters in the second filter set, and according to the third pixel value which is read out in a merging way and is corresponding to the third color photosensitive pixels of the plurality of third color sub-filters in the second filter set containing the second color sub-filters, a second merged image is obtained;

obtaining a third combined image based on the sixth pixel value read out by combining the plurality of first color photosensitive pixels in the tenth diagonal direction and the seventh pixel value read out by combining the plurality of third color photosensitive pixels in the tenth diagonal direction in the first combined image, including: obtaining a full-color combined image and a third combined image, respectively, based on a fifth pixel value of the combined readout of the plurality of full-color pixels in the ninth diagonal direction, a sixth pixel value of the combined readout of the plurality of first-color photosensitive pixels in the tenth diagonal direction, a seventh pixel value of the combined readout of the plurality of third-color photosensitive pixels in the tenth diagonal direction, and an eighth pixel value of the combined readout of the plurality of second-color photosensitive pixels in the tenth diagonal direction in the first combined image;

Obtaining a fourth combined image based on the eighth pixel value read out by combining the plurality of second color photosensitive pixels in the tenth diagonal direction in the second combined image, including: the fourth combined image is obtained based on the sixth pixel value, which is read out by combining the plurality of first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value, which is read out by combining the plurality of third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value, which is read out by combining the plurality of second-color photosensitive pixels in the tenth diagonal direction, in the second combined image.

Specifically, the plurality of filter sets includes a first filter set 231 and a second filter set 232, and the color filters 234 in the first filter set 231 include a first color sub-filter 2341 and a second color sub-filter 2342. The color filters 234 of the second filter set 232 include a third color sub-filter 2343, and further include a first color sub-filter 2341 or a second color sub-filter 2342.

In the second resolution mode, the electronics combine the plurality of panchromatic pixels 2431 corresponding to the same panchromatic filter 233 to read out the first pixel value.

For the color filters 234 corresponding to the second pixel value and the fourth pixel value, the electronic device may read the second pixel value; for the color filter 234 corresponding to the second pixel value and the third pixel value, the electronic device may read the third pixel value; for the color filter 234 corresponding to the fourth pixel value and the third pixel value, the electronic device may read the fourth pixel value; the electronic device generates a first combined image according to the read second pixel values, third pixel values, fourth pixel values and all first pixel values. The first combined image includes full-color pixels, first color photosensitive pixels, second color photosensitive pixels, and third color photosensitive pixels.

For the first combined image, the electronic device combines and reads out the fifth pixel values from the plurality of full-color pixels in the ninth diagonal direction, and generates a full-color combined image from each of the fifth pixel values. The full-color combined image is composed of full-color pixels.

For the first combined image, the electronic device combines the plurality of first-color photosensitive pixels in the tenth diagonal direction in the first combined image to read out a sixth pixel value, combines the plurality of third-color photosensitive pixels in the tenth diagonal direction in the seventh pixel value, and combines the plurality of second-color photosensitive pixels in the tenth diagonal direction in the eighth pixel value. And generating a third combined image according to the read-out sixth pixel value, seventh pixel value and eighth pixel value. The third combined image comprises a first color photosensitive pixel, a second color photosensitive pixel and a third color photosensitive pixel.

For the second combined image, the electronic device combines the plurality of first color photosensitive pixels in the tenth diagonal direction in the second combined image to read out a sixth pixel value, the plurality of third color photosensitive pixels in the tenth diagonal direction in combination to read out a seventh pixel value, and the plurality of second color photosensitive pixels in the tenth diagonal direction in combination to read out an eighth pixel value. And generating a fourth combined image according to the read-out sixth pixel value, the seventh pixel value and the eighth pixel value and the empty pixel. The fourth combined image comprises a first color photosensitive pixel, a second color photosensitive pixel, a third color photosensitive pixel and an empty pixel.

The electronic device may generate a second target image from the full-color combined image, the third combined image, and the fourth combined image.

In this embodiment, in the second resolution mode, according to the first pixel values read out by merging the full-color pixels corresponding to the full-color filters in the filter set, according to the second pixel values read out by merging the first color photosensitive pixels corresponding to the first color sub-filters in the first filter set, according to the third pixel values read out by merging the third color photosensitive pixels in the second filter set containing the first color sub-filters, and according to the fourth pixel values read out by merging the third color photosensitive pixels corresponding to the second color sub-filters in the second filter set, a first merged image is obtained, and according to the fourth pixel values read out by merging the second color photosensitive pixels corresponding to the second color sub-filters in the first filter set, according to the second pixel values read out by merging the first color photosensitive pixels corresponding to the first color sub-filters in the second filter set, and according to the third pixel values read out by merging the third color photosensitive pixels corresponding to the third color sub-filters in the second filter set containing the second color sub-filters, the second pixel values read out by merging the second color photosensitive pixels corresponding to the second color sub-filters can be separated from each other, and the two photo-merged images can be formed at the same positions.

The size of the image can be reduced while separating the full-color pixels from the first combined image based on the fifth pixel value of the plurality of full-color pixels in the ninth diagonal direction in the first combined image to combine readout, so as to reduce power consumption.

And obtaining a third combined image containing the first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels based on the sixth pixel value read out by combining the plurality of first color photosensitive pixels in the tenth diagonal direction in the first combined image, the seventh pixel value read out by combining the plurality of third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by combining the plurality of second color photosensitive pixels in the tenth diagonal direction, so that the distribution of RGB pixels in the image is more uniform. Based on the sixth pixel value read out by the combination of the plurality of first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by the combination of the plurality of third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by the combination of the plurality of second-color photosensitive pixels in the tenth diagonal direction in the second combined image, a fourth combined image is obtained such that the size of the fourth combined image is kept identical to the size of the third combined image, the full-color combined image. Based on the full-color combined image, the third combined image and the fourth combined image, different first-color photosensitive pixels, second-color photosensitive pixels and third-color photosensitive pixels can be arranged in a mixed mode, so that RGB pixels in the generated second target image are distributed more uniformly, and the image quality is higher. In addition, the resolution of the obtained second target image is further reduced, the panchromatic pixels have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of the combined output of the two-stage pixels is achieved.

As shown in fig. 14, in the second resolution mode, an original image 1402 is obtained by an image sensor, partial 4 adjacent full-color pixels in the original image 1402 are merged to read out first pixel values, 2 first-color photosensitive pixels on a diagonal are merged to read out second pixel values, 2 third-color photosensitive pixels on a diagonal are merged to read out third pixel values, and 2 second-color photosensitive pixels on a diagonal are merged to read out fourth pixel values; reading the second pixel value, the third pixel value, the fourth pixel value, and all the first pixel values to generate a first combined image 1404; the remaining fourth pixel value, second pixel value, and third pixel value are read and combined with the null pixel to generate a second combined image 1406.

The fifth pixel value is read out by merging 2 full-color pixels in the ninth diagonal D9 direction in the first merged image 1404, resulting in a full-color merged image 1408. The sixth pixel value is read out by merging 2 first-color photosensitive pixels in the direction of the tenth diagonal D10 in the first merged image 1404, the seventh pixel value is read out by merging 2 third-color photosensitive pixels in the direction of the tenth diagonal D10, and the eighth pixel value is read out by merging 2 second-color photosensitive pixels in the direction of the tenth diagonal D10. A third combined image 1410 is generated from the read out sixth pixel value, seventh pixel value, and eighth pixel value.

For the second combined image 1406, 2 first color photosensitive pixels in the direction of the tenth diagonal line D10 in the second combined image 1406 are combined to read out a sixth pixel value, and 2 third color photosensitive pixels in the direction of the tenth diagonal line D10 are combined to read out a seventh pixel value, and 2 second color photosensitive pixels in the direction of the tenth diagonal line D10 are combined to read out an eighth pixel value. A fourth combined image 1412 is generated from the read sixth pixel value, seventh pixel value, and eighth pixel value.

The electronic device may generate a second target image based on the full-color combined image 1408, the third combined image 1410, and the fourth combined image 1412.

In one embodiment, the method further comprises: in the second resolution mode, merging the read first pixel values according to a plurality of panchromatic pixels corresponding to the filter set to obtain a panchromatic image; combining the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the filter set and combining the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the filter set to obtain a fifth combined image; combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the filter set to obtain a sixth combined image; a third target image is obtained based on the full-color image, the fifth combined image, and the sixth combined image.

The plurality of filter sets includes a first filter set 231 and a second filter set 232, and at least one color filter 234 in the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342, so that at least one color filter 234 in the first filter set 231 corresponds to the first color sensitive pixel 2441 and the second color sensitive pixel 2442. The color filters 234 in the second filter set 232 include at least a third color sub-filter 2343, and the color filters 234 in the second filter set 232 correspond to at least the third color sensitive pixels 2443.

In the second resolution mode, the electronic device combines and reads out the first pixel values of the panchromatic pixels 2431 corresponding to all the panchromatic filters 233 in the same filter set, and obtains a panchromatic image according to each first pixel value. Further, the electronic device respectively combines and reads out the first pixel values from the full-color pixels 2431 corresponding to all the full-color filters 233 in each first filter set 231, so that one first filter set 231 corresponds to one first pixel value, so as to obtain the first pixel values respectively corresponding to each first filter set 231. The electronic device respectively combines and reads out the first pixel values from the full-color pixels 2431 corresponding to all the full-color filters 233 in each second filter set 232, so that one second filter set 232 corresponds to one first pixel value, and the first pixel values corresponding to each second filter set 232 are obtained. And obtaining a full-color image according to the first pixel value corresponding to each first filter set 231 and the first pixel value corresponding to each second filter set 232.

The electronic device combines the first color sensitive pixels 2441 corresponding to the plurality of first color sub-filters 2341 of all the color filters 234 in the same filter set to read out the second pixel value. The electronic device combines the third color sensitive pixels 2443 corresponding to the plurality of third color sub-filters 2343 of all the color filters 234 in the same filter set to read out the third pixel value. And the electronic equipment obtains a fifth combined image according to each second pixel value and each third pixel value.

Further, the electronic device respectively combines and reads the second pixel values from the first color sensitive pixels 2441 corresponding to the plurality of first color sub-filters 2341 of all the color filters 234 in each first filter set 231, so that one first filter set 231 corresponds to one second pixel value, so as to obtain the second pixel value corresponding to each first filter set 231. The electronic device respectively combines and reads the third pixel values of the third color sensitive pixels 2443 corresponding to the plurality of third color sub-filters 2343 of all the color filters 234 in each second filter set 232, so that one second filter set 232 corresponds to one third pixel value, so as to obtain the third pixel values respectively corresponding to the second filter sets 232. And obtaining a fifth combined image according to the second pixel value corresponding to each first filter set 231 and the third pixel value corresponding to each second filter set 232.

The electronic device combines the second color sensitive pixels 2442 corresponding to the second color sub-filters 2342 of all the color filters 234 in the same filter set to read out the fourth pixel value. Further, the electronic device respectively combines and reads the fourth pixel values from the second color sensitive pixels 2442 corresponding to the second color sub-filters 2342 of all the color filters 234 in each first filter set 231, so that one first filter set 231 corresponds to one fourth pixel value, so as to obtain the fourth pixel values corresponding to each first filter set 231. The electronic device obtains a sixth combined image according to the fourth pixel values corresponding to the first filter sets 231 respectively.

The electronic device may read pixel values from the full-color image, the fifth combined image, and the sixth combined image according to a preset pixel reading manner to generate a third target image.

In this embodiment, in the second resolution mode, the full-color image can be separated by combining the read first pixel values with the plurality of full-color pixels corresponding to the filter set; according to the second pixel values read out by combining the plurality of first color photosensitive pixels corresponding to the optical filter set and the third pixel values read out by combining the plurality of third color photosensitive pixels corresponding to the optical filter set, the first color photosensitive pixels and the third color photosensitive pixels can be separated to form a fifth combined image, and the image size can be reduced rapidly and effectively. And according to the fourth pixel values read out by combining the plurality of second color photosensitive pixels corresponding to the filter set, the second color photosensitive pixels can be separated to form a sixth combined image, so that the coordinate positions of the second color photosensitive pixels in the sixth combined image and the coordinate positions of the first color photosensitive pixels in the fifth combined image are kept consistent. And obtaining a third target image based on the full-color image, the fifth combined image and the sixth combined image, and fusing full-color channel information into the image to improve the integral light inlet amount, so that the third target image with more information and clearer detail analysis can be generated. And the size of the generated third target image is reduced by combining and reading out a plurality of pixels corresponding to the filter set, so that the power consumption required for generating the image is low.

In one embodiment, the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; combining the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the filter set and combining the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the filter set to obtain a fifth combined image, wherein the fifth combined image comprises:

obtaining a fifth combined image according to the second pixel value which is read out in a combined mode according to the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the first filter set, the third pixel value which is read out in a combined mode according to the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the first color sub-filters, and the fourth pixel value which is read out in a combined mode according to the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the second filter set;

And merging the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the filter set to obtain a sixth merged image, wherein the sixth merged image comprises:

And according to the fourth pixel value which is read out in a merging way by the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way by the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, and according to the third pixel value which is read out in a merging way by the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the second color sub-filters, obtaining a sixth merging image.

The plurality of filter sets includes a first filter set 231 and a second filter set 232, and at least one color filter 234 in the first filter set 231 includes a first color sub-filter 2341 and a second color sub-filter 2342, so that at least one color filter 234 in the first filter set 231 corresponds to the first color sensitive pixel 2441 and the second color sensitive pixel 2442. The color filters 234 in the second filter set 232 include a third color sub-filter 2343, and further include a first color sub-filter 2341 or a second color sub-filter 2342.

The electronic device respectively combines and reads the second pixel values from the first color sensitive pixels 2441 corresponding to the plurality of first color sub-filters 2341 of all the color filters 234 in each first filter set 231, so that one first filter set 231 corresponds to one second pixel value, and the second pixel values corresponding to each first filter set 231 are obtained.

For the second filter sets 232 including the first color sub-filter 2341 and the third color sub-filter 2343, the electronic device respectively combines and reads out the third pixel values of the third color sensitive pixels 2443 corresponding to the third color sub-filters 2343 of all the color filters 234 in each second filter set 232, so that one second filter set 232 including the first color sub-filter 2341 corresponds to one third pixel value, so as to obtain the third pixel value corresponding to each second filter set 232 including the first color sub-filter 2341.

For the second filter sets 232 including the second color sub-filters 2342 and the third color sub-filters 2343, the electronic device respectively combines the second color sensitive pixels 2442 corresponding to the second color sub-filters 2342 of all the color filters 234 in each second filter set 232 to read out the fourth pixel value, so that one second filter set 232 including the second color sub-filters 2342 corresponds to one fourth pixel value, so as to obtain the fourth pixel value corresponding to each second filter set 232 including the second color sub-filters 2342.

The electronic device obtains a fifth combined image according to the second pixel values corresponding to the first filter sets 231, the third pixel values corresponding to the second filter sets 232 including the first color sub-filters 2341, and the fourth pixel values corresponding to the second filter sets 232 including the second color sub-filters 2342.

The electronic device respectively combines and reads the fourth pixel values from the second color sensitive pixels 2442 corresponding to the second color sub-filters 2342 of all the color filters 234 in each first filter set 231, so that one first filter set 231 corresponds to one fourth pixel value, and a fourth pixel value corresponding to each first filter set 231 is obtained.

For the second filter sets 232 including the first color sub-filters 2341 and the third color sub-filters 2343, the electronic device respectively combines the first color sensitive pixels 2441 corresponding to the first color sub-filters 2341 of all the color filters 234 in each second filter set 232 to read out the second pixel values, so that one second filter set 232 including the first color sub-filters 2341 corresponds to one second pixel value, so as to obtain the second pixel value corresponding to each second filter set 232 including the first color sub-filters 2341.

For the second filter sets 232 including the second color sub-filters 2342 and the third color sub-filters 2343, the electronic device respectively combines the third color sensitive pixels 2443 corresponding to the third color sub-filters 2343 of all the color filters 234 in each second filter set 232 to read out the third pixel values, so that one second filter set 232 including the second color sub-filters 2342 corresponds to one third pixel value, so as to obtain the third pixel value corresponding to each second filter set 232 including the second color sub-filters 2342.

The electronic device obtains a sixth combined image according to the fourth pixel value corresponding to each first filter set 231, the second pixel value corresponding to each second filter set 232 including the first color sub-filter 2341, and the third pixel value corresponding to each second filter set 232 including the second color sub-filter 2342.

In this embodiment, in the second resolution mode, by combining the second pixel values read out according to the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the first filter set, the third pixel values read out according to the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set including the first color sub-filters, and the fourth pixel values read out according to the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the second filter set, a part of the first color photosensitive pixels, the third color photosensitive pixels, and the second color photosensitive pixels can be separated to form a fifth combined image, and the image size can be reduced by pixel combination, so as to reduce the processing power consumption. According to the fourth pixel value which is read out by merging the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set and the second pixel value which is read out by merging the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, according to the third pixel value which is read out by merging the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the second color sub-filters, part of the first color photosensitive pixels, the third color photosensitive pixels and the second color photosensitive pixels can be separated out to form a sixth merged image, so that the coordinate positions of the second color photosensitive pixels in the sixth merged image and the coordinate positions of the first color photosensitive pixels in the fifth merged image are kept consistent. And obtaining a third target image based on the full-color image, the fifth combined image and the sixth combined image, and fusing full-color channel information into the image to improve the integral light inlet amount, so that the third target image with more information and clearer detail analysis can be generated. And the size of the generated third target image is reduced by combining and reading out a plurality of pixels corresponding to the filter set, so that the power consumption required for generating the image is low.

In one embodiment, an image generating method is provided, applied to an image sensor of an electronic device, where the image sensor includes a filter array and a pixel array, the filter array includes a minimum repeating unit, the minimum repeating unit includes a plurality of filter sets, each filter set includes a color filter and a full-color filter, an amount of light entering the full-color filter is greater than an amount of light entering the full-color filter, the color filter and the full-color filter each include 4 sub-filters, the plurality of filter sets includes at least a first filter set, and at least one color filter in the first filter set includes a first color sub-filter and a second color sub-filter; the color filters in the second filter set comprise third color sub-filters, and the color filters in the second filter set further comprise first color sub-filters or second color sub-filters; the first color photosensitive pixels of the pixel array are arranged corresponding to the first color sub-filters of the filter array, the second color photosensitive pixels of the pixel array are arranged corresponding to the second color sub-filters of the filter array, the third color photosensitive pixels of the pixel array are arranged corresponding to the third color sub-filters of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

The method comprises the following steps:

In a first resolution mode, according to a first pixel value which is read out in a merging mode according to a plurality of full-color pixels corresponding to a full-color filter in the filter set, a second pixel value which is read out in a merging mode according to a plurality of first color photosensitive pixels corresponding to a color filter, and a third pixel value which is read out in a merging mode according to a plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained; combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image; and traversing the pixel values in the first combined image, reading a fourth pixel value which is the same as the second pixel value from the second combined image under the condition that the traversed pixel value is the second pixel value, and adjusting the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain the first target image.

Optionally, in the first resolution mode, according to the first pixel values read by merging the plurality of full-color pixels corresponding to the full-color filters in the filter set, according to the second pixel values read by merging the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the first filter set, according to the third pixel values read by merging the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set including the first color sub-filters, and according to the fourth pixel values read by merging the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the second filter set, obtaining a first merged image; according to the fourth pixel value which is read out in a merging way and is corresponding to the second color photosensitive pixels of the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way and is corresponding to the first color photosensitive pixels of the plurality of first color sub-filters in the second filter set, and according to the third pixel value which is read out in a merging way and is corresponding to the third color photosensitive pixels of the plurality of third color sub-filters in the second filter set containing the second color sub-filters, a second merged image is obtained; and obtaining a first target image based on the first combined image and the second combined image.

Optionally, in the second resolution mode, merging the read first pixel values according to a plurality of panchromatic pixels corresponding to the panchromatic filters in the filter set, merging the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the color filters, and merging the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the color filters to obtain a first merged image; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode; combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image; combining the read-out fifth pixel values based on a plurality of full-color pixels in a ninth diagonal direction in the first combined image to obtain a full-color combined image; obtaining a third combined image based on the sixth pixel value read by combining the plurality of first-color photosensitive pixels in the tenth diagonal direction in the first combined image and the seventh pixel value read by combining the plurality of third-color photosensitive pixels in the tenth diagonal direction, wherein the ninth diagonal direction is different from the tenth diagonal direction; obtaining a fourth combined image based on the eighth pixel value read out by combining the plurality of second color photosensitive pixels in the tenth diagonal direction in the second combined image; traversing pixels in the third combined image; in the case where the traversed pixel value is the sixth pixel value or the seventh pixel value, reading a fifth pixel value corresponding to the traversed pixel value from the full-color combined image, and adjusting the read fifth pixel value to be adjacent to the traversed pixel value; and under the condition that the traversed pixel value is the sixth pixel value, reading an eighth pixel value which is the same as the sixth pixel value from the fourth combined image, and adjusting the read eighth pixel value to be adjacent to the sixth pixel value until the pixel values in the full-color combined image, the third combined image and the fourth combined image are all read, so as to obtain a second target image.

Optionally, in the second resolution mode, according to the first pixel values read by merging the plurality of full-color pixels corresponding to the full-color filters in the filter set, according to the second pixel values read by merging the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the first filter set, according to the third pixel values read by merging the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set including the first color sub-filters, and according to the fourth pixel values read by merging the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the second filter set, obtaining a first merged image; according to the fourth pixel value which is read out in a merging way and is corresponding to the second color photosensitive pixels of the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way and is corresponding to the first color photosensitive pixels of the plurality of first color sub-filters in the second filter set, and according to the third pixel value which is read out in a merging way and is corresponding to the third color photosensitive pixels of the plurality of third color sub-filters in the second filter set containing the second color sub-filters, a second merged image is obtained; obtaining a full-color combined image and a third combined image, respectively, based on a fifth pixel value of a plurality of full-color pixel combined readouts in a ninth diagonal direction, a sixth pixel value of a plurality of first-color photosensitive pixel combined readouts in a tenth diagonal direction, a seventh pixel value of a plurality of third-color photosensitive pixel combined readouts in the tenth diagonal direction, and an eighth pixel value of a plurality of second-color photosensitive pixel combined readouts in the tenth diagonal direction in the first combined image; obtaining a fourth combined image based on the sixth pixel value of the combined readout of the plurality of first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value of the combined readout of the plurality of third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value of the combined readout of the plurality of second-color photosensitive pixels in the tenth diagonal direction in the second combined image; and obtaining a second target image based on the full-color combined image, the third combined image and the fourth combined image.

Optionally, in the second resolution mode, merging the read first pixel values according to a plurality of panchromatic pixels corresponding to the filter set to obtain a panchromatic image; combining the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the filter set and combining the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the filter set to obtain a fifth combined image; combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the filter set to obtain a sixth combined image; a third target image is obtained based on the full-color image, the fifth combined image, and the sixth combined image.

Optionally, in the second resolution mode, merging the read first pixel values according to a plurality of panchromatic pixels corresponding to the filter set to obtain a panchromatic image; obtaining a fifth combined image according to the second pixel value which is read out in a combined mode according to the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the first filter set, the third pixel value which is read out in a combined mode according to the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the first color sub-filters, and the fourth pixel value which is read out in a combined mode according to the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the second filter set; according to the fourth pixel value which is read out in a merging way by the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way by the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, according to the third pixel value which is read out in a merging way by the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the second color sub-filters, obtaining a sixth merging image; a third target image is obtained based on the full-color image, the fifth combined image, and the sixth combined image.

In this embodiment, an image sensor structure supporting two resolution outputs is provided, so that two resolution image output modes are provided, and different application scenarios can be adapted.

And under a scene with higher resolution requirements, a first resolution mode is used, and the full-color channel information can be fused into an image through combining and reading out first pixel values of a plurality of full-color pixels corresponding to the full-color filters in the filter set, combining and reading out second pixel values of a plurality of first color photosensitive pixels corresponding to the color filters and combining and reading out third pixel values of a plurality of third color photosensitive pixels corresponding to the color filters, so that the integral light inlet quantity is improved, and a first combined image with more information and clearer detail analysis can be generated. And merging the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filter to obtain a second merged image, so that the sizes of the second merged image and the first merged image are kept consistent, and therefore the positions of the second color photosensitive pixels in the second merged image are ensured to be the same as the positions of the first color photosensitive pixels in the first merged image, namely the positions of the fourth pixel values in the second merged image are ensured to be the same as the positions of the second pixel values in the first merged image.

The pixel values in the first combined image are traversed to adjust the fourth pixel value in the second combined image to be adjacent to the second pixel value with the same position, so that the first color photosensitive pixels and the second color photosensitive pixels are arranged in a mixed mode, color resolution is improved, each row and each column of the generated first target image are provided with the first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels, namely each row and each column of the target image are provided with the RGB pixels, and risks of false colors can be effectively reduced.

Or in the first resolution mode, according to the first pixel values read out by combining the plurality of full-color pixels corresponding to the full-color filters in the filter set, the second pixel values read out by combining the plurality of first color photosensitive pixels corresponding to the color filters, the third pixel values read out by combining the plurality of third color photosensitive pixels corresponding to the color filters, and the fourth pixel values read out by combining the plurality of second color photosensitive pixels corresponding to the color filters, the pixels of all colors are contained in the first combined image obtained by combining and reading, so that the resolution capability of a color channel can be improved. And obtaining a second combined image according to the second pixel values read by combining the plurality of first color photosensitive pixels corresponding to the color filter, the third pixel values read by combining the plurality of third color photosensitive pixels corresponding to the color filter and the fourth pixel values read by combining the plurality of second color photosensitive pixels corresponding to the color filter, so that the pixel distribution of each color in the first target image generated according to the first combined image and the second combined image is more uniform, and the definition of the image is effectively improved.

And under the scene with lower resolution requirements such as night view shooting, a second resolution mode is used, according to the first pixel values which are read out in a merging mode according to a plurality of full-color pixels corresponding to the full-color filters in the filter set, the second pixel values which are read out in a merging mode according to a plurality of color pixels corresponding to the color filters, and the third pixel values which are read out in a merging mode according to a plurality of third color photosensitive pixels corresponding to the color filters, a first merged image is obtained, and the fourth pixel values which are read out in a merging mode according to a plurality of second color photosensitive pixels corresponding to the color filters are obtained, so that the second color photosensitive pixels can be separated out to be used as an independent image, and the resolution of the image is reduced. The size of the image can be reduced while separating the full-color pixels from the first combined image based on the fifth pixel value of the plurality of full-color pixels in the ninth diagonal direction in the first combined image to combine readout, so as to reduce power consumption. And obtaining a third combined image containing the first color photosensitive pixels and the third color photosensitive pixels based on the sixth pixel value read by combining the plurality of first color photosensitive pixels in the tenth diagonal direction and the seventh pixel value read by combining the plurality of third color photosensitive pixels in the tenth diagonal direction in the first combined image, wherein the pixel reading mode of all-in-one is adopted to reduce the generated image noise. The size of the fourth combined image is made to coincide with the size of the third combined image, full-color combined image, based on the eighth pixel value read out by combining the plurality of second-color photosensitive pixels in the eighth diagonal direction in the second combined image. And traversing pixels in the third combined image, reading a fifth pixel value corresponding to the traversed pixel value from the full-color combined image when the traversed pixel value is the sixth pixel value or the seventh pixel value, and adjusting the read fifth pixel value to be adjacent to the traversed pixel value, so that full-color pixels are introduced into the image, and the light input quantity is improved. In the case that the traversed pixel value is the sixth pixel value, an eighth pixel value which is the same as the sixth pixel value is read from the fourth combined image, and the read eighth pixel value is adjusted to be adjacent to the sixth pixel value, so that the first color photosensitive pixels and the second color photosensitive pixels are mixed and arranged, the color resolution capability is improved, each row and each column of the generated second target image has the first color photosensitive pixels, the second color photosensitive pixels and the second color photosensitive pixels, namely each row and each column of the target image has RGB pixels, and the risk of false colors can be effectively reduced. In addition, the resolution of the obtained second target image is further reduced, the panchromatic pixels have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of the combined output of the two-stage pixels is achieved.

Or in the second resolution mode, according to the first pixel values read by combining the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read by combining the plurality of color pixels corresponding to the color filter, the third pixel values read by combining the plurality of third color photosensitive pixels corresponding to the color filter, and the fourth pixel values read by combining the plurality of third color photosensitive pixels corresponding to the color filter, the first combined image is obtained, and the second combined image is obtained by combining the second pixel values read by combining the plurality of first color photosensitive pixels corresponding to the color filter, the third pixel values read by combining the plurality of third color photosensitive pixels corresponding to the color filter, and the fourth pixel values read by combining the plurality of second color photosensitive pixels corresponding to the color filter.

The size of the image can be reduced while separating the full-color pixels from the first combined image based on the fifth pixel value of the plurality of full-color pixels in the ninth diagonal direction in the first combined image to combine readout, so as to reduce power consumption. And obtaining a third combined image containing the first color photosensitive pixels, the second color photosensitive pixels and the third color photosensitive pixels based on the sixth pixel value read out by combining the plurality of first color photosensitive pixels in the tenth diagonal direction in the first combined image, the seventh pixel value read out by combining the plurality of third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by combining the plurality of second color photosensitive pixels in the tenth diagonal direction, so that the distribution of RGB pixels in the image is more uniform. Based on the sixth pixel value read out by the combination of the plurality of first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by the combination of the plurality of third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by the combination of the plurality of second-color photosensitive pixels in the tenth diagonal direction in the second combined image, a fourth combined image is obtained such that the size of the fourth combined image is kept identical to the size of the third combined image, the full-color combined image. Based on the full-color combined image, the third combined image and the fourth combined image, different first-color photosensitive pixels, second-color photosensitive pixels and third-color photosensitive pixels can be arranged in a mixed mode, so that RGB pixels in the generated second target image are distributed more uniformly, and the image quality is higher. In addition, the resolution of the obtained second target image is further reduced, the panchromatic pixels have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of the combined output of the two-stage pixels is achieved.

It should be understood that, although the steps in the flowcharts of fig. 10 to 14 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps of fig. 10-14 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the sub-steps or stages are performed necessarily occur in sequence, but may be performed alternately or alternately with at least a portion of other steps or sub-steps or stages of other steps.

Fig. 15 is a block diagram showing the configuration of an image generating apparatus according to an embodiment. As shown in fig. 15, the image generating apparatus 1500 is applied to an image sensor, the image sensor includes a filter array and a pixel array, the filter array includes a minimal repeating unit, the minimal repeating unit includes a plurality of filter sets, each filter set includes a color filter and a full color filter, an amount of light entering the full color filter is greater than an amount of light entering the color filter, the color filter and the full color filter each include 4 sub-filters, the plurality of filter sets includes a first filter set and a second filter set, and at least one color filter in the first filter set includes a first color sub-filter and a second color sub-filter; the color filter in the second filter set at least comprises a third color sub-filter; the pixel array comprises a plurality of pixels, a first color photosensitive pixel of the pixel array is correspondingly arranged with a first color sub-filter of the filter array, a second color photosensitive pixel of the pixel array is correspondingly arranged with a second color sub-filter of the filter array, a third color photosensitive pixel of the pixel array is correspondingly arranged with a third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electric signal;

The image generating apparatus 1500 includes:

the first merging module 1502 is configured to, in a first resolution mode, merge the read first pixel values according to a plurality of full-color pixels corresponding to the full-color filters in the filter set, merge the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the color filters, and merge the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the color filters, so as to obtain a first merged image.

The second merging module 1504 is configured to merge the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters, so as to obtain a second merged image.

The image generating module 1506 is configured to obtain the first target image based on the first combined image and the second combined image.

In this embodiment, in the first resolution mode, the first pixel values read out by combining the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by combining the plurality of first color photosensitive pixels corresponding to the color filter, and the third pixel values read out by combining the plurality of third color photosensitive pixels corresponding to the color filter can fuse the full-color channel information into the image, so that the overall light incoming amount is improved, and a first combined image with more information and clearer detail analysis can be generated. And combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filter to obtain a second combined image, and obtaining a first target image based on the first combined image and the second combined image, wherein the first color photosensitive pixels in the first combined image and the second color photosensitive pixels in the second combined image can be mixed and arranged, so that the color of the generated first target image is clearer. And the pixel binning readout results in a reduced size of the generated first target image, low power consumption required to generate the image.

In one embodiment, the image generating module 1506 is further configured to traverse the pixel values in the first combined image, and in the case that the traversed pixel values are the second pixel values, read a fourth pixel value that is the same as the second pixel value from the second combined image, and adjust the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain the first target image.

In one embodiment, the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; the first merging module 1502 is further configured to, in the first resolution mode, merge the first pixel values read out according to a plurality of full-color pixels corresponding to full-color filters in the filter set, merge the second pixel values read out according to a first color photosensitive pixel corresponding to a plurality of first color sub-filters in the first filter set, merge the third pixel values read out according to a third color photosensitive pixel corresponding to a plurality of third color sub-filters in the second filter set including the first color sub-filters, and merge the fourth pixel values read out according to a second color photosensitive pixel corresponding to a plurality of second color sub-filters in the second filter set, thereby obtaining a first merged image;

The second merging module 1504 is further configured to merge the read fourth pixel values according to the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, merge the read second pixel values according to the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, and merge the read third pixel values according to the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set including the second color sub-filters, thereby obtaining a second merged image.

In one embodiment, the apparatus further comprises a third merge module;

The first merging module 1502 is further configured to, in the second resolution mode, obtain a first merged image according to a first pixel value that is read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter set, a second pixel value that is read out by merging a plurality of first color photosensitive pixels corresponding to the color filters, and a third pixel value that is read out by merging a plurality of third color photosensitive pixels corresponding to the color filters; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode;

The second merging module 1504 is further configured to merge the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters, to obtain a second merged image;

A third merging module, configured to obtain a full-color merged image based on the fifth pixel values read out by merging the plurality of full-color pixels in the ninth diagonal direction in the first merged image; obtaining a third combined image based on the sixth pixel value read by combining the plurality of first-color photosensitive pixels in the tenth diagonal direction in the first combined image and the seventh pixel value read by combining the plurality of third-color photosensitive pixels in the tenth diagonal direction, wherein the ninth diagonal direction is different from the tenth diagonal direction;

The third merging module is further used for merging the read eighth pixel values based on a plurality of second color photosensitive pixels in a tenth diagonal direction in the second merged image to obtain a fourth merged image;

The image generating module 1506 is further configured to obtain a second target image based on the full-color combined image, the third combined image, and the fourth combined image.

In this embodiment, in the second resolution mode, according to the first pixel values read by merging the plurality of full-color pixels corresponding to the full-color filters in the filter set, the second pixel values read by merging the plurality of color pixels corresponding to the color filters, and the third pixel values read by merging the plurality of third color photosensitive pixels corresponding to the color filters, a first merged image is obtained, and the fourth pixel values read by merging the plurality of second color photosensitive pixels corresponding to the color filters are obtained, so that the second merged image can be separated out as an independent image and the resolution of the image is reduced. The size of the image can be reduced while separating the full-color pixels from the first combined image based on the fifth pixel value of the plurality of full-color pixels in the ninth diagonal direction in the first combined image to combine readout, so as to reduce power consumption. And obtaining a third combined image containing the first color photosensitive pixels and the third color photosensitive pixels based on the sixth pixel value read by combining the plurality of first color photosensitive pixels in the tenth diagonal direction and the seventh pixel value read by combining the plurality of third color photosensitive pixels in the tenth diagonal direction in the first combined image, wherein the pixel reading mode of all-in-one is adopted to reduce the generated image noise. The size of the fourth combined image is made to coincide with the size of the third combined image, full-color combined image, based on the eighth pixel value read out by combining the plurality of second-color photosensitive pixels in the tenth diagonal direction in the second combined image. Based on the full-color combined image, the third combined image and the fourth combined image, different first-color photosensitive pixels and second-color photosensitive pixels can be arranged in a mixed mode, so that RGB pixels in the generated second target image are distributed more uniformly, and the image quality is higher. In addition, the resolution of the obtained second target image is further reduced, the panchromatic pixels have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of the combined output of the two-stage pixels is achieved.

In one embodiment, the image generation module 1506 is further configured to traverse pixels in the third combined image; in the case where the traversed pixel value is the sixth pixel value or the seventh pixel value, reading a fifth pixel value corresponding to the traversed pixel value from the full-color combined image, and adjusting the read fifth pixel value to be adjacent to the traversed pixel value; and under the condition that the traversed pixel value is the sixth pixel value, reading an eighth pixel value which is the same as the sixth pixel value from the fourth combined image, and adjusting the read eighth pixel value to be adjacent to the sixth pixel value until the pixel values in the full-color combined image, the third combined image and the fourth combined image are all read, so as to obtain a second target image.

In one embodiment, the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; the first merging module 1502 is further configured to, in the second resolution mode, obtain a first merged image according to a first pixel value that is read out by merging a plurality of full-color pixels corresponding to full-color filters in the filter set, a second pixel value that is read out by merging a first color photosensitive pixel corresponding to a plurality of first color sub-filters in the first filter set, a third pixel value that is read out by merging a third color photosensitive pixel corresponding to a plurality of third color sub-filters in the second filter set that includes the first color sub-filters, and a fourth pixel value that is read out by merging a second color photosensitive pixel corresponding to a plurality of second color sub-filters in the second filter set;

The second merging module 1504 is further configured to merge the read fourth pixel values according to second color photosensitive pixels corresponding to a plurality of second color sub-filters in the first filter set, merge the read second pixel values according to first color photosensitive pixels corresponding to a plurality of first color sub-filters in the second filter set, and merge the read third pixel values according to third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter set including the second color sub-filters, so as to obtain a second merged image;

a third merging module for merging the read-out sixth pixel value based on the plurality of first color photosensitive pixels in the tenth diagonal direction, the read-out seventh pixel value based on the plurality of third color photosensitive pixels in the tenth diagonal direction, and the read-out eighth pixel value based on the plurality of second color photosensitive pixels in the tenth diagonal direction, the third merged image;

The third merging module is further configured to obtain a fourth merged image based on the sixth pixel value read out by merging the plurality of first color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by merging the plurality of third color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by merging the plurality of second color photosensitive pixels in the tenth diagonal direction in the second merged image.

In one embodiment, the first combining module 1502 is further configured to combine the read first pixel values according to a plurality of panchromatic pixels corresponding to the optical filter set in the second resolution mode, to obtain a panchromatic image; combining the read second pixel values according to a plurality of first color photosensitive pixels corresponding to the filter set and combining the read third pixel values according to a plurality of third color photosensitive pixels corresponding to the filter set to obtain a fifth combined image;

The second merging module 1504 is further configured to merge the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the filter set, to obtain a sixth merged image;

the image generating module 1506 is further configured to obtain a third target image based on the full-color image, the fifth combined image, and the sixth combined image.

In one embodiment, the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; the first merging module 1502 is further configured to obtain a fifth merged image according to a second pixel value that is read by merging the first color sensitive pixels corresponding to the plurality of first color sub-filters in the first filter set, a third pixel value that is read by merging the third color sensitive pixels corresponding to the plurality of third color sub-filters in the second filter set that includes the first color sub-filters, and a fourth pixel value that is read by merging the second color sensitive pixels corresponding to the plurality of second color sub-filters in the second filter set;

The second merging module 1504 is further configured to merge the read fourth pixel values according to the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, merge the read second pixel values according to the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, and merge the read third pixel values according to the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set including the second color sub-filters, thereby obtaining a sixth merged image.

The division of the various modules in the image generation device described above is for illustration only, and in other embodiments, the image generation device may be divided into different modules as needed to perform all or part of the functions of the image generation device described above.

Fig. 16 is a schematic diagram showing an internal structure of the electronic device in one embodiment. As shown in fig. 16, the electronic device includes a processor and a memory connected through a system bus. Wherein the processor is configured to provide computing and control capabilities to support operation of the entire electronic device. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program is executable by a processor for implementing an image generation method provided in the following embodiments. The internal memory provides a cached operating environment for operating system computer programs in the non-volatile storage medium. The electronic device may be a cell phone, tablet computer or personal digital assistant or wearable device, etc.

The implementation of each module in the image generating apparatus provided in the embodiment of the present application may be in the form of a computer program. The computer program may run on a terminal or a server. Program modules of the computer program may be stored in the memory of the terminal or server. Which when executed by a processor, performs the steps of the method described in the embodiments of the application.

The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of an image generation method.

A computer program product comprising instructions which, when run on a computer, cause the computer to perform an image generation method.

Any reference to memory, storage, database, or other medium used by embodiments of the application may include non-volatile and/or volatile memory. Suitable nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The foregoing examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims

1. An image sensor comprising a filter array and a pixel array, wherein the filter array comprises a minimal repeating unit, the minimal repeating unit comprises a plurality of filter sets, each filter set comprises a color filter and a full-color filter, the amount of light entering transmitted by the full-color filter is greater than the amount of light entering transmitted by the color filter, the color filter and the full-color filter each comprise 4 sub-filters, the plurality of filter sets at least comprise a first filter set, and at least one of the color filters in the first filter set comprises a first color sub-filter and a second color sub-filter; the pixel array comprises a plurality of pixels, the pixels of the pixel array are arranged corresponding to the sub-filters of the filter array, and the pixel array is configured to receive light rays passing through the filter array to generate electric signals;

The image sensor is configured to obtain a first combined image according to a first pixel value that is read out by combining a plurality of full-color pixels corresponding to the full-color filter in the filter set, a second pixel value that is read out by combining a plurality of first color photosensitive pixels corresponding to the color filter, and a third pixel value that is read out by combining a plurality of third color photosensitive pixels corresponding to the color filter in a first resolution mode; combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image; the position of the second pixel value in the first combined image is the same as the position of the fourth pixel value in the second combined image; and traversing the pixel values in the first combined image, reading a fourth pixel value which is the same as the second pixel value from the second combined image under the condition that the traversed pixel value is the second pixel value, and adjusting the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain a first target image.

2. The image sensor of claim 1, wherein the image sensor comprises a sensor array, the plurality of filter sets further includes a second filter set, the color filters in the second filter set at least comprise a third color sub-filter.

3. The image sensor of claim 2, wherein the minimal repeating unit comprises 4 filter sets, the 4 filter sets comprising 2 of the first filter sets and 2 of the second filter sets, the 2 of the first filter sets and 2 of the second filter sets being arranged in a matrix.

4. The image sensor of claim 3, wherein the first filter set is disposed in a first diagonal direction and the second filter set is disposed in a second diagonal direction in the minimal repeating unit, the first diagonal direction being different from the second diagonal direction.

5. The image sensor of any one of claims 1 to 4, wherein in each of the filter sets, the color filters and the full-color filters are arranged in a matrix, the full-color filters are arranged in a third diagonal direction, the color filters are arranged in a fourth diagonal direction, and the third diagonal direction is different from the fourth diagonal direction.

6. The image sensor of claim 1, wherein the first color sub-filter is disposed in one of a fifth diagonal direction or a sixth diagonal direction, and the second color sub-filter is disposed in the other of the fifth diagonal direction or the sixth diagonal direction; the fifth diagonal direction is different from the sixth diagonal direction.

7. The image sensor of claim 2, wherein the third color sub-filter is disposed in a seventh diagonal direction and an eighth diagonal direction, the seventh diagonal direction being different from the eighth diagonal direction.

8. The image sensor of claim 2, wherein the filter set comprises 2 full color filters and 2 color filters, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is that:

wherein w represents a full-color sub-filter, and a, b and c each represent a color sub-filter.

9. The image sensor of claim 2, wherein the filter set comprises 2 full color filters and 2 color filters, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is that:

10. The image sensor of claim 2, wherein the color filters in the second filter set further comprise a first color sub-filter or a second color sub-filter.

11. The image sensor of claim 10, wherein the second filter set includes two color filters, the sub-filters of the same color in each color filter are diagonally arranged, and the sub-filters of the same color are oppositely arranged in the two color filters.

12. The image sensor of claim 10, wherein the filter set comprises 2 full color filters and 2 color filters, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is that:

13. The image sensor of claim 10, wherein the filter set comprises 2 full color filters and 2 color filters, and the minimum repeating unit is 8 rows and 8 columns of 64 sub-filters, and the arrangement manner is that:

14. A camera module comprising a lens and the image sensor of any one of claims 1-13; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays.

15. An electronic device, comprising the camera module of claim 14 and a housing, wherein the camera module is disposed on the housing.

16. An image generation method is applied to an image sensor, and is characterized in that the image sensor comprises an optical filter array and a pixel array, the optical filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of optical filter sets, each optical filter set comprises a color filter and a full-color filter, the light incoming quantity of the full-color filter transmitted by the full-color filter is larger than the light incoming quantity of the color filter transmitted by the full-color filter, the color filter and the full-color filter both comprise 4 sub-optical filters, the optical filter sets comprise a first optical filter set and a second optical filter set, and at least one of the color filters in the first optical filter set comprises a first color sub-optical filter and a second color sub-optical filter; the color filters in the second filter set at least comprise a third color sub-filter; the pixel array comprises a plurality of pixels, a first color photosensitive pixel of the pixel array is arranged corresponding to a first color sub-filter of the filter array, a second color photosensitive pixel of the pixel array is arranged corresponding to a second color sub-filter of the filter array, a third color photosensitive pixel of the pixel array is arranged corresponding to a third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electric signal;

The method comprises the following steps:

Combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the color filters to obtain a second combined image; the position of the second pixel value in the first combined image is the same as the position of the fourth pixel value in the second combined image;

And traversing the pixel values in the first combined image, reading a fourth pixel value which is the same as the second pixel value from the second combined image under the condition that the traversed pixel value is the second pixel value, and adjusting the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain a first target image.

17. The method of claim 16, wherein the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; in the first resolution mode, according to the first pixel values read out by merging the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by merging the plurality of first color photosensitive pixels corresponding to the color filter, and the third pixel values read out by merging the plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained, which includes:

In a first resolution mode, according to first pixel values read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter set, according to second pixel values read out by merging first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter set, according to third pixel values read out by merging third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter set containing the first color sub-filters, and according to fourth pixel values read out by merging second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter set, obtaining a first merged image;

The step of obtaining a second combined image according to the fourth pixel values read out by combining the plurality of second color photosensitive pixels corresponding to the color filter includes:

And according to the fourth pixel value read out by merging the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, according to the second pixel value read out by merging the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, and according to the third pixel value read out by merging the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the second color sub-filters, obtaining a second merged image.

18. The method of claim 16, wherein the method further comprises:

in a second resolution mode, according to the first pixel values read out by merging the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by merging the plurality of first color photosensitive pixels corresponding to the color filter, and the third pixel values read out by merging the plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode;

Obtaining a full-color combined image based on the fifth pixel value read out by combining a plurality of full-color pixels in a ninth diagonal direction in the first combined image;

Obtaining a third combined image based on a sixth pixel value read by combining a plurality of first-color photosensitive pixels in a tenth diagonal direction in the first combined image and a seventh pixel value read by combining a plurality of third-color photosensitive pixels in the tenth diagonal direction, wherein the ninth diagonal direction is different from the tenth diagonal direction;

Obtaining a fourth combined image based on the read eighth pixel values combined by the plurality of second color photosensitive pixels in the tenth diagonal direction in the second combined image;

And obtaining a second target image based on the full-color combined image, the third combined image and the fourth combined image.

19. The method of claim 18, wherein the deriving a second target image based on the full-color combined image, the third combined image, and the fourth combined image comprises:

traversing pixels in the third combined image;

Reading a fifth pixel value corresponding to the traversed pixel value from the full-color combined image and adjusting the read fifth pixel value to be adjacent to the traversed pixel value in the case that the traversed pixel value is the sixth pixel value or the seventh pixel value;

And under the condition that the traversed pixel value is a sixth pixel value, reading an eighth pixel value which is the same as the sixth pixel value from the fourth combined image, and adjusting the read eighth pixel value to be adjacent to the sixth pixel value until the pixel values in the full-color combined image, the third combined image and the fourth combined image are all read, so as to obtain a second target image.

20. The method of claim 19, wherein the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; in the second resolution mode, according to the first pixel values read out by merging the plurality of full-color pixels corresponding to the full-color filter in the filter set, the second pixel values read out by merging the plurality of first color photosensitive pixels corresponding to the color filter, and the third pixel values read out by merging the plurality of third color photosensitive pixels corresponding to the color filter, a first merged image is obtained, which includes:

In a second resolution mode, according to first pixel values read out by merging a plurality of full-color pixels corresponding to the full-color filters in the filter set, according to second pixel values read out by merging first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter set, according to third pixel values read out by merging third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter set containing the first color sub-filters, and according to fourth pixel values read out by merging second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter set, obtaining a first merged image;

The step of obtaining a third combined image based on the sixth pixel value read out by combining the plurality of first color photosensitive pixels in the tenth diagonal direction and the seventh pixel value read out by combining the plurality of third color photosensitive pixels in the tenth diagonal direction in the first combined image includes:

A third combined image based on a sixth pixel value of the combined readout of the plurality of first-color photosensitive pixels in the tenth diagonal direction, a seventh pixel value of the combined readout of the plurality of third-color photosensitive pixels in the tenth diagonal direction, and an eighth pixel value of the combined readout of the plurality of second-color photosensitive pixels in the tenth diagonal direction in the first combined image;

the obtaining a fourth combined image based on the read eighth pixel values combined by the plurality of second color photosensitive pixels in the tenth diagonal direction in the second combined image includes:

And obtaining a fourth combined image based on the sixth pixel value read out by combining the plurality of first-color photosensitive pixels in the tenth diagonal direction, the seventh pixel value read out by combining the plurality of third-color photosensitive pixels in the tenth diagonal direction, and the eighth pixel value read out by combining the plurality of second-color photosensitive pixels in the tenth diagonal direction in the second combined image.

21. The method of claim 16, wherein the method further comprises:

in a second resolution mode, merging the read first pixel values according to a plurality of panchromatic pixels corresponding to the optical filter set to obtain a panchromatic image;

Obtaining a fifth combined image according to the second pixel values read out by combining the plurality of first color photosensitive pixels corresponding to the filter set and the third pixel values read out by combining the plurality of third color photosensitive pixels corresponding to the filter set;

combining the read fourth pixel values according to a plurality of second color photosensitive pixels corresponding to the filter set to obtain a sixth combined image;

and obtaining a third target image based on the full-color image, the fifth combined image and the sixth combined image.

22. The method of claim 21, wherein the color filters of the second filter set further comprise a first color sub-filter or a second color sub-filter; the merging the read-out second pixel values according to the plurality of first color photosensitive pixels corresponding to the filter set and the merging the read-out third pixel values according to the plurality of third color photosensitive pixels corresponding to the filter set to obtain a fifth merged image, which includes:

Obtaining a fifth combined image according to second pixel values read out in a combined mode according to first color photosensitive pixels corresponding to a plurality of first color sub-filters in the first filter set, third pixel values read out in a combined mode according to third color photosensitive pixels corresponding to a plurality of third color sub-filters in the second filter set containing the first color sub-filters, and fourth pixel values read out in a combined mode according to second color photosensitive pixels corresponding to a plurality of second color sub-filters in the second filter set;

the step of obtaining a sixth combined image according to the fourth pixel values read out by combining the plurality of second color photosensitive pixels corresponding to the filter set includes:

and according to the fourth pixel value which is read out in a merging way according to the second color photosensitive pixels corresponding to the plurality of second color sub-filters in the first filter set, according to the second pixel value which is read out in a merging way according to the first color photosensitive pixels corresponding to the plurality of first color sub-filters in the second filter set, and according to the third pixel value which is read out in a merging way according to the third color photosensitive pixels corresponding to the plurality of third color sub-filters in the second filter set containing the second color sub-filters, obtaining a sixth merging image.

23. An image generating device is applied to an image sensor, and is characterized in that the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, each filter set comprises a color filter and a full-color filter, the light inlet amount of the full-color filter transmitted by the full-color filter is larger than the light inlet amount of the color filter transmitted by the full-color filter, the color filter and the full-color filter both comprise 4 sub-filters, the filter sets comprise a first filter set and a second filter set, and at least one of the color filters comprises a first color sub-filter and a second color sub-filter; the color filters in the second filter set at least comprise a third color sub-filter; the pixel array comprises a plurality of pixels, a first color photosensitive pixel of the pixel array is arranged corresponding to a first color sub-filter of the filter array, a second color photosensitive pixel of the pixel array is arranged corresponding to a second color sub-filter of the filter array, a third color photosensitive pixel of the pixel array is arranged corresponding to a third color sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electric signal;

The device comprises:

An image generation module, configured to make the position of the second pixel value in the first combined image and the position of the fourth pixel value in the second combined image identical; and traversing the pixel values in the first combined image, reading a fourth pixel value which is the same as the second pixel value from the second combined image under the condition that the traversed pixel value is the second pixel value, and adjusting the read fourth pixel value to be adjacent to the second pixel value until the pixel values in the first combined image are traversed, so as to obtain a first target image.

24. An electronic device comprising a memory, a processor and an image sensor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of the method of any of claims 16 to 22.

25. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any of claims 16 to 22.

26. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 16 to 22.