CN114363486B

CN114363486B - Image sensor, camera module, electronic device, image generation method and device

Info

Publication number: CN114363486B
Application number: CN202111524625.4A
Authority: CN
Inventors: 李小涛
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-12-14
Filing date: 2021-12-14
Publication date: 2024-08-02
Anticipated expiration: 2041-12-14
Also published as: WO2023109264A1; CN114363486A

Abstract

The present application relates to an image sensor, a camera module, an electronic device, an image generation method, an apparatus, a computer-readable storage medium, and a computer program product. The image sensor comprises a filter array and a pixel array, wherein the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, and each filter set comprises a first subunit and a second subunit; each subunit comprises a full-color filter and a color filter, the color filters in the first subunit are arranged on the diagonal line of the first subunit, and the color filters in the second subunit are arranged on the opposite diagonal line of the second subunit; the light inlet amount of the full-color filter is larger than that of the color filter; each filter comprises N lines and N columns of sub-filters with the same color as the filter, wherein N is a positive integer. The image sensor can improve the definition of the image.

Description

Image sensor, camera module, electronic device, image generation method and device

Technical Field

The present application relates to the field of computer technology, and in particular, to an image sensor, a camera module, an electronic device, an image generating method, an image generating apparatus, a computer readable storage medium, and a computer program product.

Background

With the development of computer technology, most of electronic devices such as mobile phones are configured with cameras so as to realize a photographing function through the cameras. An image sensor is arranged in the camera, and color images are acquired through the image sensor. In order to realize color image acquisition, an optical filter array arranged in a Bayer (Bayer) array is generally disposed in an image sensor, so that a plurality of pixels in the image sensor can receive light passing through corresponding optical filters, thereby generating pixel signals with different color channels and further generating an image.

However, the image sharpness generated by the conventional image sensor is low.

Disclosure of Invention

Embodiments of the present application provide an image sensor, an image capturing module, an electronic device, an image generating method, an image generating apparatus, a computer readable storage medium, and a computer program product, which can improve the definition of imaging.

An image sensor comprising a filter array and a pixel array, the filter array comprising a minimal repeating unit comprising a plurality of filter sets, each filter set comprising a first subunit and a second subunit; the first subunit and the second subunit respectively comprise a full-color filter and a color filter, the color filters in the first subunit are arranged on diagonal lines in the first subunit, and the color filters in the second subunit are arranged on opposite diagonal lines in the second subunit; the light inlet amount of the full-color filter is larger than the light inlet amount of the color filter; each full-color filter comprises N lines and N columns of full-color sub-filters, each color filter comprises N lines and N columns of color sub-filters, the colors of the N lines and the N columns of color sub-filters are the same as those of the color filters, and N is a positive integer; each pixel in the pixel array is disposed corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal.

The camera module comprises a lens and the image sensor; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays.

An electronic device, comprising:

the camera module is used for shooting the camera; and

The camera module is arranged on the shell.

The image sensor comprises an optical filter array and a pixel array, wherein the optical filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of optical filter sets, and each optical filter set comprises a first subunit and a second subunit; the first subunit and the second subunit respectively comprise a full-color filter and a color filter, the color filters in the first subunit are arranged on diagonal lines in the first subunit, the color filters in the second subunit are arranged on opposite diagonal lines in the second subunit, the arrangement of the color filters in the direction of the diagonal lines and the direction of the opposite diagonal lines can be more balanced, and the color channel has stronger resolution capability during imaging.

The light quantity transmitted by the full-color filter is larger than the light quantity transmitted by the color filter, and more light quantity can be obtained through the full-color filter during shooting, so that shooting parameters are not required to be regulated, and the definition of imaging under dark light is improved under the condition that the shooting stability is not influenced. When imaging under the dark light, the stability and the definition can be considered, and the stability and the definition of imaging under the dark light are both higher.

An image generation method is applied to an image sensor, the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, and each filter set comprises a first subunit and a second subunit; the first subunit and the second subunit respectively comprise a full-color filter and a color filter, the color filters in the first subunit are arranged on diagonal lines in the first subunit, and the color filters in the second subunit are arranged on opposite diagonal lines in the second subunit; the light inlet amount of the full-color filter is larger than the light inlet amount of the color filter; each full-color filter comprises N lines and N columns of full-color sub-filters, each color filter comprises N lines and N columns of color sub-filters, the colors of the N lines and the N columns of color sub-filters are the same as those of the color filters, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

The method comprises the following steps:

In a full resolution mode, reading out full resolution panchromatic pixel values from panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filters, and reading out full resolution color pixel values from color pixels corresponding to each color sub-filter in the color filters;

a full resolution target image is generated based on each of the full resolution panchromatic pixel values and each of the full resolution color pixel values.

An image generation device is applied to an image sensor, the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, and each filter set comprises a first subunit and a second subunit; the first subunit and the second subunit respectively comprise a full-color filter and a color filter, the color filters in the first subunit are arranged on diagonal lines in the first subunit, and the color filters in the second subunit are arranged on opposite diagonal lines in the second subunit; the light inlet amount of the full-color filter is larger than the light inlet amount of the color filter; each full-color filter comprises N lines and N columns of full-color sub-filters, each color filter comprises N lines and N columns of color sub-filters, the colors of the N lines and the N columns of color sub-filters are the same as those of the color filters, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal

The device comprises:

The reading module is used for reading out full-resolution panchromatic pixel values from panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filters and reading out full-resolution color pixel values from color pixels corresponding to each color sub-filter in the color filters in a full-resolution mode;

An image generation module for generating a full resolution target image based on each of the full resolution panchromatic pixel values and each of the full resolution color pixel values.

An electronic device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of the image generation method as described above.

A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of a method as described above.

A computer program product comprising a computer program which, when executed by a processor, implements the steps of a method as described above.

The above-described image generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, in a full resolution mode, reading out full resolution panchromatic pixel values for panchromatic pixels corresponding to each of the panchromatic sub-filters in the panchromatic filter, and reading out full resolution color pixel values for color pixels corresponding to each of the color sub-filters in the color filter; the light quantity of the full-color filter is larger than that of the color filter, so that full-color channel information can be fused into an image, the overall light quantity is improved, and a full-resolution target image with more information and clearer detail analysis can be generated based on all full-resolution full-color pixel values and all full-resolution color pixel values.

In the filter array, the minimal repeating unit includes a plurality of filter sets, each filter set including a first subunit and a second subunit; the first subunit and the second subunit respectively comprise a full-color filter and a color filter, the color filters in the first subunit are arranged on diagonal lines in the first subunit, and the color filters in the second subunit are arranged on opposite diagonal lines in the second subunit, so that the arrangement of the color filters in the direction of the diagonal lines and the direction of the opposite diagonal lines is more balanced, and the color channel has stronger resolution capability when the full-resolution target image is generated.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an electronic device in one embodiment;

FIG. 2 is an exploded view of an image sensor in one embodiment;

FIG. 3 is a schematic diagram of the connection of a pixel array and readout circuitry in one embodiment;

FIG. 4 is a schematic diagram of an arrangement of minimum repeating units in an embodiment of a filter array with N being 1;

FIG. 5 is a schematic diagram of an arrangement of minimum repeating units in a filter array with N being 1 according to another embodiment;

FIG. 6 is a schematic diagram of an arrangement of minimum repeating units in a filter array with N being 1 according to another embodiment;

FIG. 7 is a schematic diagram of an arrangement of minimum repeating units in a filter array with N being 1 according to another embodiment;

FIG. 8 is a schematic diagram of an arrangement of minimum repeating units in an embodiment of a filter array with N being 2;

FIG. 9 is a schematic diagram of an arrangement of minimum repeating units in a filter array with N being 2 according to another embodiment;

FIG. 10 is a schematic diagram of an arrangement of minimum repeating units in a filter array having N of 2 according to another embodiment;

FIG. 11 is a schematic diagram of an arrangement of minimum repeating units in a filter array with N being 2 according to another embodiment;

FIG. 12 is a schematic diagram of an arrangement of minimum repeating units in a filter array having N of 3 according to another embodiment;

FIG. 13 is a flow diagram of an image generation method in one embodiment;

FIG. 14 is a flow diagram of an image generation method in one embodiment;

FIG. 15 is a schematic view of a first target image in one embodiment;

FIG. 16 is a flow diagram of generating a second target image in one embodiment;

FIG. 17 is a schematic illustration of a first color image and a first full color image in one embodiment;

FIG. 18 is a schematic diagram of a second target image in one embodiment;

FIG. 19 is a schematic view of a second target image in another embodiment;

FIG. 20 is a schematic view of a second target image in another embodiment;

FIG. 21 is a schematic diagram of a second target image in another embodiment;

FIG. 22 is a flow diagram of generating a third target image in one embodiment;

FIG. 23 is a schematic illustration of a second full color image, a dual color second color image, and a single color second color image in one embodiment;

FIG. 24 is a schematic view of a third target image in one embodiment;

FIG. 25 is a schematic view of a third target image in another embodiment;

FIG. 26 is a flow diagram of generating a fourth target image in one embodiment;

FIG. 27 is a schematic illustration of a fourth target image in an embodiment;

FIG. 28 is a block diagram showing the structure of an image generating apparatus in one embodiment;

Fig. 29 is a schematic diagram showing an internal structure of the electronic device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

It will be understood that the terms first, second, etc. as used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another element. For example, a first target image may be referred to as a second target image, and similarly, a second target image may be referred to as a first target image, without departing from the scope of the application. The first target image and the second target image are both target images, but they are not the same target image.

In one embodiment, the electronic device 100 includes a mobile phone, a tablet computer, a notebook computer, an atm, a gate, a smart watch, a head display device, etc., and it is understood that the electronic device 100 may be any other device with an image processing function. The electronic device 100 includes a camera module 20, a processor 30, and a housing 40. The camera module 20 and the processor 30 are both disposed in the housing 40, and the housing 40 can be further used for mounting functional modules such as a power supply device and a communication device of the electronic device 100, so that the housing 40 provides protection such as dust protection, drop protection, water protection, etc. for the functional modules.

The camera module 20 may be a front camera module, a rear camera module, a side camera module, an under-screen camera module, etc., which is not limited herein. The image capturing module 20 includes a lens and an image sensor 21, and when the image capturing module 20 captures an image, light passes through the lens and reaches the image sensor 21, and the image sensor 21 is configured to convert an optical signal applied to the image sensor 21 into an electrical signal.

In one embodiment, as shown in FIG. 2, the image sensor 21 includes a microlens array 22, a filter array 23, and a pixel array 24.

The microlens array 22 includes a plurality of microlenses 221, the sub-filters in the filter array 23, and the pixels in the pixel array 24 are arranged in a one-to-one correspondence, the microlenses 221 are configured to collect incident light, and the collected light passes through the corresponding sub-filters and then is projected onto the pixels, received by the corresponding pixels, and the pixels convert the received light into electrical signals.

In one embodiment, the filter array 23 includes a plurality of minimal repeating units 230. The minimal repeating unit 230 includes a plurality of filter sets, each including a first subunit 233 and a second subunit 234; the first subunit 233 and the second subunit 234 each include a full-color filter 235 and a color filter 236, the color filter 236 in the first subunit 233 is arranged on a diagonal line in the first subunit 233, and the color filter 236 in the second subunit 234 is arranged on an opposite diagonal line in the second subunit 234; the amount of light entering through the full color filter 235 is larger than the amount of light entering through the color filter 236; each of the full-color filters 235 includes N rows and N columns of full-color sub-filters, each of the color filters 236 includes N rows and N columns of color sub-filters, the N rows and N columns of color sub-filters are the same color as the color filters, and N is a positive integer. In fig. 2, N is 2.

The diagonal line may be a line connecting the upper left corner and the lower right corner, or may be a line connecting the upper right corner and the lower left corner. The anti-diagonal line may be a line connecting the upper left corner and the lower right corner, or may be a line connecting the upper right corner and the lower left corner. The diagonal and the anti-diagonal are perpendicular to each other. That is, if the diagonal is the line connecting the upper left corner and the lower right corner, the anti-diagonal is the line connecting the upper right corner and the lower left corner; if the diagonal is the line connecting the upper right corner and the lower left corner, then the anti-diagonal is the line connecting the upper left corner and the lower right corner.

The image sensor includes a filter array 23 and a pixel array 24, the filter array 23 includes a minimal repeating unit 230, the minimal repeating unit 230 includes a plurality of filter sets, and each filter set includes a first subunit 233 and a second subunit 234; the first subunit 233 and the second subunit 234 each include a full-color filter 235 and a color filter 236, the color filters 236 in the first subunit 233 are arranged on a diagonal line in the first subunit 233, and the color filters in the second subunit 234 are arranged on an opposite diagonal line in the second subunit 234, so that the arrangement of the color filters 236 in the direction of the diagonal line and in the direction of the opposite diagonal line is more balanced, and the color channel has stronger resolution capability during imaging.

The light quantity transmitted by the full-color filter 235 is larger than the light quantity transmitted by the color filter 236, so that more light quantity can be obtained through the full-color filter 236 during shooting, thus the shooting parameters are not required to be regulated, and the definition of imaging under dark light is improved under the condition that the shooting stability is not influenced. When imaging under the dark light, the stability and the definition can be considered, and the stability and the definition of imaging under the dark light are both higher.

In one embodiment, each row and each column of the minimal repeating unit 230 includes color filters 236 of each color, i.e., the color filters 236 of each color are arranged in a dispersed manner, which improves color resolution and brightness variation resolution and reduces the risk of false colors. It will be appreciated that alternating the full color filter 235 with the color filter 236 in each row and each column, i.e., 50% of the full color filter 235 in the minimal repeating unit, the first filter set, or the second filter set, may increase the amount of light entering each localized area in the image.

In one embodiment, the panchromatic filter 235 and the color filter 236 are alternately arranged on each row or each column, which can improve the color resolution of each row or each column of the image, resulting in a richer color of the image.

In one embodiment, the minimal repeating unit 230 includes 2 first filter sets 231 and 2 second filter sets 232,2 first filter sets 231 arranged on a diagonal of the minimal repeating unit 230, and 2 second filter sets 232 arranged on an opposite diagonal of the minimal repeating unit 230. The 2 first filter sets 231 and the 2 second filter sets 232 are arranged in a matrix.

Wherein, 2 first filter sets are identical, and 2 second filter sets are identical.

In one embodiment, each filter set includes only a full color filter 235 and 2 color filters 236. The color filters 236 include a first color filter, a second color filter, and a third color filter. The first color filter, the second color filter and the third color filter are three different color filters. The colors of the first color filter, the second color filter and the third color filter can be set according to the needs. For example, the first color filter may be a red filter, the second color filter may be a green filter, and the third color filter may be a blue filter.

The width of the band of light transmitted by the color filter 236 is smaller than the width of the band of light transmitted by the full color filter 235, for example, the band of light transmitted by the color filter 236 may correspond to the band of red light, the band of green light, or the band of blue light, and the band of light transmitted by the full color filter 235 is all the bands of visible light, that is, the color filter 236 allows only light of a specific color to pass, while the full color filter 235 may pass light of all colors. Of course, the wavelength band of the transmitted light of the color filter 236 may also correspond to the wavelength band of other color light, such as magenta, violet, cyan, yellow, etc., which is not limited herein.

Further, the color filters 236 in the first filter set 231 include a first color filter and a second color filter, and the color filters 236 in the second filter set 232 include a second color filter and a third color filter.

In one embodiment, each filter set includes 2 first subunits 233 and 2 second subunits 234; the 2 first sub-units 233 are arranged in a first row direction of the filter set, and the 2 second sub-units 234 are arranged in a second row direction of the filter set, the first row direction and the second row direction being adjacently arranged; or 2 first sub-units 233 are arranged in a first column direction of the filter set, and 2 second sub-units 234 are arranged in a second column direction of the filter set, the first column direction and the second column direction being adjacently arranged.

As shown in fig. 2, the first filter set 231 includes 2 first sub-units 233 and 2 second sub-units 234,2 first sub-units 233 arranged in a first row direction of the first filter set 231, and 2 second sub-units 234 arranged in a second row direction of the filter set. In other embodiments, 2 first subunits 233 are arranged in a first column direction of the first filter set 231 and 2 second subunits 234 are arranged in a second column direction of the filter set.

Further, the first subunit 233 and the second subunit 234 in the same filter set, which include the same color filters, are arranged on a diagonal line of the same filter set.

The first sub-unit 233 and the second sub-unit 234, which include the same color filters, in the same filter set are arranged on the diagonal of the same filter set, and the color filters 236 in the first sub-unit are arranged on the diagonal of the first sub-unit, and the color filters 236 in the second sub-unit are arranged on the opposite diagonal of the second sub-unit, so that the color filters 236 in the first sub-unit 233 arranged on the diagonal of the same filter set are arranged on the diagonal of the first sub-unit, and the color filters 236 in the second sub-unit 234 arranged on the diagonal of the same filter set are arranged on the opposite diagonal of the second sub-unit, thereby improving the distribution balance of the color filters 236 on the diagonal and the opposite diagonal.

It will be appreciated that the first sub-unit 233 and the second sub-unit 234 each include a full color filter 235 and a color filter 236, and then the first sub-unit 233 and the second sub-unit 234 including the same color filter, that is, the first sub-unit 233 and the second sub-unit 234 including the same color filter 236. As shown in fig. 2, the first filter set 231 includes 2 first sub-units 233 and 2 second sub-units 234,2 of the first sub-units 233 arranged in the row direction of the first filter set 231, and 2 second sub-units 234 arranged in the row direction of the first filter set 231, and the first sub-units 233 and the second sub-units 234 including the same color filters are arranged on the diagonal of the first filter set 231.

Further, the color filter 236 includes a first color filter, a second color filter, and a third color filter; one of the 2 first sub-units 233 in the first filter set 231 includes a first color filter, and the other first sub-unit 233 includes a second color filter; one of the 2 second sub-units 234 in the first filter set 231 includes a first color filter, and the other second sub-unit 234 includes a second color filter.

Likewise, the color filters 236 include a first color filter, a second color filter, and a third color filter; one of the 2 first sub-units 233 in the second filter set 232 includes a second color filter, and the other first sub-unit 233 includes a third color filter; one of the 2 second sub-units 234 in the second filter set 232 includes a second color filter, and the other second sub-unit 234 includes a third color filter.

The pixel array 24 includes a plurality of pixels, and the pixels of the pixel array 24 are disposed corresponding to the sub-filters of the filter array 23. The pixel array 24 is configured to receive light passing through the filter array 23 to generate an electrical signal.

Wherein pixel array 24 is configured to receive light rays passing through filter array 23 to generate electrical signals, refers to pixel array 24 being configured to photoelectrically convert light rays of a scene of a given set of subjects passing through filter array 23 to generate electrical signals. Rays of a scene of a given set of objects are used to generate image data. For example, a subject is a building, and a scene of a given set of subjects refers to a scene in which the building is located, and other subjects may be included in the scene.

In one embodiment, pixel array 24 includes a plurality of minimal repeating units 240, minimal repeating units 240 further including a plurality of panchromatic pixels 241 and a plurality of different color pixels 242, each row and each column including a color pixel of each color; each panchromatic pixel 242 corresponds to one of the panchromatic filters 235, and the panchromatic pixel 242 receives light passing through the corresponding sub-filter to generate an electrical signal. Each color pixel 242 corresponds to one of the sub-filters of the color filter 236, and the color pixels 242 receive light passing through the corresponding sub-filter to generate an electrical signal.

As shown in fig. 3, a readout circuit 25 is electrically connected to the pixel array 24 for controlling the exposure of the pixel array 24 and the reading and outputting of the pixel values of the pixels. The readout circuit 25 includes a vertical driving unit 251, a control unit 252, a column processing unit 253, and a horizontal driving unit 254. The vertical driving unit 251 includes a shift register and an address decoder. The vertical driving unit 251 includes a readout scan and a reset scan function. The control unit 252 configures timing signals according to an operation mode, and controls the vertical driving unit 251, the column processing unit 253, and the horizontal driving unit 254 to operate cooperatively using various timing signals. The column processing unit 253 may have an analog-to-digital (a/D) conversion function for converting analog pixel signals into a digital format. The horizontal driving unit 254 includes a shift register and an address decoder. The horizontal driving unit 254 sequentially scans the pixel array 24 column by column.

When N is 1, each of the full-color filters 235 includes 1 row and 1 column of full-color sub-filters, and each of the color filters 236 includes 1 row and 1 column of color sub-filters, that is, each of the full-color sub-filters is the full-color filter 235 and each of the color sub-filters is the color filter 236.

In one embodiment, as shown in fig. 4, N is 1, and the minimum repeating unit includes 8 rows and 8 columns of 64 filters, which are arranged in the following manner:

where w denotes a full color filter 235, and a, b, and c each denote a color filter 236.

In one embodiment, as shown in fig. 5, N is 1, and the minimum repeating unit includes 8 rows and 8 columns of 64 filters, which are arranged in the following manner:

In one embodiment, as shown in fig. 6, N is 1, and the minimum repeating unit includes 8 rows and 8 columns of 64 filters, which are arranged in the following manner:

In one embodiment, as shown in fig. 7, N is 1, and the minimum repeating unit includes 8 rows and 8 columns of 64 filters, which are arranged in the following manner:

Where w may be a white filter, a red filter, b green filter, c blue filter, or, for example, a magenta filter, b cyan filter, c yellow filter, etc., without limitation.

It will be appreciated that the electronic device adjusts the minimal repeating unit to obtain a new minimal repeating unit. For example, the electronic device rotates the minimal repeating unit of fig. 4 by 90 degrees in a counterclockwise direction, resulting in the minimal repeating unit of fig. 7; exchanging positions of the a color filter and the c color filter in the minimum repeating unit of fig. 4, and obtaining the minimum repeating unit of fig. 5; after the positions of the a color filter and the c color filter in the minimum repeating unit of fig. 4 are exchanged, the minimum repeating unit of fig. 6 can be obtained by rotating the positions of the a color filter and the c color filter in the counterclockwise direction by 90 degrees.

In the minimum repeating units of fig. 4 to 7 described above, the b color filters have a minimum area of 4 by 4 as an arrangement period, and the sampling rates of the b color filters in the diagonal and anti-diagonal directions are uniform, so that the arrangement is more uniform, and thus the b channel has a stronger resolving power in the horizontal direction, the vertical direction, and the diagonal direction. Likewise, the sampling rates of the a color filter and the c color filter in the diagonal and opposite diagonal directions of the local area are consistent, and the arrangement is more balanced, so that the a channel and the c channel have stronger resolution capability. The arrangement of the a color filter, the b color filter, the c color filter and the w color filter is more dispersed, so that the filter array has the property of full arrangement in the horizontal and vertical directions, and the resolution of diagonal and diagonal directions is considered.

In one embodiment, as shown in fig. 8, N is 2, and the minimum repeating unit includes 16 rows and 16 columns of 256 sub-filters, which are arranged in the following manner:

wherein w represents a full-color sub-filter, and a, b and c each represent a color sub-filter.

In one embodiment, as shown in fig. 9, N is 2, and the minimum repeating unit includes 16 rows and 16 columns of 256 sub-filters, which are arranged in the following manner:

In one embodiment, as shown in fig. 10, N is 2, and the minimum repeating unit includes 16 rows and 16 columns of 256 sub-filters, which are arranged in the following manner:

In one embodiment, as shown in fig. 11, N is 2, and the minimum repeating unit includes 16 rows and 16 columns of 256 sub-filters, which are arranged in the following manner:

Where w may be a white sub-filter, a is a red sub-filter, b is a green sub-filter, c is a blue sub-filter, or, for example, a is a magenta sub-filter, b is a cyan sub-filter, c is a yellow sub-filter, etc., without limitation.

It will be appreciated that the electronic device adjusts the minimal repeating unit to obtain a new minimal repeating unit. For example, the electronic device rotates the minimal repeating unit of fig. 8 by 90 degrees in a counter-clockwise direction, resulting in the minimal repeating unit of fig. 11; the minimum repeating unit of fig. 9 can be obtained by exchanging positions of the a color filter and the c color filter in the minimum repeating unit of fig. 8; after the positions of the a color filter and the c color filter in the minimum repeating unit of fig. 8 are exchanged, the minimum repeating unit of fig. 10 can be obtained by rotating the positions of the a color filter and the c color filter in the counterclockwise direction by 90 degrees.

In the minimum repeating units of fig. 8 to 11 described above, the b color sub-filters have a minimum area of 8 by 8 as an arrangement period, and the sampling rates of the b color sub-filters in diagonal and anti-diagonal directions are uniform, so that the arrangement is more uniform, and thus the b channel has a stronger resolving power in horizontal, vertical and diagonal directions. Likewise, the sampling rates of the a-color sub-filter and the c-color sub-filter in the diagonal and opposite diagonal directions of the local area are consistent, and the arrangement is more balanced, so that the a-channel and the c-channel have stronger resolution capability. The arrangement of the color sub-filter a, the color sub-filter b, the color sub-filter c and the full-color sub-filter w is more dispersed, so that the filter array has the property of full arrangement in the horizontal and vertical directions, and the resolution of diagonal and diagonal directions is considered.

It should be noted that N may also be other positive integers such as 3, 4 or 5, and the arrangement manner is similar to that of N being 1 or 2, and will not be described herein. Fig. 12 is a diagram of a minimal repeating unit with N being 3 in one embodiment.

In one embodiment, there is also provided an image capturing module, including a lens and the image sensor described above; the image sensor is used for receiving light rays passing through the lens, and the pixels generate electric signals according to the light rays.

In one embodiment, an electronic device is further provided, including the above-mentioned camera module; and the shell is provided with the camera shooting module.

In one implementation, an image generation method is provided and applied to an image sensor, wherein the image sensor comprises an optical filter array and a pixel array, the optical filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of optical filter sets, and each optical filter set comprises a first subunit and a second subunit; the first subunit and the second subunit respectively comprise a full-color filter and a color filter, the color filters in the first subunit are arranged on diagonal lines in the first subunit, and the color filters in the second subunit are arranged on opposite diagonal lines in the second subunit; the light inlet amount of the full-color filter is larger than that of the color filter; each full-color filter comprises N lines and N columns of full-color sub-filters, each color filter comprises N lines and N columns of color sub-filters, the N lines and the N columns of color sub-filters are the same as the color of the color filter, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is disposed corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal.

As shown in fig. 13, the image generation method includes:

In step 1302, in full resolution mode, full resolution panchromatic pixel values are read from panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filter, and full resolution color pixel values are read from color pixels corresponding to each color sub-filter in the color filter.

The full resolution mode is a mode in which each sub-filter is read out as one pixel.

The color filter has narrower spectral response than the full-color filter, so that the light quantity of the full-color filter transmitted is larger than that of the full-color filter, namely the wave band width of the light transmitted by the color filter is smaller than that of the light transmitted by the full-color filter, the full-color filter transmits more light, the corresponding full-color pixel obtained through the full-color filter has higher signal to noise ratio, the full-color pixel contains more information, and more texture details can be analyzed. Where the signal-to-noise ratio refers to the ratio between the normal signal and the noise signal. The higher the signal-to-noise ratio of a pixel, the higher the proportion of normal signals contained in that pixel, and the more information is parsed from that pixel.

The color pixel 242 may be a G (Green) pixel, an R (Red) pixel, a B (Blue) pixel, or the like, but is not limited thereto.

In the case of receiving a photographing instruction, whether the user selects a resolution mode to be used is detected, and when it is detected that the user selects a full resolution mode to be used, or when the user does not select a resolution mode to be used, preview photographing is not used, and the current environment is not a night scene mode, the full resolution mode is used in response to the photographing instruction.

In the full resolution mode, light transmitted through the full color sub-filters in the full color filter is projected onto the corresponding full color pixels 241, and the full color pixels 241 receive the light passing through the full color sub-filters to generate an electrical signal. The light transmitted through the color sub-filters in the color filters is projected onto the corresponding color pixels 242, and the color pixels 242 pass through the light of the corresponding color sub-filters to generate an electrical signal.

Each of the full-color filters includes N rows and N columns of full-color sub-filters, and then each of the full-color filters corresponds to N rows and N columns of full-color pixels 241. Each color filter includes N rows and N columns of color sub-filters of the same color, and each color filter corresponds to N rows and N columns of color pixels 242.N is a positive integer greater than or equal to 2.

In other embodiments, N may be 1, i.e., 1 full color pixel 241 for each full color filter and 1 color pixel 242 for each color filter.

In step 1304, a full resolution target image is generated based on each full resolution panchromatic pixel value and each full resolution color pixel value.

The electronic device may read pixel values from each full-resolution panchromatic pixel value and each full-resolution color pixel value according to a preset pixel reading scheme to generate a full-resolution target image. The preset pixel reading mode is a preset pixel reading mode.

In the image generation method, under the full resolution mode, full resolution panchromatic pixel values are read out from panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filter, and full resolution color pixel values are read out from color pixels corresponding to each color sub-filter in the color filter; the light quantity of the full-color filter is larger than that of the color filter, so that full-color channel information can be fused into an image, the overall light quantity is improved, and a full-resolution target image with more information and clearer detail analysis can be generated based on all full-resolution full-color pixel values and all full-resolution color pixel values.

In one embodiment, as shown in fig. 14, the method further includes:

Step 1402, in a first resolution mode, merging the panchromatic pixels corresponding to the panchromatic sub-filters in each panchromatic filter to read out a first panchromatic pixel value, and merging the color pixels corresponding to the color sub-filters in each color filter to read out a first color pixel value; the resolution corresponding to the first resolution mode is smaller than the resolution corresponding to the full resolution mode.

The first resolution mode refers to a primary pixel merging read-out mode in which resolution, power consumption, signal-to-noise ratio and frame rate are relatively balanced. The first resolution mode may be a default mode for image or video capturing.

In the case of receiving a photographing instruction, whether a user selects a resolution mode to be used is detected, and when it is detected that the user selects the first resolution mode to be used, or when the user does not select the resolution mode to be used, preview photographing is not used, and the current environment is not night scene mode, the first resolution mode is used in response to the photographing instruction.

In the first resolution mode, light transmitted through the full color sub-filters in the full color filter is projected onto the corresponding full color pixels 241, and the full color pixels 241 receive the light passing through the full color sub-filters to generate an electrical signal. The light transmitted through the color sub-filters in the color filters is projected onto the corresponding color pixels 242, and the color pixels 242 pass through the light of the corresponding color sub-filters to generate an electrical signal.

The merged readout refers to summing pixel values of a plurality of pixels or calculating a mean value of pixel values of a plurality of pixels.

In one embodiment, for each panchromatic filter, the panchromatic pixels 241 corresponding to each panchromatic sub-filter are averaged, and the average is read out as the first panchromatic pixel value. In another embodiment, for each panchromatic filter, the corresponding panchromatic pixels 241 for each panchromatic sub-filter are summed and the summed sum is read out as the first panchromatic pixel value. In other embodiments, the electronic device may also combine the full-color pixels 241 corresponding to the full-color sub-filters to read out the first full-color pixel value in other manners, which is not limited herein.

In one embodiment, for each color filter, the color pixels 242 corresponding to each color sub-filter are averaged, and the average is read as the first color pixel value. In another embodiment, for each color filter, the color pixels corresponding to the respective color sub-filters are added, and the resulting sum is read out as the first color pixel value. In other embodiments, the electronic device may combine the color pixels 242 corresponding to the color sub-filters to read out the first color pixel value in other manners, which is not limited herein.

The manner of combining and reading out the first full-color pixel values may be the same or different for each full-color filter. The manner of combining and reading out the first color pixel values may be the same or different for each color filter. The first full-color pixel value and the first color pixel value may be read out in a combined manner for the full-color filter and the color filter, or may be read out in a combined manner.

A first target image is generated based on each of the first panchromatic pixel values and each of the first color pixel values, step 1404.

The electronic device may read pixel values from each of the first panchromatic pixel values and each of the first color pixel values according to a preset pixel reading pattern to generate the first target image. The preset pixel reading mode is a preset pixel reading mode. Taking the arrangement of fig. 8 as an example, the minimum repeating unit of the filter array 23 is taken as a minimum repeating unit, the first target image is generated as shown in fig. 15. In the first target image of fig. 15, w represents a first full-color pixel value, and a, b, and c represent first color pixel values of three different colors.

In this embodiment, in the first resolution mode, the panchromatic pixels 241 corresponding to the panchromatic sub-filters in each panchromatic filter are combined and read out to obtain the first panchromatic pixel value, the color pixels 242 corresponding to the color sub-filters in each color filter are combined and read out to obtain the first color pixel value, and the light incoming quantity transmitted by the panchromatic filter is larger than the light incoming quantity transmitted by the color filter, so that the panchromatic channel information can be fused into the image, the overall light incoming quantity is improved, and a first target image with more information and clearer detail analysis can be generated based on the first panchromatic pixel values and the first color pixel values.

In one embodiment, as shown in fig. 16, after generating the first target image, the method further includes:

A step 1602 of reading out second full-color pixel values by combining the plurality of first full-color pixel values corresponding to each sub-unit in the first target image in the second resolution mode, and generating a first full-color image based on the respective second full-color pixel values; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode, and the subunit comprises a first subunit and a second subunit.

The second resolution mode is a mode used in a scene with lower resolution requirements than the first resolution mode, and is a low resolution, low power consumption, high signal to noise ratio and high frame rate two-level pixel merging read-out mode. The resolution and power consumption corresponding to the second resolution mode are smaller than those corresponding to the first resolution mode. The signal-to-noise ratio and the frame rate corresponding to the second resolution mode are larger than those corresponding to the first resolution mode.

The second resolution mode may be, but not limited to, a preview mode at the time of image capturing, a preview mode at the time of video capturing, or a scene with low resolution requirements such as a night scene mode for image capturing and video capturing under night scenes. The preview mode of video capturing is, for example, 1080p video preview, application video preview, or the like.

In the case where a photographing instruction is received, it is determined whether the photographing instruction is a preview photographing. When the photographing instruction is a preview photographing, the second resolution mode is triggered. Or the electronic equipment detects whether the current environment is a night scene or not, and triggers the second resolution mode under the condition that the current environment is the night scene. Or when the user selects the second resolution mode, triggering a read-out mode corresponding to the second resolution mode.

Specifically, the electronic device combines and reads out the second full-color pixel values corresponding to the plurality of first full-color pixel values in each sub-unit in the first target image, and reads out the pixel values from the respective second full-color pixel values according to a preset pixel reading manner to generate the first full-color image.

It will be appreciated that, in the first resolution mode, each pixel value in the first target image is obtained by combining pixels corresponding to sub-filters in each filter in the filter array, and each pixel value in the first target image corresponds to each filter in the filter array and also corresponds to a plurality of sub-filters in each filter.

In the second resolution mode, the electronics determine a plurality of pixel values for each sub-unit in the first target image, obtain a plurality of first panchromatic pixel values from the plurality of pixel values, read out a second panchromatic pixel value, and obtain a plurality of first color pixel values of the same color from the plurality of pixel values, read out a second color pixel value.

It will be appreciated that combining the readouts may include one of averaging, summing, or weighted averaging, without limitation.

Step 1604, the second color pixel values are read out in combination of the first color pixel values corresponding to the plurality of same colors in each sub-unit in the first target image, and the first color image is generated based on the respective second color pixel values.

Specifically, in the second resolution mode, the electronic device combines and reads out the second color pixel values from the first color pixel values corresponding to the plurality of same colors in each sub-unit in the first target image, and reads out the pixel values from the respective second color pixel values according to a preset pixel reading mode to generate the first color image.

Taking fig. 15 as an example of the first target image, a first color image is generated as indicated by 1702 in fig. 17, and a first full-color image is generated as indicated by 1704. In fig. 17, w denotes a second full-color pixel value, and a, b, and c denote second color pixel values of three different colors.

In step 1606, a second target image is generated based on the first panchromatic image and the first color image.

When the first full-color image and the first color image need to be transmitted in a packaged manner, the electronic device can generate a second target image based on the first full-color image and the first color image, and transmit the second target image.

Specifically, the electronic device alternately arranges the second panchromatic pixel values of each row in the first panchromatic image and the second color pixel values of each row in the first color image to generate a second target image; or arranging the second panchromatic pixel values of each column in the first panchromatic image and the second color pixel values of each column in the first color image alternately to generate a second target image.

Fig. 18 and 19 are schematic diagrams of a second target image obtained by alternately arranging the second full-color pixel values of each line in the first full-color image and the second color pixel values of each line in the first color image. Fig. 20 and 21 are second target images obtained by alternately arranging the second full-color pixel values of each column in the first full-color image and the second color pixel values of each column in the first color image.

In another embodiment, the electronic device may further combine pixel values at the same position in the first full-color image and the first color image to obtain combined pixel values at the corresponding positions, and form the second target image based on each combined pixel value. The combination can adopt one of mean value calculation, weighted average calculation or addition summation.

In other embodiments, the electronic device may also generate the second target image in other ways, which are not limited herein.

In the present embodiment, in the second resolution mode, the second panchromatic pixel values are read out by combining the plurality of first panchromatic pixel values corresponding to each sub-unit in the first target image, and the second color pixel values are read out by combining the plurality of first color pixel values corresponding to the same color in each sub-unit in the first target image, so that the respective different color pixels 242 can be mixed and arranged, so that the respective second color pixels, such as RGB pixels, in the generated second target image are distributed more uniformly, and the image quality is higher. In addition, the resolution and the image size of the obtained second target image are further reduced, the panchromatic pixels 241 have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of the combined output of the secondary pixels is achieved. In addition, in the second resolution mode, full-color pixels with full arrangement are arranged, interpolation is not needed, and the overall resolution is improved. Meanwhile, in the case of full size, color pixels of respective colors such as a pixel of a first color and a pixel of a third color are more dispersed and balanced in a diagonal direction or an opposite direction.

In another embodiment, the method further comprises: in the second resolution mode, merging and reading out fifth panchromatic pixel values from panchromatic pixels corresponding to all panchromatic sub-filters of a plurality of panchromatic filters in each first sub-unit or each second sub-unit, and generating a third panchromatic image based on the fifth panchromatic pixel values; combining and reading out fifth color pixel values of color pixels corresponding to all color sub-filters of a plurality of color filters with the same color in each first sub-unit or each second sub-unit, and generating a third color image based on all the fifth color pixel values; a fifth target image is generated based on the third full-color image and the third color image.

The mode of combining and reading can be one of averaging, weighted averaging, adding and the like.

When the third full-color image and the third color image need to be transmitted in a packaging mode, the electronic device can generate a fifth target image based on the third full-color image and the third color image and then transmit the fifth target image.

Wherein generating a fifth target image based on the third full-color image and the third color image, comprises: arranging fifth panchromatic pixel values of each row in the third panchromatic image and fifth color pixel values of each row in the third color image alternately to generate a fifth target image; or arranging the fifth panchromatic pixel value of each column in the third panchromatic image and the fifth color pixel value of each column in the third color image alternately to generate a fifth target image.

In another embodiment, the electronic device may further combine pixel values at the same position in the third full-color image and the third color image to obtain combined pixel values at the corresponding positions, and form the fifth target image based on each combined pixel value. The combined reading can adopt one of mean value calculation, weighted average calculation or addition summation.

In other embodiments, the electronic device may also generate the fifth target image in other ways, not limited herein.

In the present embodiment, in the second resolution mode, the third full-color image and the third color image can be generated more quickly, and the fifth target image can be generated more quickly by combining and reading out the fifth full-color pixel value for the full-color pixel corresponding to each of the plurality of full-color sub-filters of the plurality of full-color filters in each of the first sub-unit or the second sub-unit, and combining and reading out the fifth color pixel value for the color pixel corresponding to each of the plurality of color sub-filters of the same color in each of the first sub-unit or the second sub-unit.

Moreover, the above embodiment can mix and arrange the different color pixels, so that the fourth color pixel values, such as RGB pixels, in the generated fifth target image are distributed more uniformly and the image quality is higher. In addition, the resolution and the image size of the obtained fifth target image are further reduced, the panchromatic pixels 241 have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of the combined output of the secondary pixels is achieved.

In one embodiment, as shown in fig. 22, the method further includes:

Step 2202, in the third resolution mode, combining and reading out third panchromatic pixel values corresponding to the plurality of second panchromatic pixel values in the same filter set in the first panchromatic image, and generating a second panchromatic image based on each third panchromatic pixel value; the resolution corresponding to the third resolution mode is smaller than the resolution corresponding to the second resolution mode.

The third resolution mode is a mode used in a scene with lower resolution requirements than the second resolution mode, and is a three-level pixel merging read-out mode with low resolution, low power consumption, high signal-to-noise ratio and high frame rate. The resolution and power consumption corresponding to the third resolution mode are smaller than those corresponding to the second resolution mode. The signal-to-noise ratio and the frame rate corresponding to the third resolution mode are larger than those corresponding to the second resolution mode.

The third resolution mode may be, but not limited to, a preview mode at the time of image capturing, a preview mode at the time of video capturing, or a scene having a low resolution requirement such as a night scene mode for image capturing or video capturing under a night scene. The preview mode of video capturing is, for example, 720p video preview, application video preview, or the like.

The electronics determine each filter set from the filter array, acquire a plurality of second panchromatic pixel values in the first panchromatic image obtained for each filter set, and combine the plurality of second panchromatic pixel values to read out a third panchromatic pixel value. The electronic device reads pixel values from the respective third full-color pixel values to a second full-color image according to a preset pixel reading mode.

Step 2204, merging and reading out the third color pixel value of the first color, the third color pixel value of the second color and the third color pixel value of the third color from the second color pixel values of the plurality of same colors corresponding to the same filter set in the first color image, and generating a second color image of double colors and a second color image of single colors based on the third color pixel value of the first color, the third color pixel value of the second color and the third color pixel value of the third color; the second color image of the double color includes a third color pixel value of the first color and a third color pixel value of the third color, and the second color image of the single color includes a third color pixel value of the second color.

The third color pixel value of the first color is the pixel value read out by the pixel corresponding to the first color filter, the third color pixel value of the second color is the pixel value read out by the pixel corresponding to the second color filter, and the third color pixel value of the third color is the pixel value read out by the pixel corresponding to the third color filter.

The electronic device determines each filter set from the filter array, acquires a plurality of second color pixel values of the same color in the first color image obtained by each filter set, wherein the second color pixel values of the same color comprise a second color pixel value of a first color, a second color pixel value of a second color and a second color pixel value of a third color, combines the second color pixel values of the plurality of first colors to read out a third color pixel value of the first color, combines the second color pixel values of the plurality of second colors to read out a third color pixel value of the second color, and combines the second color pixel values of the plurality of third colors to read out a third color pixel value of the third color.

The second color image of the dual color includes third color pixel values of the first color and third color pixel values of the third color. The single color second color image includes third color pixel values of the second color. Wherein the third color pixel values of the first color are arranged on a diagonal of the second color image of the double color and the third color pixel values of the third color are arranged on an opposite diagonal of the second color image of the double color.

Taking the first color image and the first panchromatic image of FIG. 17 as an example, in the third resolution mode, the electronics combine the plurality of second panchromatic pixel values in the first panchromatic image 1704 corresponding to the same set of filters to read out third panchromatic pixel values and generate second panchromatic image 2302 in FIG. 23 based on each third panchromatic pixel value; the resolution corresponding to the third resolution mode is smaller than the resolution corresponding to the second resolution mode; the second color pixel values of the first color image 1702 corresponding to the plurality of the same colors in the same filter set are combined and read out to form a third color pixel value of the first color, a third color pixel value of the second color, and a third color pixel value of the third color, and the second color image 2304 of the double color and the second color image 2306 of the single color in fig. 23 are generated based on the third color pixel value of the first color, the third color pixel value of the second color, and the third color pixel value of the third color. In fig. 23, w denotes a third full-color pixel value, and a, b, and c denote third color pixel values of three different colors.

Step 2206 generates a third target image based on the second full color image, the bi-color second color image, and the single color second color image.

When the second full-color image, the second color image of the double color, and the second color image of the single color need to be packaged for transmission, the electronic device may generate a third target image based on the second full-color image, the second color image of the double color, and the second color image of the single color, and then transmit the third target image.

Specifically, the electronic device alternately arranges the third full-color pixel values of each row in the second full-color image, the third color pixel values of each row in the double-color second color image and the third color pixel values of each row in the single-color second color image to generate a second target image; or arranging the third full-color pixel value of each column in the second full-color image, the third color pixel value of each column in the double-color second color image and the third color pixel value of each column in the single-color second color image alternately to generate a second target image.

Taking the second full-color image 2302, the second color image 2304 of two colors, and the second color image 2306 of a single color as an example, fig. 24 is a third target image generated by arranging third full-color pixel values of each line in the second full-color image, third color pixel values of each line in the second color image of two colors, and third color pixel values of each line in the second color image of a single color alternately in one embodiment; fig. 25 is a third target image generated by alternately arranging third full-color pixel values of each column in a second full-color image, third color pixel values of each column in a double-color second color image, and third color pixel values of each column in a single-color second color image in another embodiment.

The third full-color pixel value in the second full-color image, the third color pixel value in the second color image, and the third color pixel value in the second color image are not limited to the order of arrangement.

In other embodiments, the electronic device may also generate the third target image in other ways, which are not limited herein.

In this embodiment, in the third resolution mode, the third panchromatic pixel value is read by combining the plurality of second panchromatic pixel values corresponding to the same filter set in the first panchromatic image, the third color pixel value of the first color, the third color pixel value of the second color and the third color pixel value of the third color are read by combining the plurality of second color pixel values corresponding to the same color in the same filter set in the first color image, and different color pixels can be mixed and arranged, so that the third color pixels in the generated third target image, such as RGB pixels, are distributed more uniformly and have higher image quality. And the resolution and the image size of the obtained third target image are further reduced, the panchromatic pixels have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of three-level pixel merging output is achieved. In the third resolution mode, the third target image includes full-color pixels arranged in full, so that the overall resolution can be improved. Meanwhile, in the third resolution mode, pixels of the same color do not need to be combined across the period, interpolation does not need to be carried out, and the overall resolution is improved. In the full-size case, color pixels of the respective colors, such as the pixel value of the first color and the pixel value of the third color, are more scatter-balanced on the diagonal or the anti-diagonal. Wherein, the full arrangement means that each coordinate has the pixel, and interpolation estimation is not needed.

In another embodiment, the method further comprises: in the third resolution mode, merging and reading out sixth panchromatic pixel values from panchromatic pixels corresponding to all panchromatic sub-filters of the plurality of panchromatic filters in each filter set, and generating a fourth panchromatic image based on the sixth panchromatic pixel values; combining and reading out a sixth color pixel value of the first color, a sixth color pixel value of the second color and a sixth color pixel value of the third color from color pixels corresponding to each color sub-filter of the plurality of color filters of the same color in each filter set, and generating a fourth color image of double colors and a fourth color image of single color based on the sixth color pixel value of the first color, the sixth color pixel value of the second color and the sixth color pixel value of the third color; the fourth color image of the double color comprises a sixth color pixel value of the first color and a sixth color pixel value of the third color, and the fourth color image of the single color comprises a sixth color pixel value of the second color; a sixth target image is generated based on the fourth full-color image, the double-color fourth color image, and the single-color fourth color image.

When the fourth full-color image, the double-color fourth color image and the single-color fourth color image need to be packaged and transmitted, the electronic device may generate a sixth target image based on the fourth full-color image, the double-color fourth color image and the single-color fourth color image, and then transmit the sixth target image.

Wherein generating a sixth target image based on the fourth full-color image, the double-color fourth color image, and the single-color fourth color image, includes: arranging each row of sixth panchromatic pixel values in the fourth panchromatic image, each row of sixth color pixel values in the double-color fourth color image and each row of sixth color pixel values in the single-color fourth color image at intervals to generate a sixth target image; or arranging each column of sixth panchromatic pixel values in the fourth panchromatic image, each column of sixth color pixel values in the double-color fourth color image and each column of sixth color pixel values in the single-color fourth color image alternately to generate a sixth target image.

In other embodiments, the electronic device may also generate the sixth target image in other ways, which are not limited herein.

In the present embodiment, in the third resolution mode, the fifth panchromatic pixel value is read out by combining the panchromatic pixels corresponding to the respective panchromatic sub-filters of the plurality of panchromatic filters in each filter group, so that the fourth panchromatic image can be generated more quickly; and combining and reading out the fifth color pixel value of the first color, the fifth color pixel value of the second color and the fifth color pixel value of the third color from the color pixels corresponding to the color sub-filters of the plurality of color filters with the same color in each filter set, so that a double-color fourth color image and a single-color fourth color image can be generated more quickly. Moreover, the above embodiment can mix and arrange the different color pixels, so that the sixth color pixel values, such as RGB pixels, in the generated sixth target image are distributed more uniformly and the image quality is higher. In addition, the resolution and the image size of the obtained sixth target image are further reduced, the panchromatic pixels 241 have higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect of lower power consumption and better signal-to-noise ratio of three-level pixel merging output is achieved.

In one embodiment, as shown in fig. 26, the method further includes:

Step 2602, in a fourth resolution mode, combining and reading out fourth panchromatic pixel values from each third panchromatic pixel value in the second panchromatic image; the fourth resolution mode corresponds to a resolution less than the resolution corresponding to the third resolution mode.

The fourth resolution mode is a mode used in a scene with lower resolution requirements than the third resolution mode, and is a four-level pixel merging read mode with low resolution, low power consumption, high signal to noise ratio and high frame rate. The resolution and power consumption corresponding to the fourth resolution mode are smaller than those corresponding to the third resolution mode. The signal-to-noise ratio and the frame rate corresponding to the fourth resolution mode are larger than those corresponding to the third resolution mode.

The fourth resolution mode may be, but not limited to, a preview mode at the time of image capturing, a preview mode at the time of video capturing, or a scene having a low resolution requirement such as a night scene mode for image capturing or video capturing under a night scene. Preview modes of video capturing are, for example, 480p video preview, application video preview, and the like.

The electronic device determines each minimal repeating unit from the filter array, acquires a plurality of third panchromatic pixel values in the second panchromatic image obtained by each minimal repeating unit, and combines the plurality of third panchromatic pixel values to read out a fourth panchromatic pixel value.

Step 2604, combining the third color pixel values of the plurality of first colors in the second color image of the double color to read the fourth color pixel value of the first color, combining the third color pixel values of the plurality of third colors in the second color image of the double color to read the fourth color pixel value of the third color, and combining the third color pixel values of the plurality of second colors in the second color image of the single color to read the fourth color pixel value of the second color.

The electronic equipment determines each minimum repeating unit from the optical filter array, acquires a plurality of third color pixel values of the same color in the second color image obtained by each minimum repeating unit, wherein the third color pixel values of the same color comprise a third color pixel value of a first color, a third color pixel value of a second color and a third color pixel value of the third color, combines the third color pixel values of the first color to read out a fourth full-color pixel value of the first color, combines the third color pixel values of the second color to read out a fourth full-color pixel value of the second color, and combines the third color pixel values of the third color to read out a fourth full-color pixel value of the third color.

Step 2606 generates a fourth target image based on the fourth panchromatic pixel value, the fourth color pixel value of the first color, the fourth color pixel value of the second color, and the fourth color pixel value of the third color.

Specifically, the electronic device alternately arranges fourth full-color pixel values corresponding to the same minimum repeating unit, fourth color pixel values of the first color, fourth color pixel values of the second color, and fourth color pixel values of the third color, to generate a fourth target image. The fourth full-color pixel value corresponding to the same minimum repeating unit, the fourth color pixel value of the first color, the fourth color pixel value of the second color, and the fourth color pixel value of the third color are not limited to the order in arrangement.

Taking the second full-color image 2302, the second color image 2304 of two colors, and the second color image 2306 of a single color in fig. 23 as an example, fig. 27 is a schematic diagram of a fourth target image in one embodiment.

In this embodiment, in the fourth resolution mode, the fourth panchromatic pixel value is read out by combining the third panchromatic pixel values in the second panchromatic image, the fourth color pixel value of the first color is read out by combining the third color pixel values of the first colors in the second color image, the fourth color pixel value of the third color is read out by combining the third color pixel values of the third colors in the second color image, and the fourth color pixel value of the second color is read out by combining the third color pixel values of the second colors in the second color image, so that the resolution and the image size of the fourth target image are further reduced, the panchromatic pixel has a higher signal-to-noise ratio, and the frame rate of the image is high, thereby achieving the image processing effects of lower power consumption and better signal-to-noise ratio of the four-level pixel combining output. In addition, in the fourth resolution mode, the fourth target image can be matched with the high-pixel image sensor, and the high resolution under high pixels and the high signal-to-noise ratio under low pixels are both considered. Meanwhile, in the fourth resolution mode, pixels of the same color do not need to be combined across the period, interpolation does not need to be carried out, and the overall resolution is improved.

In another embodiment, the method further comprises: in the fourth resolution mode, combining and reading out a seventh panchromatic pixel value from panchromatic pixels corresponding to all panchromatic sub-filters of the plurality of panchromatic filters in the minimum repetition, and combining and reading out a seventh color pixel value from color pixels corresponding to all color sub-filters of the plurality of color filters with the same color in the minimum repetition unit; a seventh target image is generated based on each of the seventh panchromatic pixel values and each of the seventh color pixel values.

The seventh full-color pixel value and each of the seventh color pixel values corresponding to the same minimum repeating unit are not limited in order in arrangement. The color filter comprises a first color filter, a second color filter and a third color filter. The electronic device combines the color pixels corresponding to the sub-filters of the plurality of first color filters in the minimum repeating unit to read out a seventh color pixel value of the first color, combines the color pixels corresponding to the sub-filters of the plurality of second color filters in the minimum repeating unit to read out a seventh color pixel value of the second color, and combines the color pixels corresponding to the sub-filters of the plurality of third color filters in the minimum repeating unit to read out a seventh color pixel value of the third color.

In this embodiment, in the fourth resolution mode, the seventh panchromatic pixel value is read by merging panchromatic pixels corresponding to each panchromatic sub-filter of the plurality of panchromatic filters in the minimum repetition, the seventh color pixel value is read by merging color pixels corresponding to each color sub-filter of the plurality of color filters of the same color in the minimum repetition unit, the resolution and the image size of the seventh target image obtained based on each seventh panchromatic pixel value and each seventh color pixel value are further reduced, the panchromatic pixels have a higher signal-to-noise ratio, and the frame rate of the image is high, so that the image processing effect that the power consumption of the merging output of the fourth-level pixels is lower and the signal-to-noise ratio is better is achieved. In addition, in the fourth resolution mode, the fourth target image can be matched with the high-pixel image sensor, and the high resolution under high pixels and the high signal-to-noise ratio under low pixels are both considered. Meanwhile, in the fourth resolution mode, pixels of the same color do not need to be combined across the period, interpolation does not need to be carried out, and the overall resolution is improved.

In one embodiment, another image generation method is provided and applied to an image sensor, wherein the image sensor comprises a filter array and a pixel array, the filter array comprises a minimum repeating unit, the minimum repeating unit comprises a plurality of filter sets, and each filter set comprises a first subunit and a second subunit; the first subunit and the second subunit respectively comprise a full-color filter and a color filter, the color filters in the first subunit are arranged on diagonal lines in the first subunit, and the color filters in the second subunit are arranged on opposite diagonal lines in the second subunit; the light inlet amount of the full-color filter is larger than that of the color filter; each pixel in the pixel array is arranged corresponding to a filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal; the image generation method comprises the following steps: in the full resolution mode, reading out full resolution panchromatic pixel values from panchromatic pixels corresponding to each panchromatic filter, and reading out full resolution color pixel values from color pixels corresponding to each color filter; a full resolution target image is generated based on each full resolution panchromatic pixel value and each full resolution color pixel value.

The principle of generating the full-resolution target image in this embodiment is similar to that of generating the full-resolution target image in the embodiment of fig. 12, and will not be described here.

It should be noted that, if N is 1, that is, the optical filter does not include a sub-optical filter, the optical filter array further has a first resolution mode, a second resolution mode and a third resolution mode, and the principle of the first resolution mode corresponding to N is similar to the principle of the second resolution mode corresponding to N greater than or equal to 2, the principle of the second resolution mode corresponding to N1 is similar to the principle of the third resolution mode corresponding to N greater than or equal to 2, and the principle of the third resolution mode corresponding to N1 is similar to the principle of the fourth resolution mode corresponding to N greater than or equal to 2, which will not be repeated herein.

It should be understood that, although the steps in the flowcharts of fig. 13, 14, 16, 22, and 26 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps of fig. 13, 14, 16, 22, and 26 may include a plurality of sub-steps or stages that are not necessarily performed at the same time but may be performed at different times, nor does the order in which the sub-steps or stages are performed necessarily occur sequentially, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

Fig. 28 is a block diagram of the structure of an image generating apparatus of an embodiment. As shown in fig. 28, there is provided an image generating apparatus applied to an image sensor including a filter array including a minimum repeating unit including a plurality of filter sets each including a first subunit and a second subunit, and a pixel array; the first subunit and the second subunit respectively comprise a full-color filter and a color filter, the color filters in the first subunit are arranged on diagonal lines in the first subunit, and the color filters in the second subunit are arranged on opposite diagonal lines in the second subunit; the light inlet amount of the full-color filter is larger than that of the color filter; each full-color filter comprises N lines and N columns of full-color sub-filters, each color filter comprises N lines and N columns of color sub-filters, the N lines and the N columns of color sub-filters are the same as the color of the color filter, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

the image generation device includes: a readout module 2802 and an image generation module 2804, wherein:

the readout module 2802 is configured to, in the full resolution mode, read out full resolution panchromatic pixel values from panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filters, and read out full resolution color pixel values from color pixels corresponding to each color sub-filter in the color filters.

An image generation module 2804 generates a full resolution target image based on each full resolution panchromatic pixel value and each full resolution color pixel value.

The image generating device reads out full-resolution panchromatic pixel values from panchromatic pixels corresponding to each panchromatic sub-filter in the panchromatic filters and reads out full-resolution color pixel values from color pixels corresponding to each color sub-filter in the color filters in a full-resolution mode; the light quantity of the full-color filter is larger than that of the color filter, so that full-color channel information can be fused into an image, the overall light quantity is improved, and a full-resolution target image with more information and clearer detail analysis can be generated based on all full-resolution full-color pixel values and all full-resolution color pixel values.

In one embodiment, the readout module 2802 is further configured to, in the first resolution mode, combine and read out the first panchromatic pixel value with the panchromatic pixel corresponding to each panchromatic sub-filter in each panchromatic filter, and combine and read out the first color pixel value with the color pixel corresponding to each color sub-filter in each color filter; the resolution corresponding to the first resolution mode is smaller than the resolution corresponding to the full resolution mode; the image generation module 2804 described above is also configured to generate a first target image based on each first panchromatic pixel value and each first color pixel value.

In one embodiment, the readout module 2802 is further configured to combine, in the second resolution mode, the plurality of first panchromatic pixel values corresponding to each sub-unit in the first target image to read out the second panchromatic pixel value, where the sub-units include a first sub-unit and a second sub-unit; the image generation module 2804 described above is also configured to generate a first full-color image based on each second full-color pixel value; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode; the readout module 2802 is further configured to combine and read out second color pixel values from the first target image corresponding to a plurality of first color pixel values of the same color in each sub-unit, and the image generating module 2804 is further configured to generate a first color image based on each of the second color pixel values; the image generation module 2804 described above is also configured to generate a second target image based on the first full color image and the first color image.

In one embodiment, the image generating module 2804 is further configured to alternately arrange the second panchromatic pixel values of each line in the first panchromatic image and the second color pixel values of each line in the first color image to generate a second target image; or arranging the second panchromatic pixel values of each column in the first panchromatic image and the second color pixel values of each column in the first color image alternately to generate a second target image.

In one embodiment, the readout module 2802 is further configured to combine, in a third resolution mode, a plurality of second panchromatic pixel values corresponding to the same filter set in the first panchromatic image to read out third panchromatic pixel values, and the image generation module 2804 is further configured to generate the second panchromatic image based on each of the third panchromatic pixel values; the resolution corresponding to the third resolution mode is smaller than the resolution corresponding to the second resolution mode; the readout module 2802 is further configured to combine and read out a third color pixel value of the first color, a third color pixel value of the second color, and a third color pixel value of the third color in the first color image corresponding to a plurality of second color pixels of the same color in the same filter set, and the image generating module 2804 is further configured to generate a second color image of a double color and a second color image of a single color based on the first color pixel value of the first color, the third color pixel value of the second color, and the third color pixel value of the third color; the second color image of the double color comprises a third color pixel value of the first color and a third color pixel value of the third color, and the second color image of the single color comprises a third color pixel value of the second color; the image generation module 2804 described above is also configured to generate a third target image based on the second full-color image, the second color image of two colors, and the second color image of a single color.

In one embodiment, the image generating module 2804 is further configured to alternately arrange the third panchromatic pixel values of each row in the second panchromatic image, the third color pixel values of each row in the dual-color second color image, and the third color pixel values of each row in the single-color second color image, so as to generate a second target image; or arranging the third full-color pixel value of each column in the second full-color image, the third color pixel value of each column in the double-color second color image and the third color pixel value of each column in the single-color second color image alternately to generate a second target image.

In one embodiment, the readout module 2802 is further configured to combine the third panchromatic pixel values in the second panchromatic image to read out the fourth panchromatic pixel values in a fourth resolution mode; the resolution corresponding to the fourth resolution mode is smaller than the resolution corresponding to the third resolution mode; the readout module 2802 is further configured to combine the third color pixel values of the plurality of first colors in the second color image of the two colors to read out the fourth color pixel value of the first color, combine the third color pixel values of the plurality of third colors in the second color image of the two colors to read out the fourth color pixel value of the third color, and combine the third color pixel values of the plurality of second colors in the second color image of the single color to read out the fourth color pixel value of the second color; the image generation module 2804 described above is also configured to generate a fourth target image based on a fourth panchromatic pixel value, a fourth color pixel value of the first color, a fourth color pixel value of the second color, and a fourth color pixel value of the third color.

In one embodiment, the readout module 2802 is further configured to combine, in the second resolution mode, the panchromatic pixels corresponding to the panchromatic sub-filters of the plurality of panchromatic filters in each of the first sub-unit or the second sub-unit to read out the fifth panchromatic pixel values, and the image generating module 2804 is further configured to generate the third panchromatic image based on the respective fifth panchromatic pixel values; the readout module 2802 is further configured to combine and read out fifth color pixel values from color pixels corresponding to respective color sub-filters of a plurality of color filters of the same color in each of the first sub-unit or the second sub-unit, and the image generating module 2804 is further configured to generate a third color image based on the respective fifth color pixel values; the image generation module 2804 described above is also configured to generate a fifth target image based on the third full-color image and the third color image.

In one embodiment, the image generating module 2804 is further configured to alternately arrange the fifth panchromatic pixel values of each row in the third panchromatic image and the fifth color pixel values of each row in the third color image to generate a fifth target image; or arranging the fifth panchromatic pixel value of each column in the third panchromatic image and the fifth color pixel value of each column in the third color image alternately to generate a fifth target image.

In one embodiment, the readout module 2802 is further configured to combine, in the third resolution mode, full-color pixels corresponding to the respective full-color sub-filters of the plurality of full-color filters in each filter set to read out sixth full-color pixel values, and the image generating module 2804 is further configured to generate a fourth full-color image based on the respective sixth full-color pixel values; the readout module 2802 is further configured to combine color pixels corresponding to color sub-filters of the plurality of color filters of the same color in each filter set to read out a sixth color pixel value of the first color, a sixth color pixel value of the second color, and a sixth color pixel value of the third color, and the image generating module 2804 is further configured to generate a fourth color image of a double color and a fourth color image of a single color based on the sixth color pixel value of the first color, the sixth color pixel value of the second color, and the sixth color pixel value of the third color; the fourth color image of the double color comprises a sixth color pixel value of the first color and a sixth color pixel value of the third color, and the fourth color image of the single color comprises a sixth color pixel value of the second color; the image generation module 2804 described above is also configured to generate a sixth target image based on a fourth full-color image, a double-color fourth color image, and a single-color fourth color image.

In one embodiment, the image generating module 2804 is further configured to alternately arrange each row of sixth panchromatic pixel values in the fourth panchromatic image, each row of sixth color pixel values in the dual-color fourth color image, and each row of sixth color pixel values in the single-color fourth color image, to generate a sixth target image; or arranging each column of sixth panchromatic pixel values in the fourth panchromatic image, each column of sixth color pixel values in the double-color fourth color image and each column of sixth color pixel values in the single-color fourth color image alternately to generate a sixth target image.

In one embodiment, the readout module 2802 is further configured to, in the fourth resolution mode, combine and read out a seventh panchromatic pixel value with panchromatic pixels corresponding to each of the panchromatic sub-filters of the plurality of panchromatic filters in the minimum repetition, and combine and read out a seventh color pixel value with color pixels corresponding to each of the color sub-filters of the plurality of color filters of the same color in the minimum repetition unit; the image generation module 2804 described above is also configured to generate a seventh target image based on each seventh panchromatic pixel value and each seventh color pixel value.

The division of the various modules in the image generation device described above is for illustration only, and in other embodiments, the image generation device may be divided into different modules as needed to perform all or part of the functions of the image generation device described above.

For specific limitations of the image generating apparatus, reference may be made to the above limitations of the image generating method, and no further description is given here. The respective modules in the above-described image generating apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

Fig. 29 is a schematic diagram showing an internal structure of the electronic device in one embodiment. The electronic device may be any terminal device such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a PDA (Personal digital assistant), a POS (Point of Sales), a car-mounted computer, and a wearable device. The electronic device includes a processor and a memory connected by a system bus. Wherein the processor may comprise one or more processing units. The processor may be a CPU (Central Processing Unit ) or DSP (DIGITAL SIGNAL Processing, digital signal processor) or the like. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program is executable by a processor for implementing an image generation method provided in the following embodiments. The internal memory provides a cached operating environment for operating system computer programs in the non-volatile storage medium.

The implementation of each module in the image generating apparatus provided in the embodiment of the present application may be in the form of a computer program. The computer program may run on a terminal or a server. Program modules of the computer program may be stored in the memory of the electronic device. Which when executed by a processor, performs the steps of the method described in the embodiments of the application.

The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of an image generation method.

Embodiments of the present application also provide a computer program product containing instructions which, when run on a computer, cause the computer to perform an image generation method.

Any reference to memory, storage, database, or other medium used in the present application may include non-volatile and/or volatile memory. The nonvolatile Memory may include a ROM (Read-Only Memory), a PROM (Programmable Read-Only Memory), an EPROM (Erasable Programmable Read-Only Memory ), an EEPROM (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory), or a flash Memory. Volatile memory can include RAM (Random Access Memory ), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as SRAM (Static Random Access Memory ), DRAM (Dynamic Random Access Memory, dynamic random access memory), SDRAM (Synchronous Dynamic Random Access Memory ), double data rate DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access memory, double data rate synchronous dynamic random access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory ), SLDRAM (SYNC LINK DYNAMIC Random Access Memory, synchronous link dynamic random access memory), RDRAM (Rambus Dynamic Random Access Memory, bus dynamic random access memory), DRDRAM (Direct Rambus Dynamic Random Access Memory, interface dynamic random access memory).

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims

1. An image sensor comprising a filter array and a pixel array, the filter array comprising a minimal repeating unit, the minimal repeating unit comprising 2 first filter sets and 2 second filter sets, the 2 first filter sets arranged on a diagonal of the minimal repeating unit, the 2 second filter sets arranged on an opposite corner of the minimal repeating unit, each filter set comprising 2 first subunits and 2 second subunits; the first subunit and the second subunit each comprise a full-color filter and a color filter, the color filters with the same color in the first subunit are arranged on diagonal lines in the first subunit, the color filters with the same color in the second subunit are arranged on opposite diagonal lines in the second subunit, the color filters in the first filter set comprise a first color filter and a second color filter, and the color filters in the second filter set comprise a second color filter and a third color filter; one of the 2 first subunits in the first filter set comprises a first color filter, the other first subunit comprises a second color filter, one of the 2 second subunits in the first filter set comprises a first color filter, and the other second subunit comprises a second color filter; one of the 2 first subunits in the second filter set comprises a second color filter, and the other first subunit comprises a third color filter; one of the 2 second subunits in the second filter set comprises a second color filter, and the other second subunit comprises a third color filter; the light inlet amount of the full-color filter is larger than the light inlet amount of the color filter; each full-color filter comprises N lines and N columns of full-color sub-filters, each color filter comprises N lines and N columns of color sub-filters, the colors of the N lines and the N columns of color sub-filters are the same as those of the color filters, and N is a positive integer; each pixel in the pixel array is disposed corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal.

2. The image sensor of claim 1, wherein each row and each column includes a color filter for each color.

3. The image sensor of claim 1, wherein the 2 first filter sets are identical and the 2 second filter sets are identical.

4. The image sensor of claim 1, wherein each filter set comprises only full color filters and 2 color filters.

5. The image sensor of claim 1, wherein the 2 first subunits are arranged in a first row direction of the filter set and the 2 second subunits are arranged in a second row direction of the filter set, the first row direction and the second row direction being adjacently arranged; or the 2 first subunits are arranged in a first column direction of the optical filter set, and the 2 second subunits are arranged in a second column direction of the optical filter set, wherein the first column direction and the second column direction are adjacently arranged.

6. The image sensor of claim 5, wherein the first sub-unit and the second sub-unit comprising the same color filter in the same filter set are arranged on a diagonal or an anti-diagonal of the same filter set.

7. The image sensor of claim 1, wherein N is 1, and the minimal repeating unit comprises 8 rows and 8 columns of 64 filters arranged in a manner that:

b w a w b w c w

w b w a w b w c

w a w b w c w b

a w b w c w b w

b w c w b w a w

w b w c w b w a

w c w b w a w b

c w b w a w b w

Or alternatively

b w c w b w a w

w b w c w b w a

w c w b w a w b

c w b w a w b w

b w a w b w c w

w b w a w b w c

w a w b w c w b

a w b w c w b w

Or alternatively

w a b w w c b w

a w w b c w w b

w b a w w b c w

b w w a b w w c

w c b w w a b w

c w w b a w w b

w b c w w b a w

b w w c b w w a

Or alternatively

w c b w w a b w

c w w b a w w b

w b c w w b a w

b w w c b w w a

w a b w w c b w

a w w b c w w b

w b a w w b c w

b w w a b w w c

Wherein w represents a full-color filter, and a, b and c each represent a color filter.

8. The image sensor of claim 1, wherein N is 2, and the minimal repeating unit comprises 16 rows and 16 columns of 256 sub-filters arranged in a manner that:

b b w w a a w w b b w w c c w w

w w b b w w a a w w b b w w c c

w w a a w w b b w w c c w w b b

a a w w b b w w c c w w b b w w

b b w w c c w w b b w w a a w w

w w b b w w c c w w b b w w a a

w w c c w w b b w w a a w w b b

c c w w b b w w a a w w b b w w

Or alternatively

b b w w c c w w b b w w a a w w

w w b b w w c c w w b b w w a a

w w c c w w b b w w a a w w b b

c c w w b b w w a a w w b b w w

b b w w a a w w b b w w c c w w

w w b b w w a a w w b b w w c c

w w a a w w b b w w c c w w b b

a a w w b b w w c c w w b b w w

Or alternatively

w w a a b b w w w w c c b b w w

a a w w w w b b c c w w w w b b

w w b b a a w w w w b b c c w w

b b w w w w a a b b w w w w c c

w w c c b b w w w w a a b b w w

c c w w w w b b a a w w w w b b

w w b b c c w w w w b b a a w w

b b w w w w c c b b w w w w a a

Or alternatively

w w c c b b w w w w a a b b w w

c c w w w w b b a a w w w w b b

w w b b c c w w w w b b a a w w

b b w w w w c c b b w w w w a a

w w a a b b w w w w c c b b w w

a a w w w w b b c c w w w w b b

w w b b a a w w w w b b c c w w

b b w w w w a a b b w w w w c c

9. A camera module comprising a lens and the image sensor of any one of claims 1-8; the image sensor is used for receiving the light passing through the lens and generating an electric signal according to the light.

10. An electronic device, comprising:

The camera module of claim 9; and

The camera module is arranged on the shell.

11. An image generation method is applied to an image sensor, and is characterized in that the image sensor comprises an optical filter array and a pixel array, the optical filter array comprises a minimum repeating unit, the minimum repeating unit comprises 2 first optical filter sets and 2 second optical filter sets, the 2 first optical filter sets are arranged on the diagonal of the minimum repeating unit, the 2 second optical filter sets are arranged on the opposite corner of the minimum repeating unit, and each optical filter set comprises 2 first subunits and 2 second subunits; the first subunit and the second subunit each comprise a full-color filter and a color filter, the color filters with the same color in the first subunit are arranged on diagonal lines in the first subunit, the color filters with the same color in the second subunit are arranged on opposite diagonal lines in the second subunit, the color filters in the first filter set comprise a first color filter and a second color filter, and the color filters in the second filter set comprise a second color filter and a third color filter; one of the 2 first subunits in the first filter set comprises a first color filter, the other first subunit comprises a second color filter, one of the 2 second subunits in the first filter set comprises a first color filter, and the other second subunit comprises a second color filter; one of the 2 first subunits in the second filter set comprises a second color filter, and the other first subunit comprises a third color filter; one of the 2 second subunits in the second filter set comprises a second color filter, and the other second subunit comprises a third color filter; the light inlet amount of the full-color filter is larger than the light inlet amount of the color filter; each full-color filter comprises N lines and N columns of full-color sub-filters, each color filter comprises N lines and N columns of color sub-filters, the colors of the N lines and the N columns of color sub-filters are the same as those of the color filters, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

The method comprises the following steps:

12. The method of claim 11, wherein the method further comprises:

In a first resolution mode, merging and reading out a first panchromatic pixel value from panchromatic pixels corresponding to all panchromatic sub-filters in each panchromatic filter, and merging and reading out a first color pixel value from color pixels corresponding to all color sub-filters in each color filter; the resolution corresponding to the first resolution mode is smaller than the resolution corresponding to the full resolution mode;

a first target image is generated based on each of the first panchromatic pixel values and each of the first color pixel values.

13. The method of claim 12, wherein after generating the first target image, further comprising:

In a second resolution mode, combining and reading out second panchromatic pixel values corresponding to the plurality of first panchromatic pixel values in each subunit in the first target image, and generating a first panchromatic image based on each of the second panchromatic pixel values; the resolution corresponding to the second resolution mode is smaller than the resolution corresponding to the first resolution mode, and the subunit comprises the first subunit and the second subunit;

combining and reading out second color pixel values of a plurality of first color pixel values corresponding to the same color in each subunit in the first target image, and generating a first color image based on each second color pixel value;

a second target image is generated based on the first full color image and the first color image.

14. The method of claim 13, wherein the generating a second target image based on the first full color image and the first color image comprises:

Arranging second panchromatic pixel values of each row in the first panchromatic image and second color pixel values of each row in the first color image alternately to generate a second target image; or alternatively

And arranging the second panchromatic pixel values of each column in the first panchromatic image and the second color pixel values of each column in the first color image alternately to generate a second target image.

15. The method of claim 13, wherein the method further comprises:

In a third resolution mode, combining and reading out third panchromatic pixel values corresponding to a plurality of second panchromatic pixel values in the same filter set in the first panchromatic image, and generating a second panchromatic image based on each third panchromatic pixel value; the resolution corresponding to the third resolution mode is smaller than the resolution corresponding to the second resolution mode;

Combining and reading out a third color pixel value of a first color, a third color pixel value of a second color and a third color pixel value of a third color from a plurality of second color pixel values of the same color corresponding to the same filter set in the first color image, and generating a second color image of double colors and a second color image of single colors based on the third color pixel value of the first color, the third color pixel value of the second color and the third color pixel value of the third color; the second color image of the double color comprises a third color pixel value of the first color and a third color pixel value of the third color, and the second color image of the single color comprises a third color pixel value of the second color;

a third target image is generated based on the second full-color image, the two-color second color image, and the single-color second color image.

16. The method of claim 15, wherein the generating a third target image based on the second full-color image, the bi-color second color image, and the single-color second color image comprises:

Arranging the third full-color pixel value of each row in the second full-color image, the third color pixel value of each row in the double-color second color image and the third color pixel value of each row in the single-color second color image at intervals to generate a second target image; or alternatively

And arranging the third full-color pixel value of each column in the second full-color image, the third color pixel value of each column in the double-color second color image and the third color pixel value of each column in the single-color second color image at intervals to generate a second target image.

17. The method of claim 15, wherein the method further comprises:

in a fourth resolution mode, combining and reading out fourth panchromatic pixel values from each third panchromatic pixel value in the second panchromatic image; the resolution corresponding to the fourth resolution mode is smaller than the resolution corresponding to the third resolution mode;

Combining and reading out the fourth color pixel values of the first colors of the plurality of first color pixel values in the double-color second color image, combining and reading out the fourth color pixel values of the third colors of the plurality of third color pixel values in the double-color second color image, and combining and reading out the fourth color pixel values of the second colors of the plurality of second color pixel values in the single-color second color image;

A fourth target image is generated based on the fourth panchromatic pixel value, the fourth color pixel value of the first color, the fourth color pixel value of the second color, and the fourth color pixel value of the third color.

18. The method of claim 11, wherein the method further comprises:

In the second resolution mode, merging and reading out fifth panchromatic pixel values from panchromatic pixels corresponding to all panchromatic sub-filters of a plurality of panchromatic filters in each first sub-unit or each second sub-unit, and generating a third panchromatic image based on the fifth panchromatic pixel values;

combining and reading out fifth color pixel values of color pixels corresponding to all color sub-filters of a plurality of color filters with the same color in each first sub-unit or each second sub-unit, and generating a third color image based on all the fifth color pixel values;

A fifth target image is generated based on the third full-color image and the third color image.

19. The method of claim 18, wherein the generating a fifth target image based on the third full-color image and the third color image comprises:

Arranging fifth panchromatic pixel values of each row in the third panchromatic image and fifth color pixel values of each row in the third color image at intervals to generate a fifth target image; or alternatively

And arranging the fifth panchromatic pixel value of each column in the third panchromatic image and the fifth color pixel value of each column in the third color image alternately to generate a fifth target image.

20. The method of claim 11, wherein the method further comprises:

In a third resolution mode, merging and reading out sixth panchromatic pixel values from panchromatic pixels corresponding to all panchromatic sub-filters of the plurality of panchromatic filters in each filter set, and generating a fourth panchromatic image based on the sixth panchromatic pixel values;

Combining and reading out the sixth color pixel value of the first color, the sixth color pixel value of the second color and the sixth color pixel value of the third color from the color pixels corresponding to the color sub-filters of the plurality of color filters of the same color in each filter set, and generating a fourth color image of double colors and a fourth color image of single color based on the sixth color pixel value of the first color, the sixth color pixel value of the second color and the sixth color pixel value of the third color; the fourth color image of the double color comprises a sixth color pixel value of the first color and a sixth color pixel value of the third color, and the fourth color image of the single color comprises a sixth color pixel value of the second color;

a sixth target image is generated based on the fourth full-color image, the double-color fourth color image, and the single-color fourth color image.

21. The method of claim 20, wherein the generating a sixth target image based on the fourth full-color image, the dual-color fourth color image, and the single-color fourth color image comprises:

Arranging the sixth panchromatic pixel value of each row in the fourth panchromatic image, the sixth color pixel value of each row in the double-color fourth color image and the sixth color pixel value of each row in the single-color fourth color image at intervals to generate a sixth target image; or alternatively

And arranging each column of sixth panchromatic pixel values in the fourth panchromatic image, each column of sixth color pixel values in the double-color fourth color image and each column of sixth color pixel values in the single-color fourth color image at intervals to generate a sixth target image.

22. The method of claim 11, wherein the method further comprises:

In a fourth resolution mode, merging the panchromatic pixels corresponding to all the panchromatic sub-filters of the multiple panchromatic filters in the minimum repeating unit to read out a seventh panchromatic pixel value, merging the color pixels corresponding to all the color sub-filters of the multiple color filters with the same color in the minimum repeating unit to read out a seventh color pixel value;

A seventh target image is generated based on each of the seventh panchromatic pixel values and each of the seventh color pixel values.

23. An image generating device applied to an image sensor, wherein the image sensor comprises an optical filter array and a pixel array, the optical filter array comprises a minimum repeating unit, the minimum repeating unit comprises 2 first optical filter sets and 2 second optical filter sets, the 2 first optical filter sets are arranged on the diagonal of the minimum repeating unit, the 2 second optical filter sets are arranged on the opposite corners of the minimum repeating unit, and each optical filter set comprises 2 first subunits and 2 second subunits; the first subunit and the second subunit each comprise a full-color filter and a color filter, the color filters with the same color in the first subunit are arranged on diagonal lines in the first subunit, the color filters with the same color in the second subunit are arranged on opposite diagonal lines in the second subunit, the color filters in the first filter set comprise a first color filter and a second color filter, and the color filters in the second filter set comprise a second color filter and a third color filter; one of the 2 first subunits in the first filter set comprises a first color filter, the other first subunit comprises a second color filter, one of the 2 second subunits in the first filter set comprises a first color filter, and the other second subunit comprises a second color filter; one of the 2 first subunits in the second filter set comprises a second color filter, and the other first subunit comprises a third color filter; one of the 2 second subunits in the second filter set comprises a second color filter, and the other second subunit comprises a third color filter; the light inlet amount of the full-color filter is larger than the light inlet amount of the color filter; each full-color filter comprises N lines and N columns of full-color sub-filters, each color filter comprises N lines and N columns of color sub-filters, the colors of the N lines and the N columns of color sub-filters are the same as those of the color filters, and N is a positive integer greater than or equal to 2; each pixel in the pixel array is arranged corresponding to a sub-filter of the filter array, and the pixel array is configured to receive light passing through the filter array to generate an electrical signal;

The device comprises:

24. An electronic device comprising a memory and a processor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of the image generation method of any of claims 11 to 22.

25. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any of claims 11 to 22.

26. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 11 to 22.