CN112752009A - Image processing method, module, readable storage medium and image sensor - Google Patents
Image processing method, module, readable storage medium and image sensor Download PDFInfo
- Publication number
- CN112752009A CN112752009A CN201911036821.XA CN201911036821A CN112752009A CN 112752009 A CN112752009 A CN 112752009A CN 201911036821 A CN201911036821 A CN 201911036821A CN 112752009 A CN112752009 A CN 112752009A
- Authority
- CN
- China
- Prior art keywords
- image
- array
- sub
- pixels
- pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Color Television Image Signal Generators (AREA)
Abstract
The present disclosure provides an image processing method, including: acquiring a Bayer raw image, wherein the Bayer raw image comprises a plurality of pixel unit groups arranged in an array, each pixel unit group comprises 4 heterogeneous single-color pixel units arranged in an array of 2 × 2, and each single-color pixel unit comprises 4 same sub-pixels arranged in an array of 2 × 2; sampling and recombining the Bayer original image to generate a first mode image, wherein the first mode image comprises a plurality of reconstruction pixel units which are arranged in an array, each reconstruction pixel unit comprises 4 sub-pixels which are arranged in an array of 2 x 2, the 4 sub-pixels in each reconstruction pixel unit are sampled in the 4 sub-pixels which are arranged in the array of 2 x 2 in the Bayer original image and correspond to 4 different types of single-color pixel units, and the arrangement positions of the 4 sub-pixels in each reconstruction pixel unit in the 2 x 2 array are the same.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, an image processing module, a readable storage medium, an image sensor, and an image capture device.
Background
To enhance the effect of shooting, image sensors adopting a four-in-one pixel aggregation technique (also referred to as a four-in-one pixel combination technique) are now frequently used. Specifically, four small-sized sub-pixels of the same color arranged in an array of 2 × 2 are combined as one large-sized monochrome pixel unit.
For an image sensor of a four-in-one pixel (Quad bayer) aggregation technology, manufacturers may provide a corresponding demosaic (demosaic) algorithm to sample and recombine a bayer raw image generated by a photosensitive pixel array when a user needs to acquire a high-resolution image, so as to obtain a high-resolution image. However, in practical applications, it is found that, in the prior art, a uniformly distributed sampling mode is often adopted for sampling a Bayer (Bayer) original image, and at this time, four sub-pixels constituting a complete pixel are not compact enough in the Bayer original image, and the display resolution of the complete pixel is low, so that the real display effect of a high-resolution image is not good.
Disclosure of Invention
The present disclosure is directed to at least one of the problems in the related art, and provides an image processing method, a module, a readable storage medium, an image sensor and an image capturing device
In a first aspect, an embodiment of the present disclosure provides an image processing method, including:
acquiring a Bayer raw image, wherein the Bayer raw image comprises a plurality of pixel unit groups arranged in an array, each pixel unit group comprises 4 heterogeneous single-color pixel units arranged in an array of 2 × 2, and each single-color pixel unit comprises 4 same sub-pixels arranged in an array of 2 × 2;
sampling and recombining the Bayer original image to generate a first mode image, wherein the first mode image comprises a plurality of reconstruction pixel units which are arranged in an array, each reconstruction pixel unit comprises 4 sub-pixels which are arranged in an array of 2 x 2, the 4 sub-pixels in each reconstruction pixel unit are sampled in the 4 sub-pixels which are arranged in the array of 2 x 2 in the Bayer original image and correspond to 4 different types of single-color pixel units, and the arrangement positions of the 4 sub-pixels in each reconstruction pixel unit in the 2 x 2 array are the same.
In a second aspect, an embodiment of the present disclosure further provides an image processing module, including:
one or more processors;
a memory having one or more programs stored thereon;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the steps in the method as provided by the foregoing embodiments.
In a third aspect, the embodiments of the present disclosure further provide a readable storage medium, on which a program is stored, wherein when the program is executed, the steps in the method provided by the foregoing embodiments are implemented.
In a fourth aspect, embodiments of the present disclosure also provide an image sensor, including:
the photosensitive pixel array is used for sensing light and generating a corresponding Bayer original image; the photosensitive pixel array includes: the device comprises a plurality of photosensitive unit groups arranged in an array, wherein each photosensitive unit group comprises 4 different types of monochromatic photosensitive units arranged in an array of 2 x 2, and each monochromatic photosensitive unit comprises 4 photosensitive elements arranged in an array of 2 x 2;
program module, in which a program is stored, which, when being executed by a processor, carries out the steps of the method as provided in the preceding embodiments.
In a fifth aspect, an embodiment of the present disclosure further provides an image capturing apparatus, including: the image sensor as provided in the previous embodiments.
The present disclosure has the following beneficial effects:
the embodiment of the disclosure provides an image processing method, which includes: acquiring a Bayer original image; sampling and recombining a Bayer original image to generate a first mode image, wherein the first mode image comprises a plurality of reconstruction pixel units which are arranged in an array, each reconstruction pixel unit comprises 4 sub-pixels which are arranged in an array of 2 x 2, the 4 sub-pixels in each reconstruction pixel unit are sampled in the 4 sub-pixels which are arranged in the array of 2 x 2 in the Bayer original image and correspond to 4 different types of single-color pixel units, and the arrangement positions of the 4 sub-pixels in each reconstruction pixel unit in the 2 x 2 array are the same. In the disclosure, 4 sub-pixels constituting the reconstructed pixel unit do not have other sub-pixels between any two sub-pixels in the bayer original image, and the degree of compactness is optimal, so that the technical scheme of the disclosure can improve the display effect of the high-resolution image.
Drawings
FIG. 1 is a schematic diagram of a photosensitive pixel array in a related art using a four-in-one pixel Bayer array;
fig. 2 is a schematic diagram of a bayer original image subjected to uniform sampling and recombination to obtain a resolution image in the related art;
fig. 3 is a flowchart of an image processing method according to an embodiment of the disclosure;
FIG. 4 is a flowchart illustrating an implementation of step S2 in the embodiment of the present disclosure;
fig. 5 is a flowchart illustrating an implementation of step S201 in the embodiment of the present disclosure;
fig. 6 is a schematic diagram of extracting a repeating unit from a bayer raw image according to an embodiment of the present disclosure;
FIG. 7 is a schematic diagram illustrating adjustment of sub-pixel positions within a repeating unit in the present disclosure;
FIG. 8 is a flow chart of another image processing method provided by the embodiments of the present disclosure;
FIG. 9 is a flowchart illustrating a specific step S1a according to an embodiment of the present disclosure;
fig. 10 is a schematic diagram of a four-in-one pixel merging technique used in the embodiment of the present disclosure to perform data merging processing on a bayer original image;
fig. 11 is a block diagram of an image capturing device according to an embodiment of the present disclosure.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present disclosure, an image processing method, a module, a readable storage medium, an image sensor and an image capturing device provided by the present disclosure are described in detail below with reference to the accompanying drawings.
In order to make those skilled in the art better understand the technical solution of the present disclosure, the data monitoring method, the task monitoring method, the data monitoring module, the task monitoring system and the computer readable medium provided in the present disclosure are described in detail below with reference to the accompanying drawings.
Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but which may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a first element, component, or module discussed below could be termed a second element, component, or module without departing from the teachings of the present disclosure.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The technical scheme of the disclosure is based on a four-in-one pixel polymerization technology; specifically, the photosensitive pixel array for sensing light and generating a corresponding bayer raw image is a four-in-one pixel bayer array. Fig. 1 is a schematic layout diagram of a photosensitive pixel array using a four-in-one pixel bayer array in the related art, as shown in fig. 1, the photosensitive pixel array includes: the photosensitive unit groups comprise 4 heterogeneous single-color photosensitive units arranged in an array 2 x 2, and each single-color photosensitive unit comprises 4 photosensitive elements arranged in an array 2 x 2 and used for sensing the same color light.
In the present disclosure, the first green photosensitive unit SensePix _ G, the red photosensitive unit SensePix _ R, the blue photosensitive unit SensePix _ B, and the second green photosensitive unit SensePix _ G' are exemplified by 4 different kinds of monochrome photosensitive units, but the technical solution of the present disclosure is not limited thereto. In the disclosure, the specific photosensitive colors of the 4 different types of single-color photosensitive units can be designed and adjusted according to requirements. Taking the first green photosensitive units as an example, each of the first green photosensitive units SensePix _ G includes 4 photosensitive elements sensing green light arranged in an array of 2 × 2. Each photosensitive element in the photosensitive pixel array can independently sense light, and generate a corresponding electrical signal based on the received exposure, wherein the electrical signal can represent the exposure at the photosensitive element and the brightness corresponding to the light received by the photosensitive element.
It is assumed that the photosensitive pixel array includes MN photosensitive cell groups arranged in M rows and N columns of the array, and the photosensitive pixel array includes 4M × 4N photosensitive elements arranged in 4M rows and 4N columns of the array. With the data sensed by the photosensitive elements as sub-pixels, the bayer raw image generated by the photosensitive pixel array includes 16MN sub-pixels arranged in an array of 4 mx 4N.
Fig. 2 is a schematic diagram of a bayer original image being uniformly sampled and recombined to obtain a resolution image in the related art, and as shown in fig. 2, 16 subpixels included in 4 different types of monochrome pixel units Pix _ G/Pix _ R/Pix _ B/Pix _ G 'arranged in an array of 2 × 2 in the related art are taken as a repeating unit RU, and a subpixel G located at the upper left corner is extracted from the 4 different types of monochrome pixel units Pix _ G/Pix _ R/Pix _ B/Pix _ G', respectively11/R11/B11/G’11Then the extracted 4 sub-pixels G11/R11/B11/G’11Recombined into one reconstructed pixel unit RB. Then, based on the same sampling and recombination manner, 4 sub-pixels G located at the upper right corner in the 4 different types of monochrome pixel units Pix _ G/Pix _ R/Pix _ B/Pix _ G' are arranged12/R12/B12/G’12Recombining the two into a reconstructed pixel unit RB, and arranging 4 sub-pixels G at the lower left corner in the 4 heterogeneous monochromatic pixel units Pix _ G/Pix _ R/Pix _ B/Pix _ G21/R21/B21/G’21Recombining the two into a reconstructed pixel unit RB, and arranging 4 sub-pixels G at the bottom right corner in the 4 heterogeneous monochromatic pixel units Pix _ G/Pix _ R/Pix _ B/Pix _ G22/R22/B22/G’22And are recombined into one reconstructed pixel unit RB, thereby obtaining 4 reconstructed pixel units.
All the repeating units in the Bayer original image are subjected to the same adoption and recombination processing, and finally a high-resolution image with the pixel resolution (one reconstructed pixel unit forms a complete display pixel) of 2 Mx 2N can be obtained.
However, in practical applications, it is found that, in the related art, the positions of the 4 sub-pixels forming one reconstructed pixel unit in the bayer original image are not compact enough, and the display resolution of the reconstructed pixel unit is low, so that the real display effect of the high-resolution image is not good. Taking the reconstructed pixel unit RB in fig. 2 as an example, the 4 sub-pixels constituting the reconstructed pixel unit RB are: first green sub-pixel G11Red sub-pixel R11Blue sub-pixel B11And a second green subpixel G'11(ii) a In the Bayer raw image, for the first green sub-pixel G11Red sub-pixel R11Blue sub-pixel B11And a second green subpixel G'11One other sub-pixel (e.g., the first green sub-pixel G) is present between any two sub-pixels11And red sub-pixel R11There is a first green sub-pixel G in between12) I.e. the 4 sub-pixels G11/R11/B11/G’11The positions in the Bayer raw image are not compact enough, and complete pixels are formed after recombinationThe display resolution is low.
To solve the above technical problem, the present disclosure provides an image processing method that can obtain a high-resolution image with a better display effect based on a bayer raw image.
Fig. 3 is a flowchart of an image processing method according to an embodiment of the present disclosure, and as shown in fig. 3, the method includes:
step S1, a bayer raw image is acquired.
In step S1, a bayer raw image is acquired from a photosensitive pixel array that employs a four-in-one pixel bayer array; the bayer original image comprises a plurality of pixel unit groups arranged in an array, each pixel unit group comprises 4 heterogeneous single-color pixel units arranged in an array of 2 × 2, and each single-color pixel unit comprises 4 identical sub-pixels arranged in an array of 2 × 2.
In this disclosure, "heterogeneous" means in different locations in the 2 x 2 array to which the groups of pixel cells correspond. In addition, the colors corresponding to the two types of monochrome pixel units belonging to different types may be the same or different. In the embodiments of the present disclosure, an example is given in which 4 different types of single-color photosensitive cells in a photosensitive pixel array are respectively a first green photosensitive cell, a red photosensitive cell, a blue photosensitive cell, and a second green photosensitive cell, and each pixel cell includes four sub-pixels of a corresponding color.
It should be noted that the case where the 4 different types of monochrome photosensitive units in the present disclosure adopt the green, red, blue, and green modes (also referred to as GRBG modes) is only for exemplary description, and does not limit the technical solution of the present disclosure. In the present disclosure, the 4 different types of monochrome photosensitive units can also adopt other modes, such as red, green, blue, white mode (also referred to as RGBW mode), and red, yellow, blue mode (also referred to as RYYB mode), which are not illustrated herein.
Step S2, sampling and recombining the Bayer original image to generate a first mode image; the first mode image comprises a plurality of reconstruction pixel units arranged in an array, each reconstruction pixel unit comprises 4 sub-pixels arranged in an array of 2 x 2, the 4 sub-pixels in each reconstruction pixel unit are sampled in the Bayer original image and arranged in the array of 2 x 2 and correspond to 4 sub-pixels of 4 different types of monochrome pixel units, and the arrangement positions of the 4 sub-pixels in each reconstruction pixel unit in the array of 2 x 2 are the same.
Different from the related art, 4 sub-pixels included in each reconstructed pixel unit in the disclosure are sampled in a bayer original image and are arranged in an array of 2 × 2 and correspond to 4 different types of single-color pixel units, that is, 4 sub-pixels constituting the reconstructed pixel unit, and in the bayer original image, no other sub-pixel exists between any two sub-pixels, so that the degree of compactness is optimal, and the display resolution capability of the reconstructed pixel unit is high. Therefore, the technical scheme of the disclosure can improve the display effect of the high-resolution image (the first mode image).
Fig. 4 is a flowchart illustrating a specific implementation of step S2 in this disclosure, and as shown in fig. 4, in some embodiments, step S2 includes:
step S201, extracting a plurality of repeating units from the bayer original image, where the repeating units include 16 sub-pixels arranged in an array of 4 × 4, and the sub-pixels at four corners of the repeating unit correspond to the same type of monochrome pixel unit.
In the related art, the 16 subpixels arranged in an array of 4 × 4 corresponding to the repeating unit are from 4 monochromatic pixel units in the bayer original image, while the 16 subpixels arranged in an array of 4 × 4 corresponding to the repeating unit in the embodiment of the present disclosure are from 9 monochromatic pixel units, and the subpixels at four corners of the repeating unit correspond to the same type of monochromatic pixel units.
Fig. 5 is a flowchart illustrating a specific implementation of step S201 in an embodiment of the present disclosure, as shown in fig. 5, in some embodiments, step S201 includes:
step S2011, a sliding sampling window is set in the bayer original image, the length and the width of the sliding sampling window both correspond to 4 sub-pixels, and the sub-pixels at four corners of the sliding sampling window correspond to the same type of monochrome pixel unit.
Step S2012, scanning the bayer original image with a sampling step size of 4 sub-pixels, and sliding the sampling window to acquire a data sample each time as a repeating unit.
Fig. 6 is a schematic diagram of extracting a repeating unit from a bayer original image according to an embodiment of the disclosure, and as shown in fig. 6, first, a sliding sampling window SF may be disposed in an upper left corner region of the bayer original image, and a length and a width of the sliding sampling window SF correspond to 4 sub-pixels, the sub-pixels at four corners of the sliding sampling window SF correspond to a same type of monochrome pixel unit, and 16 sub-pixels surrounded by the sliding sampling window SF are from 9 monochrome pixel units.
In some embodiments, referring to fig. 6, the subpixels at the four corners of the sliding sampling window SF are subpixels B from within the blue pixel cell Pix _ B; of course, when setting the initial position of the sliding sampling window SF, the sub-pixels at the four corners of the sliding sampling window SF may also be simultaneously the sub-pixel G from the first green pixel unit Pix _ G, or simultaneously the sub-pixel G 'from the second green pixel unit Pix _ G', or simultaneously the sub-pixel R from the red pixel unit Pix _ R, which are not shown in the corresponding figures. Then, the bayer raw image is scan-sampled in the row direction x and the column direction y with a step size of 4 sub-pixels, thereby obtaining a plurality of repeating units.
Step S202, for each repeating unit, dividing the repeating unit into 4 sampling pixel units arranged in an array of 2 × 2, where each sampling pixel unit includes 4 sub-pixels arranged in an array of 2 × 2, and taking the arrangement positions of the 4 sub-pixels in 1 sampling pixel unit as a reference standard, respectively adjusting the arrangement positions of the 4 sub-pixels included in each of the other 3 sampling pixel units.
Fig. 7 is a schematic diagram illustrating adjustment of sub-pixel positions in a repeating unit in the present disclosure, and as shown in fig. 7, in some embodiments, the arrangement position of sub-pixels in one sampling pixel unit SU1 located at the upper left corner in the bit repeating unit RU is used as a reference standard: the upper left corner is a blue sub-pixel B, the upper right corner is a second green sub-pixel G', the lower left corner is a red word pixel R, and the upper right corner is a first green sub-pixel G. The arrangement positions of the sub-pixels in the other three sampling pixel units SU2/SU3/SU4 in the repeating unit are adjusted based on the arrangement position reference standard, respectively.
Specifically, referring to fig. 7, for one sampling pixel unit SU2 located at the upper right corner in the repeating unit RU, the arrangement positions of four sub-pixels in the sampling pixel unit SU2 are interchanged left and right; for one sampling pixel unit SU3 located at the lower left corner in the repeating unit RU, the arrangement positions of the four sub-pixels in the sampling pixel unit SU3 are interchanged from top to bottom; for a sampling pixel unit located at the lower right corner in the repeating unit, the arrangement positions of the four sub-pixels in the sampling pixel unit SU4 are exchanged diagonally.
After the adjustment of the arrangement position of the sub-pixels in the repeating unit RU is completed, the repeating unit RU includes 4 complete pixels arranged in an array of 2 × 2.
Step S203 is to generate a first pattern image by recombination based on all the repeating units for which the sub-pixel position adjustment is completed, and the sampling pixel units included in the repeating units are used as reconstruction pixel units.
In step S203, all the repeating units are recombined into the first pattern image with reference to the arrangement position at the time of sampling.
Based on each complete pixel in the first mode image with high resolution obtained by the technical scheme disclosed by the invention, the 4 sub-pixels corresponding to the complete pixel are 4 sub-pixels which are compactly arranged in an array of 2 multiplied by 2 in the Bayer original image, and the display resolution capability of the complete pixel is higher, so that the display effect of the first mode image can be effectively improved.
It should be noted that, when the technical solution of the present disclosure is used to extract the repeating unit from the bayer raw image, since some rows or some columns of sub-pixels located at the edge of the bayer raw image are not scanned (extracted), the pixel resolution of the finally obtained first pattern image may be slightly lower than the pixel resolution of the high-resolution image in the related art. In general, when the bayer raw image includes 16MN sub-pixels arranged in an array of 4M × 4N, the pixel resolution of the high resolution image extracted in the related art is 2M × 2N, and the pixel resolution of the first mode image extracted in the technical solution of the present disclosure is (2M-2) × (2M-2).
In some embodiments, the data padding process may also be performed on the edge of the first mode image so that the pixel resolution of the first mode image becomes 2M × 2N. The filling processing area is at the edge and is small, so that the whole display effect of the first mode image cannot be influenced.
Fig. 8 is a flowchart of another image processing method provided by the embodiment of the disclosure, which includes, as shown in fig. 8, not only the above steps S1 and S2, but also steps S1a to S1c, and only steps S1a to S1c are described in detail below. Wherein step S1a is performed after step S1.
And 1a, calculating the overall image brightness of the Bayer original image.
Fig. 9 is a specific flowchart of step S1a in the embodiment of the present disclosure, and as shown in fig. 9, step S1a includes:
step S101a is to divide the bayer original image into a plurality of statistical regions arranged in an array X × Y, and obtain the overall luminance of the region of each statistical region.
In step S101a, a bayer raw image including 16MN subpixels arranged in an array of 4M × 4N is divided into a plurality of statistical regions arranged in an array of X × Y, each of which has the same shape and size, where X and Y are both positive integers.
The overall luminance of each statistical region can be obtained based on the exposure of the sub-pixels included in each statistical region (represented by the electrical signals corresponding to the sub-pixels). It should be noted that the technical means for calculating the overall brightness of the region according to the exposure amount is conventional in the art and will not be described in detail here.
Step S102a, performing weighted summation on the overall brightness of the regions in each statistical region to obtain the overall brightness of the bayer original image.
In step S102a, the image overall brightness of the bayer raw image is calculated based on the following equation:
where LUMA denotes the overall brightness of the image, wijIs the weight, luma, of the statistical region in the ith row and the jth columnijI is more than or equal to 1 and less than or equal to X, and j is more than or equal to 1 and less than or equal to Y.
In some embodiments, the overall image brightness of the bayer original image is calculated by averaging photometry, where the weight w of the statistical region located in the ith row and the jth columnijSatisfies the following conditions:
in some embodiments, the overall image brightness of the bayer original image is calculated according to a center-weighted metering mode, and the weights of the statistical regions may be distributed according to a two-dimensional gaussian function, where the weight w of the statistical region located in the ith row and the jth column is the weight w of the statistical regionijSatisfies the following conditions:
wherein σ is a preset constant.
And step S1b, judging whether the overall brightness of the image is greater than a preset brightness threshold value.
Comparing the overall brightness of the image calculated in the step S1a with a preset brightness threshold, when the overall brightness of the image is determined to be greater than the brightness threshold, the overall brightness of the bayer original image is brighter, the photographing environment corresponding to the bayer original image is brighter, and imaging in a high resolution mode may be performed, and then, performing a step S2; when the overall brightness of the image is determined to be less than or equal to the brightness threshold, the overall brightness of the bayer original image is relatively poor, the photographing environment corresponding to the bayer original image is relatively dark, and the step S1c may be performed after imaging in the low resolution mode.
Step S1c, performing data merging processing on the bayer original image by a four-in-one pixel merging technique to generate a second pattern image, where the second pattern image includes a plurality of merged pixel units arranged in an array, the merged pixel units in the second pattern image correspond to the monochrome pixel units in the bayer original image one to one, and exposure data of the merged pixel units is equal to the sum of exposure data of 4 sub-pixels included in the monochrome pixel unit corresponding to the merged pixel unit.
Fig. 10 is a schematic diagram of performing data merging processing on a bayer original image by using a four-in-one pixel merging technique in an embodiment of the present disclosure, and as shown in fig. 10, for each monochrome pixel unit Pix _ G/Pix _ R/Pix _ B/Pix _ G ' in the bayer original image, the exposure amounts of 4 sub-pixels included in the monochrome pixel unit Pix _ G/Pix _ R/Pix _ B/Pix _ G ' are summed, so as to obtain the exposure amount (characterized by a digital electrical signal) of the corresponding merged pixel unit Pix _ G/Pix _ R/Pix _ B/Pix _ G '.
When the bayer raw image includes 16MN sub-pixels arranged in an array of 4 mx 4N, the pixel resolution (four merged pixel units constitute one complete pixel) of the second pattern image obtained by step 1c is mxn.
In the embodiment, when the ambient light is bright, the imaging is performed in a high-resolution mode; when the ambient light is darker, the imaging is performed in a low-resolution mode, and the technical scheme disclosed by the invention can meet the use requirements of different environments.
An embodiment of the present disclosure further provides an image processing module, including: one or more processors, and memory storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the steps in the method of providing image processing as in the preceding embodiments.
The disclosed embodiments also provide a readable storage medium on which a program is stored, wherein the program, when executed, implements the steps in the image processing method as provided in the foregoing embodiments.
The disclosed embodiment also provides an image sensor, including: a photosensitive pixel array and a program module.
The photosensitive pixel array is used for sensing light and generating a corresponding Bayer original image; the photosensitive pixel array includes: the photosensitive unit groups comprise 4 heterogeneous monochromatic photosensitive units arranged in an array of 2 x 2, and the monochromatic photosensitive units comprise 4 photosensitive elements arranged in an array of 2 x 2.
In some embodiments, the photosensitive element may be a Charge-coupled Device (CCD) Device or a Complementary Metal Oxide Semiconductor (CMOS) Device
The program module stores a program, wherein the program realizes the steps in the image processing method as provided in the foregoing embodiments when executed by the processor.
Fig. 11 is a block diagram of an image capturing device according to an embodiment of the present disclosure, as shown in fig. 11, the image capturing device includes an image sensor according to the foregoing embodiment, and the image capturing device may be any device with a photographing or shooting function, such as a camera, a smart phone, and a tablet computer.
The Image acquisition equipment also comprises an Image Signal Processing (Image Signal Processing) module. The image sensor sends the generated Raw Data (Raw Data) corresponding to the first mode image or the second mode image to the image signal processing module, and the image signal processing module carries out denoising, gamma mapping, color control conversion and other processing on the received Raw Data.
It will be understood by those of ordinary skill in the art that all or some of the steps of the methods disclosed above, functional modules/units in the apparatus, may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted in a generic and descriptive sense only and not for purposes of limitation. In some instances, features, characteristics and/or elements described in connection with a particular embodiment may be used alone or in combination with features, characteristics and/or elements described in connection with other embodiments, unless expressly stated otherwise, as would be apparent to one skilled in the art. Accordingly, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure as set forth in the appended claims. .
Claims (11)
1. An image processing method comprising:
acquiring a Bayer raw image, wherein the Bayer raw image comprises a plurality of pixel unit groups arranged in an array, each pixel unit group comprises 4 heterogeneous single-color pixel units arranged in an array of 2 × 2, and each single-color pixel unit comprises 4 same sub-pixels arranged in an array of 2 × 2;
sampling and recombining the Bayer original image to generate a first mode image, wherein the first mode image comprises a plurality of reconstruction pixel units which are arranged in an array, each reconstruction pixel unit comprises 4 sub-pixels which are arranged in an array of 2 x 2, the 4 sub-pixels in each reconstruction pixel unit are sampled in the 4 sub-pixels which are arranged in the array of 2 x 2 in the Bayer original image and correspond to 4 different types of single-color pixel units, and the arrangement positions of the 4 sub-pixels in each reconstruction pixel unit in the 2 x 2 array are the same.
2. The method of claim 1, wherein the step of sampling the bayer raw image to rebin a first pattern image comprises:
extracting a plurality of repeating units from the Bayer original image, wherein the repeating units comprise 16 sub-pixels arranged in an array of 4 x 4, and the sub-pixels at four corners of each repeating unit correspond to single-color pixel units of the same type;
for each repeating unit, dividing the repeating unit into 4 sampling pixel units arranged in an array of 2 × 2, wherein each sampling pixel unit comprises 4 sub-pixels arranged in an array of 2 × 2, and respectively adjusting the arrangement positions of the 4 sub-pixels contained in the other 3 sampling pixel units by taking the arrangement positions of the 4 sub-pixels in 1 sampling pixel unit as a reference standard;
and generating the first mode image by recombination based on all the repeating units with the sub-pixel position adjustment, wherein sampling pixel units contained in the repeating units are used as reconstruction pixel units.
3. The method of claim 2, wherein the step of extracting a plurality of repeating units in the bayer raw image comprises:
setting a sliding sampling window in the Bayer original image, wherein the length and the width of the sliding sampling window correspond to 4 sub-pixels, and the sub-pixels at four corners of the sliding sampling window correspond to single-color pixel units of the same class;
and scanning the Bayer raw image by taking the sampling step size as 4 sub-pixels, wherein the data sample acquired by the sliding sampling window at each time is taken as the repeating unit.
4. The method of claim 1, wherein after the step of obtaining the bayer raw image and before the step of sampling and rebinning the bayer raw image to generate the first pattern image, further comprising:
calculating the overall image brightness of the Bayer original image;
judging whether the overall brightness of the image is greater than a preset brightness threshold value or not;
and when the integral brightness of the image is judged to be larger than the brightness threshold value, the step of sampling and recombining the Bayer original image to generate a first mode image is executed.
5. The method according to claim 4, wherein when it is determined that the overall luminance is less than or equal to the luminance threshold, a second pattern image is generated by performing data merging processing on the bayer raw image through a four-in-one pixel merging technique, the second pattern image includes a plurality of merged pixel units arranged in an array, the merged pixel units in the second pattern image correspond to the monochrome pixel units in the bayer raw image in a one-to-one manner, and exposure data of the merged pixel units is equal to a sum of exposure data of 4 sub-pixels included in the monochrome pixel unit corresponding to the merged pixel unit.
6. The method of claim 4, wherein the step of calculating the image bulk luminance of the Bayer raw image comprises:
dividing the Bayer original image into a plurality of statistical regions arranged in an array X multiplied by Y, and obtaining the overall brightness of the regions of the statistical regions, wherein X and Y are positive integers;
calculating the overall image brightness of the Bayer raw image according to the overall area brightness of each statistical area based on the following formula:
wherein LUMA represents the overall brightness of the image, wijIs the weight, luma, of the statistical region in the ith row and the jth columnijThe overall brightness of the region of the statistical region in the ith row and the jth column.
7. The method according to claim 6, wherein the weight w of the statistical region of the ith row and jth columnijSatisfies the following conditions:
or, the weight w of the statistical region in the ith row and the jth columnijSatisfies the following conditions:
wherein σ is a preset constant.
8. An image processing module comprising:
one or more processors;
a memory having one or more programs stored thereon;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the steps in the method of any of claims 1-7.
9. A readable storage medium having a program stored thereon, wherein the program when executed implements the steps in the method of any of claims 1-7.
10. An image sensor, comprising:
the photosensitive pixel array is used for sensing light and generating a corresponding Bayer original image; the photosensitive pixel array includes: the device comprises a plurality of photosensitive unit groups arranged in an array, wherein each photosensitive unit group comprises 4 different types of monochromatic photosensitive units arranged in an array of 2 x 2, and each monochromatic photosensitive unit comprises 4 photosensitive elements arranged in an array of 2 x 2;
program module storing a program, wherein the program realizes the steps in the method according to any one of claims 1-7 when executed by a processor.
11. An image acquisition apparatus comprising: an image sensor as claimed in claim 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911036821.XA CN112752009B (en) | 2019-10-29 | 2019-10-29 | Image processing method, module, readable storage medium, and image sensor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911036821.XA CN112752009B (en) | 2019-10-29 | 2019-10-29 | Image processing method, module, readable storage medium, and image sensor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112752009A true CN112752009A (en) | 2021-05-04 |
| CN112752009B CN112752009B (en) | 2023-06-09 |
Family
ID=75641589
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911036821.XA Active CN112752009B (en) | 2019-10-29 | 2019-10-29 | Image processing method, module, readable storage medium, and image sensor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112752009B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113888411A (en) * | 2021-09-29 | 2022-01-04 | 豪威科技(武汉)有限公司 | Resolution improving method and readable storage medium |
| CN117956300A (en) * | 2024-03-25 | 2024-04-30 | 上海元视芯智能科技有限公司 | Image processing architecture, image processing method and image processing chip |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101288296A (en) * | 2005-09-12 | 2008-10-15 | 赛普拉斯半导体公司 | Reconstructing a full color image from an image encoded by a bayer pattern |
| US20090290052A1 (en) * | 2008-05-23 | 2009-11-26 | Panavision Imaging, Llc | Color Pixel Pattern Scheme for High Dynamic Range Optical Sensor |
| CN102646262A (en) * | 2012-02-28 | 2012-08-22 | 西安交通大学 | Reconfigurable visual preprocessor and visual processing system |
| CN104253983A (en) * | 2008-12-08 | 2014-12-31 | 索尼株式会社 | Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus |
| CN104793341A (en) * | 2015-05-12 | 2015-07-22 | 京东方科技集团股份有限公司 | Display driving method and device |
| CN105611125A (en) * | 2015-12-18 | 2016-05-25 | 广东欧珀移动通信有限公司 | Imaging method, imaging device and electronic device |
| CN106488153A (en) * | 2015-08-26 | 2017-03-08 | 意法半导体国际有限公司 | Image Sensor Device With Macropixel Processing And Related Devices And Methods |
| CN109005392A (en) * | 2017-06-07 | 2018-12-14 | 联发科技股份有限公司 | A kind of demosaicing methods and its circuit |
-
2019
- 2019-10-29 CN CN201911036821.XA patent/CN112752009B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101288296A (en) * | 2005-09-12 | 2008-10-15 | 赛普拉斯半导体公司 | Reconstructing a full color image from an image encoded by a bayer pattern |
| US20090290052A1 (en) * | 2008-05-23 | 2009-11-26 | Panavision Imaging, Llc | Color Pixel Pattern Scheme for High Dynamic Range Optical Sensor |
| CN104253983A (en) * | 2008-12-08 | 2014-12-31 | 索尼株式会社 | Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus |
| CN102646262A (en) * | 2012-02-28 | 2012-08-22 | 西安交通大学 | Reconfigurable visual preprocessor and visual processing system |
| CN104793341A (en) * | 2015-05-12 | 2015-07-22 | 京东方科技集团股份有限公司 | Display driving method and device |
| CN106488153A (en) * | 2015-08-26 | 2017-03-08 | 意法半导体国际有限公司 | Image Sensor Device With Macropixel Processing And Related Devices And Methods |
| CN105611125A (en) * | 2015-12-18 | 2016-05-25 | 广东欧珀移动通信有限公司 | Imaging method, imaging device and electronic device |
| CN109005392A (en) * | 2017-06-07 | 2018-12-14 | 联发科技股份有限公司 | A kind of demosaicing methods and its circuit |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113888411A (en) * | 2021-09-29 | 2022-01-04 | 豪威科技(武汉)有限公司 | Resolution improving method and readable storage medium |
| CN117956300A (en) * | 2024-03-25 | 2024-04-30 | 上海元视芯智能科技有限公司 | Image processing architecture, image processing method and image processing chip |
| CN117956300B (en) * | 2024-03-25 | 2024-05-28 | 上海元视芯智能科技有限公司 | Image processing architecture, image processing method and image processing chip |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112752009B (en) | 2023-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8203633B2 (en) | Four-channel color filter array pattern | |
| JP4253655B2 (en) | Color interpolation method for digital camera | |
| Jiang et al. | Learning the image processing pipeline | |
| US20100302418A1 (en) | Four-channel color filter array interpolation | |
| CN102362501B (en) | Method for modifying color and panchromatic channel CFA image | |
| CN101889452B (en) | Image generation device and image generation method | |
| US20120188389A1 (en) | Method and apparatus for parallax correction in fused array imaging systems | |
| US9628725B2 (en) | Pixel array including pixel groups of long and short exposure pixels and image sensor including same | |
| CN103650487B (en) | Image pickup apparatus, image pickup element, and sensitivity difference correction method | |
| TWI547169B (en) | Image processing method and module | |
| CN103053163A (en) | Image generation device, and image generation system, method, and program | |
| CN111726549B (en) | Image sensor, electronic device, and chip | |
| KR20150029590A (en) | Method and associated apparatus for correcting color artifact of image | |
| US20190139189A1 (en) | Image remosaicing | |
| EP3247107B1 (en) | Method and device for obtaining a hdr image by graph signal processing | |
| CN104601970A (en) | Super-resolution in processing images such as from multi-layer sensors | |
| CN112752009B (en) | Image processing method, module, readable storage medium, and image sensor | |
| CN102713972A (en) | Pixel information reproduction using neural networks | |
| CN117392161B (en) | Calibration plate corner point for long-distance large perspective distortion and corner point number determination method | |
| CN113747066A (en) | Image correction method, image correction device, electronic equipment and computer-readable storage medium | |
| US20180365801A1 (en) | Method for processing signals from a matrix for taking colour images, and corresponding sensor | |
| US10887533B2 (en) | Infrared crosstalk compensation method and apparatus thereof | |
| US20130011050A1 (en) | Filter Setup Learning for Binary Sensor | |
| CN111050097B (en) | Infrared crosstalk compensation method and device | |
| JP4805294B2 (en) | Color binning to reduce the image resolution of digital images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |