CN112752009B - Image processing method, module, readable storage medium, and image sensor - Google Patents
Image processing method, module, readable storage medium, and image sensor Download PDFInfo
- Publication number
- CN112752009B CN112752009B CN201911036821.XA CN201911036821A CN112752009B CN 112752009 B CN112752009 B CN 112752009B CN 201911036821 A CN201911036821 A CN 201911036821A CN 112752009 B CN112752009 B CN 112752009B
- Authority
- CN
- China
- Prior art keywords
- image
- array
- sub
- pixels
- bayer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
Abstract
The present disclosure provides an image processing method, including: acquiring a Bayer original image, wherein the Bayer original image comprises a plurality of pixel unit groups which are arranged in an array, the pixel unit groups comprise 4 different types of single-color pixel units which are arranged in an array of 2 multiplied by 2, and the single-color pixel units comprise 4 identical sub-pixels which are arranged in the array of 2 multiplied by 2; sampling and recombining the Bayer original image to generate a first mode image, wherein the first mode image comprises a plurality of reconstructed pixel units which are arranged in an array, each reconstructed pixel unit comprises 4 sub-pixels which are arranged in an array of 2 multiplied by 2, the 4 sub-pixels in the reconstructed pixel units are sampled in the Bayer original image, are arranged in the array of 2 multiplied by 2 and correspond to the 4 sub-pixels of 4 monochromatic pixel units of different types, and the arrangement positions of the 4 sub-pixels in each reconstructed pixel unit in the 2 multiplied by 2 array are the same.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, a module, a readable storage medium, an image sensor, and an image acquisition device.
Background
To enhance the effect of photographing, image sensors employing a four-in-one pixel aggregation technique (also referred to as a four-in-one pixel merging technique) are now widely used. Specifically, four same-color small-size subpixels arranged in an array of 2×2 are combined to be used as one large-size single-color pixel unit.
For an image sensor adopting a Quad-pixel (Quad-layer) aggregation technology, manufacturers provide corresponding demosaic (demosaic) algorithms to sample and reorganize bayer raw images generated by a photosensitive pixel array when a user needs to acquire high-resolution images, so as to obtain a high-resolution image. However, in practical applications, it is found that the existing sampling method for sampling a Bayer (Bayer) raw image often adopts a uniformly distributed sampling manner, at this time, four sub-pixels forming a complete pixel are not compact enough in the Bayer raw image, and the display resolution of the complete pixel is low, so that the actual display effect of the high-resolution image is poor.
Disclosure of Invention
The present disclosure aims to solve at least one of the technical problems in the related art, and proposes an image processing method, a module, a readable storage medium, an image sensor, and an image acquisition apparatus
In a first aspect, an embodiment of the present disclosure provides an image processing method, including:
acquiring a Bayer original image, wherein the Bayer original image comprises a plurality of pixel unit groups which are arranged in an array, the pixel unit groups comprise 4 different types of single-color pixel units which are arranged in an array of 2 multiplied by 2, and the single-color pixel units comprise 4 identical sub-pixels which are arranged in the array of 2 multiplied by 2;
sampling and recombining the Bayer original image to generate a first mode image, wherein the first mode image comprises a plurality of reconstructed pixel units which are arranged in an array, each reconstructed pixel unit comprises 4 sub-pixels which are arranged in an array of 2 multiplied by 2, the 4 sub-pixels in the reconstructed pixel units are sampled in the Bayer original image, are arranged in the array of 2 multiplied by 2 and correspond to the 4 sub-pixels of 4 monochromatic pixel units of different types, and the arrangement positions of the 4 sub-pixels in each reconstructed pixel unit in the 2 multiplied by 2 array are the same.
In a second aspect, embodiments of the present disclosure further provide an image processing module, including:
one or more processors;
a memory having one or more programs stored thereon;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the steps in the methods as provided by the foregoing embodiments.
In a third aspect, the disclosed embodiments also provide a readable storage medium having stored thereon a program, wherein the steps of the method as provided by the previous embodiments are implemented when the program is executed.
In a fourth aspect, embodiments of the present disclosure further provide an image sensor, including:
the photosensitive pixel array is used for sensing light and generating a corresponding Bayer original image; the photosensitive pixel array includes: the device comprises a plurality of photosensitive unit groups which are arranged in an array, wherein each photosensitive unit group comprises 4 different types of monochromatic photosensitive units which are arranged in an array of 2 multiplied by 2, and each monochromatic photosensitive unit comprises 4 photosensitive elements which are arranged in an array of 2 multiplied by 2;
program modules storing a program, wherein the program when executed by a processor realizes steps in a method as provided by the previous embodiments.
In a fifth aspect, embodiments of the present disclosure further provide an image capturing apparatus, including: the image sensor provided by the foregoing embodiment.
The present disclosure has the following beneficial effects:
the embodiment of the disclosure provides an image processing method, which comprises the following steps: acquiring a Bayer original image; sampling and recombining the Bayer original image to generate a first mode image, wherein the first mode image comprises a plurality of reconstructed pixel units which are arranged in an array, the reconstructed pixel units comprise 4 sub-pixels which are arranged in an array of 2 multiplied by 2, the 4 sub-pixels in the reconstructed pixel units are sampled in the Bayer original image to be arranged in the array of 2 multiplied by 2 and correspond to the 4 sub-pixels of 4 monochromatic pixel units of different types, and the arrangement positions of the 4 sub-pixels in each reconstructed pixel unit in the 2 multiplied by 2 array are the same. In the present disclosure, 4 sub-pixels forming a reconstructed pixel unit, no other sub-pixel exists between any two sub-pixels in a bayer original image, so that the compactness degree is optimal, and therefore, the technical scheme of the present disclosure can improve the display effect of a high resolution image.
Drawings
FIG. 1 is a schematic layout diagram of a photosensitive pixel array using a four-in-one pixel Bayer array in the related art;
FIG. 2 is a schematic diagram of a Bayer original image uniformly sampled and recombined to obtain a resolution image in the related art;
fig. 3 is a flowchart of an image processing method according to an embodiment of the present disclosure;
FIG. 4 is a flowchart showing a specific implementation of step S2 in an embodiment of the disclosure;
FIG. 5 is a flowchart showing a specific implementation of step S201 in an embodiment of the disclosure;
FIG. 6 is a schematic diagram of the extraction of repeat units from a Bayer raw image in an embodiment of the disclosure;
FIG. 7 is a schematic diagram of adjusting subpixel positions within a repeating unit in the present disclosure;
FIG. 8 is a flowchart of another image processing method provided by an embodiment of the present disclosure;
FIG. 9 is a specific flowchart of step S1a in an embodiment of the present disclosure;
FIG. 10 is a schematic diagram of a data merging process for Bayer raw images using a four-in-one pixel merging technique in an embodiment of the disclosure;
fig. 11 is a block diagram of an image capturing apparatus according to an embodiment of the present disclosure.
Detailed Description
In order to better understand the technical solutions of the present disclosure, the following describes in detail an image processing method, a module, a readable storage medium, an image sensor and an image acquisition device provided in the present disclosure with reference to the accompanying drawings.
For better understanding of the technical solutions of the present disclosure, the data monitoring method, the task monitoring method, the data monitoring module, the task monitoring system and the computer readable medium provided in the present disclosure are described in detail below with reference to the accompanying drawings.
Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Accordingly, a first element, component, or module discussed below could be termed a second element, component, or module without departing from the teachings of the present disclosure.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The technical scheme of the disclosure is based on a four-in-one pixel aggregation technology; specifically, the photosensitive pixel array for sensing light and generating a corresponding bayer primary image is a four-in-one pixel bayer array. Fig. 1 is a schematic layout diagram of a photosensitive pixel array using a four-in-one pixel bayer array in the related art, as shown in fig. 1, the photosensitive pixel array includes: the light-sensitive unit group comprises 4 different types of single-color light-sensitive units which are arranged in an array manner, wherein each single-color light-sensitive unit comprises 4 light-sensitive elements which are arranged in an array manner and used for sensing light rays with the same color, and the light-sensitive elements are arranged in an array manner, namely 2X 2.
In the present disclosure, the 4 different types of monochromatic light sensing units are exemplified as a first green light sensing unit senspix_g, a red light sensing unit senspix_r, a blue light sensing unit senspix_b, and a second green light sensing unit senspix_g', but they do not limit the technical solution of the present disclosure. In the present disclosure, the specific photosensitive colors of the 4 different types of monochromatic photosensitive units can also be designed and adjusted as required. Taking the first green light sensing units as an example, each first green light sensing unit sensepix_g includes 4 light sensing elements that sense green light arranged in an array of 2×2. Each photosensitive element in the photosensitive pixel array can independently sense light and generate corresponding electric signals based on the received exposure, wherein the electric signals can represent the exposure at the photosensitive element and the brightness corresponding to the light received by the photosensitive element.
Suppose that the photosensitive pixel array includes MN photosensitive cell groups arranged in M rows and N columns of the array, and the photosensitive pixel array includes 4M x 4N photosensitive elements arranged in 4M rows and 4N columns of the array. The bayer primary image generated by the photosensitive pixel array includes 16MN subpixels arranged in an array 4 mx 4N with the data sensed by the photosensitive element as subpixels.
FIG. 2 is a schematic diagram of a Bayer original image uniformly sampled and recombined to obtain a resolution image in the related art, as shown in FIG. 2, in the related art, 16 sub-pixels contained in 4 different types of single-color pixel units Pix_G/Pix_R/Pix_B/Pix_G 'arranged in an array of 2×2 are used as a repeating unit RU, and one sub-pixel G positioned at the upper left corner is extracted from the 4 different types of single-color pixel units Pix_G/Pix_R/Pix_B/Pix_G', respectively 11 /R 11 /B 11 /G’ 11 Then the extracted 4 sub-pixels G 11 /R 11 /B 11 /G’ 11 Recombined into a reconstructed pixel unit RB. Then, 4 sub-pixels G positioned at the upper right corner in the 4 different types of single-color pixel units Pix_G/Pix_R/Pix_B/Pix_G' are recombined based on the same sampling mode 12 /R 12 /B 12 /G’ 12 Recombined into a reconstructed pixel unit RB, and 4 sub-pixels G positioned at the lower left corner in the 4 different types of monochromatic pixel units Pix_G/Pix_R/Pix_B/Pix_G 21 /R 21 /B 21 /G’ 21 Recombined into a reconstructed pixel unit RB, and 4 sub-pixels G positioned at the lower right corner in the 4 different types of monochromatic pixel units Pix_G/Pix_R/Pix_B/Pix_G 22 /R 22 /B 22 /G’ 22 And recombined into one reconstructed pixel unit RB, so as to obtain 4 reconstructed pixel units.
And carrying out the same adoption and recombination treatment on all the repeated units in the Bayer original image, and finally obtaining the high-resolution image with the pixel resolution (one reconstructed pixel unit forms a complete display pixel) of 2M multiplied by 2N.
However, in practical applications, it is found that, in the related technical solutions, the 4 sub-pixels forming one reconstructed pixel unit are not compact enough in positions in the bayer primary image, and the display resolution of the reconstructed pixel unit is low, so that the actual display effect of the high-resolution image is poor. Taking the reconstructed pixel unit RB in fig. 2 as an example, the 4 sub-pixels constituting the reconstructed pixel unit RB are respectively: first green sub-pixel G 11 Red subpixel R 11 Blue sub-pixel B 11 A second green sub-pixel G' 11 The method comprises the steps of carrying out a first treatment on the surface of the In the bayer primary image, for the first green subpixel G 11 Red subpixel R 11 Blue sub-pixel B 11 A second green sub-pixel G' 11 Between any two sub-pixels there is one other sub-pixel (e.g. the first green sub-pixel G 11 And red sub-pixel R 11 With a first green sub-pixel G in between 12 ) I.e. the 4 sub-pixels G 11 /R 11 /B 11 /G’ 11 The positions in the bayer primary image are not compact enough, and the display resolution of the complete pixels formed after recombination is low.
To solve the above-described technical problems, the present disclosure provides an image processing method that can obtain a high-resolution image with a better display effect based on a bayer raw image.
Fig. 3 is a flowchart of an image processing method according to an embodiment of the present disclosure, as shown in fig. 3, including:
step S1, acquiring Bayer original images.
In step S1, a bayer primary image is acquired from a photosensitive pixel array employing a four-in-one pixel bayer array; the Bayer original image comprises a plurality of pixel unit groups which are arranged in an array, wherein each pixel unit group comprises 4 different types of single-color pixel units which are arranged in an array of 2 multiplied by 2, and each single-color pixel unit comprises 4 identical sub-pixels which are arranged in an array of 2 multiplied by 2.
In this disclosure, "different types" refers to different positions in the 2×2 array corresponding to the pixel unit group. In addition, the colors corresponding to the two types of monochromatic pixel units belonging to different types can be the same or different. In the embodiments of the present disclosure, an example is described in which 4 different types of monochrome photosensitive units in a photosensitive pixel array are a first green photosensitive unit, a red photosensitive unit, a blue photosensitive unit, and a second green photosensitive unit, respectively, and each pixel unit includes four sub-pixels of corresponding colors.
It should be noted that, the case that the 4 different types of monochrome photosensitive units in the present disclosure adopt the green, red, blue, and green modes (also referred to as GRBG modes) is only used for exemplary description, which does not limit the technical solution of the present disclosure. In the present disclosure, the 4 different types of monochromatic photosensitive cells may also use other modes, such as red, green, blue, and white modes (also referred to as RGBW modes), and red, yellow, and blue modes (also referred to as RYYB modes), which are not exemplified herein.
S2, sampling and recombining Bayer original images to generate a first mode image; the first mode image comprises a plurality of reconstructed pixel units which are arranged in an array, the reconstructed pixel units comprise 4 sub-pixels which are arranged in an array of 2 multiplied by 2, the 4 sub-pixels in the reconstructed pixel units are sampled in the Bayer original image and are arranged in the array of 2 multiplied by 2 and correspond to the 4 sub-pixels of 4 monochromatic pixel units of different types, and the arrangement positions of the 4 sub-pixels in each reconstructed pixel unit in the 2 multiplied by 2 array are the same.
Different from the related art, in the present disclosure, the 4 sub-pixels included in each reconstructed pixel unit are sampled in the bayer raw image and are arranged in an array of 2×2 and correspond to the 4 sub-pixels of the 4 monochromatic pixel units of different types, that is, the 4 sub-pixels forming the reconstructed pixel unit, no other sub-pixel exists between any two sub-pixels in the bayer raw image, so that the compactness degree is optimal, and the display resolution capability of the reconstructed pixel unit is higher. Therefore, the technical scheme of the present disclosure can improve the display effect of the high resolution image (first mode image).
Fig. 4 is a flowchart showing a specific implementation of step S2 in the embodiment of the disclosure, as shown in fig. 4, in some embodiments, step S2 includes:
step S201, extracting a plurality of repeating units in the bayer raw image, where the repeating units include 16 sub-pixels arranged in an array of 4×4, and the sub-pixels at four corners of the repeating units correspond to the same type of monochrome pixel units.
The 16 sub-pixels in the array 4×4 arrangement corresponding to the repeating units in the related art are from 4 single-color pixel units in the bayer original image, while the 16 sub-pixels in the array 4×4 arrangement corresponding to the repeating units in the embodiment of the disclosure are from 9 single-color pixel units, and the sub-pixels at the four corners of the repeating units correspond to the single-color pixel units of the same class.
Fig. 5 is a flowchart showing a specific implementation of step S201 in the embodiment of the disclosure, as shown in fig. 5, in some embodiments, step S201 includes:
in step S2011, a sliding sampling window is set in the bayer original image, the length and the width of the sliding sampling window both correspond to 4 sub-pixels, and the sub-pixels at the four corners of the sliding sampling window correspond to the same type of monochrome pixel units.
In step S2012, the bayer primary image is scanned with a sampling step of 4 sub-pixels, and the data samples collected by the sliding sampling window each time are used as a repeating unit.
Fig. 6 is a schematic diagram of extracting a repeating unit from a bayer raw image in the embodiment of the disclosure, as shown in fig. 6, first, a sliding sampling window SF may be placed in an upper left corner area of the bayer raw image, and the length and width of the sliding sampling window SF correspond to 4 sub-pixels, the sub-pixels at four corners of the sliding sampling window SF correspond to the same type of monochrome pixel units, and 16 sub-pixels enclosed by the sliding sampling window SF are from 9 monochrome pixel units.
In some embodiments, see fig. 6, the subpixels at the four corners of the sliding sampling window SF are subpixels B from within the blue pixel cell pix_b; of course, in setting the initial position of the sliding sampling window SF, the sub-pixels at the four corners of the sliding sampling window SF may also be the sub-pixels G from the first green pixel unit pix_g at the same time, or the sub-pixels G 'from the second green pixel unit pix_g' at the same time, or the sub-pixels R from the red pixel unit pix_r at the same time, which three cases do not give the corresponding drawings. Then, the bayer raw image is scanned and sampled in the row direction x and the column direction y with a step size of 4 sub-pixels, thereby obtaining a plurality of repeating units.
Step S202, for each repeating unit, dividing the repeating unit into 4 sampling pixel units arranged in an array of 2×2, each sampling pixel unit includes 4 sub-pixels arranged in an array of 2×2, and respectively adjusting the arrangement positions of the 4 sub-pixels included in each of the other 3 sampling pixel units by using the arrangement positions of the 4 sub-pixels in the 1 sampling pixel unit as a reference standard.
Fig. 7 is a schematic diagram of adjusting the positions of sub-pixels in the repeating unit in the present disclosure, as shown in fig. 7, and in some embodiments, the arrangement positions of sub-pixels in the sampling pixel unit SU1 located in the upper left corner in the bit repeating unit RU are used as reference criteria: the upper left corner is the blue sub-pixel B, the upper right corner is the second green sub-pixel G', the lower left corner is the red word pixel R, and the upper right corner is the first green sub-pixel G. And respectively adjusting the arrangement positions of the sub-pixels in the other three sampling pixel units SU2/SU3/SU4 in the repeating unit based on the arrangement position reference standard.
Specifically, referring to fig. 7, for one sampling pixel unit SU2 located in the upper right corner in the repeating unit RU, the arrangement positions of four sub-pixels in the sampling pixel unit SU2 are interchanged left and right; for one sampling pixel unit SU3 positioned at the lower left corner in the repeating unit RU, the arrangement positions of four sub-pixels in the sampling pixel unit SU3 are interchanged up and down; for one sampling pixel unit positioned at the lower right corner in the repeating unit, the arrangement positions of four sub-pixels in the sampling pixel unit SU4 are exchanged diagonally.
After the arrangement position of the sub-pixels in the repeating unit RU is adjusted, the repeating unit contains 4 complete pixels arranged in an array of 2×2.
Step S203, based on all the repeated units with the sub-pixel position adjusted, the first mode image is generated by recombination, and the sampling pixel units contained in the repeated units are used as reconstructed pixel units.
In step S203, all the repeated units are reorganized into the first pattern image with reference to the arrangement position at the time of sampling.
According to the high-resolution first mode image obtained based on the technical scheme, each complete pixel in the high-resolution first mode image is composed of 4 sub-pixels which are arranged in a 2 multiplied by 2 compact mode in the Bayer original image, and the display resolution of the complete pixels is high, so that the display effect of the first mode image can be effectively improved.
It should be noted that, when the technical solution of the present disclosure is adopted to extract the repeating unit from the bayer primary image, since some rows or columns of sub-pixels located at the edge of the bayer primary image are not scanned (extracted), the pixel resolution of the finally obtained first mode image may be slightly lower than that of the high resolution image in the related art. In general, when the bayer primary image includes 16MN subpixels arranged in an array of 4m×4n, the pixel resolution of the high-resolution image extracted in the related art is 2m×2n, and the pixel resolution of the first mode image extracted in the technical scheme of the present disclosure is (2M-2) × (2M-2).
In some embodiments, the data padding process may be further performed on the edges of the first mode image such that the pixel resolution of the first mode image becomes 2m×2n. Since the area of the padding process is at the edge and the area is small, the overall display effect of the first mode image is not affected.
Fig. 8 is a flowchart of another image processing method according to an embodiment of the present disclosure, as shown in fig. 8, which includes not only the step S1 and the step S2, but also the steps S1a to S1c, and only the steps S1a to S1c will be described in detail below. Wherein step S1a is performed after step S1.
Step 1a, calculating the overall brightness of the Bayer original image.
Fig. 9 is a specific flowchart of step S1a in the embodiment of the disclosure, where, as shown in fig. 9, step S1a includes:
step S101a, dividing the bayer raw image into a plurality of statistical regions arranged in an array x×y, and acquiring the regional overall brightness of each statistical region.
In step S101a, a bayer original image including 16MN subpixels arranged in an array 4m×4n is divided into a plurality of statistical regions arranged in an array x×y, each of which has the same shape and size, wherein X and Y are both positive integers.
Based on the exposure of the sub-pixels included in each statistical region (characterized by the electrical signals corresponding to the sub-pixels), the overall brightness of the region of each statistical region can be obtained. The technical means for calculating the overall brightness of the region according to the exposure amount is a conventional technique in the art, and will not be described in detail here.
Step S102a, the overall brightness of the areas in each statistical area is weighted and summed to obtain the overall brightness of the Bayer original image.
In step S102a, the image overall brightness of the bayer original image is calculated based on the following equation:
wherein LUMA represents the overall brightness of the image, w ij Weighting the statistical region of row i and column j, luma ij The overall brightness of the region of the statistical region of the ith row and jth column is 1.ltoreq.i.ltoreq.X, 1.ltoreq.j.ltoreq.Y.
In some embodiments, the average photometry is used to calculate the overall brightness of the Bayer primary image, and the weight w of the statistical region in the ith row and the jth column ij The method meets the following conditions:
in some embodiments, the image of the bayer raw image is calculated as a center-heavy-spot photometryThe weight of the statistical region can be distributed according to a two-dimensional Gaussian function, and the weight w of the statistical region positioned in the ith row and the jth column ij The method meets the following conditions:
wherein σ is a preset constant.
And S1b, judging whether the overall brightness of the image is larger than a preset brightness threshold value.
Comparing the overall brightness of the image calculated in the step S1a with a preset brightness threshold, when the overall brightness of the image is judged to be larger than the brightness threshold, the overall brightness of the Bayer original image is brighter, the photographing environment corresponding to the Bayer original image is brighter, imaging can be performed in a high resolution mode, and then the step S2 is executed; when it is determined that the overall brightness of the image is less than or equal to the brightness threshold, it indicates that the overall brightness of the bayer primary image is relatively dark, and the bayer primary image corresponds to a photographing environment, and step S1c may be executed after imaging in the low resolution mode.
Step S1c, carrying out data merging processing on the Bayer original image through a four-in-one pixel merging technology to generate a second mode image, wherein the second mode image comprises a plurality of merging pixel units which are arranged in an array, the merging pixel units in the second mode image are in one-to-one correspondence with the monochromatic pixel units in the Bayer original image, and the exposure data of the merging pixel units is equal to the sum of the exposure data of 4 subpixels contained in the monochromatic pixel units corresponding to the merging pixel units.
Fig. 10 is a schematic diagram of performing data merging processing on a bayer raw image using a four-in-one pixel merging technique in the embodiment of the present disclosure, as shown in fig. 10, for each single-color pixel unit pix_g/pix_r/pix_b/pix_g ' in the bayer raw image, the exposure amounts of the 4 sub-pixels included in the single-color pixel unit pix_g/pix_r/pix_b/pix_g ' are summed, so as to obtain the exposure amounts (represented by digital electrical signals) of the corresponding merged pixel unit pix_g/pix_r/pix_b/pix_g '.
When the bayer raw image includes 16MN sub-pixels arranged in an array of 4 mx 4N, the pixel resolution of the second mode image obtained by step 1c (four merged pixel units constitute one complete pixel) is mxn.
In this embodiment, when the ambient light is brighter, then the imaging is performed in a high resolution mode; when the ambient light is darker, then adopt the low resolution mode to image, the technical scheme of this disclosure can be satisfied the user demand of different environment.
The embodiment of the disclosure also provides an image processing module, which comprises: the image processing apparatus includes one or more processors and a memory storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the steps in providing an image processing method according to the foregoing embodiments.
The present disclosure also provides a readable storage medium having a program stored thereon, wherein the steps in the image processing method provided by the foregoing embodiments are implemented when the program is executed.
The embodiment of the disclosure also provides an image sensor, including: a photosensitive pixel array and a program module.
The light-sensitive pixel array is used for sensing light and generating a corresponding Bayer original image; the photosensitive pixel array includes: the photosensitive unit group comprises 4 different types of monochromatic photosensitive units which are arranged in an array of 2 multiplied by 2, and the monochromatic photosensitive units comprise 4 photosensitive elements which are arranged in an array of 2 multiplied by 2.
In some embodiments, the photosensitive element may be a Charge-coupled Device (CCD) Device or a complementary metal oxide semiconductor (Complementary Metal Oxide Semiconductor, CMOS) Device
The program modules store a program wherein the steps of the method for providing an image processing according to the previous embodiments are implemented when the program is executed by a processor.
Fig. 11 is a block diagram of an image capturing device according to an embodiment of the present disclosure, and as shown in fig. 11, the image capturing device includes an image sensor provided in the foregoing embodiment, and the image capturing device may be any device having a photographing or image capturing function, such as a camera, a video camera, a smart phone, a tablet computer, or the like.
The image acquisition device also comprises an image signal processing (Image Signal Processing) module. The image sensor sends the generated original Data (Raw Data) corresponding to the first mode image or the second mode image to an image signal processing module, and the image signal processing module performs denoising, gamma mapping, color control conversion and other processes on the received original Data.
Those of ordinary skill in the art will appreciate that all or some of the steps of the methods, functional modules/units in the apparatus disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted in a generic and descriptive sense only and not for purpose of limitation. In some instances, it will be apparent to one skilled in the art that features, characteristics, and/or elements described in connection with a particular embodiment may be used alone or in combination with other embodiments unless explicitly stated otherwise. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure as set forth in the appended claims. .
Claims (10)
1. An image processing method, comprising:
acquiring a Bayer original image, wherein the Bayer original image comprises a plurality of pixel unit groups which are arranged in an array, the pixel unit groups comprise 4 different types of single-color pixel units which are arranged in an array of 2 multiplied by 2, and the single-color pixel units comprise 4 identical sub-pixels which are arranged in the array of 2 multiplied by 2;
sampling and recombining the Bayer original image to generate a first mode image, wherein the first mode image comprises a plurality of reconstructed pixel units which are arranged in an array, the reconstructed pixel units comprise 4 sub-pixels which are arranged in an array of 2 multiplied by 2, the 4 sub-pixels in the reconstructed pixel units are sampled in the Bayer original image and are arranged in the array of 2 multiplied by 2 and correspond to the 4 sub-pixels of 4 monochromatic pixel units of different types, and the arrangement positions of the 4 sub-pixels in each reconstructed pixel unit in the 2 multiplied by 2 array are the same;
the step of sampling and recombining the bayer primary image to generate a first mode image includes:
extracting a plurality of repeated units from the Bayer original image, wherein the repeated units comprise 16 sub-pixels which are arranged in an array of 4 multiplied by 4, and the sub-pixels at four corners of the repeated units correspond to monochromatic pixel units of the same class;
for each repeating unit, dividing the repeating unit into 4 sampling pixel units which are arranged in an array of 2 multiplied by 2, wherein each sampling pixel unit comprises 4 sub-pixels which are arranged in the array of 2 multiplied by 2, and respectively adjusting the arrangement positions of the 4 sub-pixels contained in each of the other 3 sampling pixel units by taking the arrangement positions of the 4 sub-pixels in the 1 sampling pixel units as a reference standard;
and recombining and generating the first mode image based on all the repeated units with the sub-pixel position adjusted, wherein the sampling pixel units contained in the repeated units are used as reconstruction pixel units.
2. The method of claim 1, wherein the extracting of the plurality of repeat units in the bayer raw image comprises:
setting a sliding sampling window in the Bayer original image, wherein the length and the width of the sliding sampling window correspond to 4 sub-pixels, and the sub-pixels at the four corners of the sliding sampling window correspond to monochromatic pixel units of the same class;
and scanning the Bayer original image by using 4 sub-pixels with sampling step length, wherein the data sample acquired by the sliding sampling window each time is used as the repeating unit.
3. The method of claim 1, wherein after the step of acquiring the bayer raw image and before the step of sampling, recombining the bayer raw image to generate a first pattern image, further comprises:
calculating the overall brightness of the Bayer original image;
judging whether the overall brightness of the image is larger than a preset brightness threshold value or not;
and when the integral brightness of the image is larger than the brightness threshold value, executing the steps of sampling and recombining the Bayer original image to generate a first mode image.
4. The method of claim 3, wherein when the overall brightness is determined to be less than or equal to the brightness threshold, performing data merging processing on the bayer primary image by a four-in-one pixel merging technology to generate a second mode image, where the second mode image includes a plurality of merged pixel units arranged in an array, the merged pixel units in the second mode image are in one-to-one correspondence with the monochrome pixel units in the bayer primary image, and exposure data of the merged pixel units is equal to a sum of exposure data of 4 subpixels included in the monochrome pixel unit corresponding to the merged pixel units.
5. A method according to claim 3, wherein the step of calculating the overall image brightness of the bayer raw image comprises:
dividing the Bayer original image into a plurality of statistical areas which are arranged in an array X multiplied by Y, and obtaining the overall brightness of the areas of each statistical area, wherein X and Y are positive integers;
calculating the image overall brightness of the bayer raw image based on the following equation according to the area overall brightness of each statistical area:
wherein LUMA represents the overall brightness of the image, w ij Weighting the statistical region of row i and column j, luma ij The overall brightness of the region is the statistical region of the ith row and jth column.
6. The method according to claim 5, wherein the weight w of the statistical region of the ith row and jth column ij The method meets the following conditions:
alternatively, the weight of the statistical region of the ith row and jth columnValue w ij The method meets the following conditions:
wherein σ is a preset constant.
7. An image processing module, comprising:
one or more processors;
a memory having one or more programs stored thereon;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the steps in the method of any of claims 1-6.
8. A readable storage medium having stored thereon a program, wherein the steps of the method according to any of claims 1-6 are implemented when said program is executed.
9. An image sensor, comprising:
the photosensitive pixel array is used for sensing light and generating a corresponding Bayer original image; the photosensitive pixel array includes: the device comprises a plurality of photosensitive unit groups which are arranged in an array, wherein each photosensitive unit group comprises 4 different types of monochromatic photosensitive units which are arranged in an array of 2 multiplied by 2, and each monochromatic photosensitive unit comprises 4 photosensitive elements which are arranged in an array of 2 multiplied by 2;
program module storing a program, wherein the program when executed by a processor realizes the steps in the method according to any of claims 1-6.
10. An image acquisition apparatus comprising: an image sensor as claimed in claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911036821.XA CN112752009B (en) | 2019-10-29 | 2019-10-29 | Image processing method, module, readable storage medium, and image sensor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911036821.XA CN112752009B (en) | 2019-10-29 | 2019-10-29 | Image processing method, module, readable storage medium, and image sensor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112752009A CN112752009A (en) | 2021-05-04 |
| CN112752009B true CN112752009B (en) | 2023-06-09 |
Family
ID=75641589
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911036821.XA Active CN112752009B (en) | 2019-10-29 | 2019-10-29 | Image processing method, module, readable storage medium, and image sensor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112752009B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101288296A (en) * | 2005-09-12 | 2008-10-15 | 赛普拉斯半导体公司 | Reconstructing a full color image from an image encoded by a bayer pattern |
| CN102646262A (en) * | 2012-02-28 | 2012-08-22 | 西安交通大学 | Reconfigurable visual preprocessor and visual processing system |
| CN104253983A (en) * | 2008-12-08 | 2014-12-31 | 索尼株式会社 | Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus |
| CN104793341A (en) * | 2015-05-12 | 2015-07-22 | 京东方科技集团股份有限公司 | Display driving method and device |
| CN105611125A (en) * | 2015-12-18 | 2016-05-25 | 广东欧珀移动通信有限公司 | Imaging method, imaging device and electronic device |
| CN106488153A (en) * | 2015-08-26 | 2017-03-08 | 意法半导体国际有限公司 | There is image sensor apparatus and relevant apparatus and the method for grand processes pixel |
| CN109005392A (en) * | 2017-06-07 | 2018-12-14 | 联发科技股份有限公司 | A kind of demosaicing methods and its circuit |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090290052A1 (en) * | 2008-05-23 | 2009-11-26 | Panavision Imaging, Llc | Color Pixel Pattern Scheme for High Dynamic Range Optical Sensor |
-
2019
- 2019-10-29 CN CN201911036821.XA patent/CN112752009B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101288296A (en) * | 2005-09-12 | 2008-10-15 | 赛普拉斯半导体公司 | Reconstructing a full color image from an image encoded by a bayer pattern |
| CN104253983A (en) * | 2008-12-08 | 2014-12-31 | 索尼株式会社 | Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus |
| CN102646262A (en) * | 2012-02-28 | 2012-08-22 | 西安交通大学 | Reconfigurable visual preprocessor and visual processing system |
| CN104793341A (en) * | 2015-05-12 | 2015-07-22 | 京东方科技集团股份有限公司 | Display driving method and device |
| CN106488153A (en) * | 2015-08-26 | 2017-03-08 | 意法半导体国际有限公司 | There is image sensor apparatus and relevant apparatus and the method for grand processes pixel |
| CN105611125A (en) * | 2015-12-18 | 2016-05-25 | 广东欧珀移动通信有限公司 | Imaging method, imaging device and electronic device |
| CN109005392A (en) * | 2017-06-07 | 2018-12-14 | 联发科技股份有限公司 | A kind of demosaicing methods and its circuit |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112752009A (en) | 2021-05-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8237831B2 (en) | Four-channel color filter array interpolation | |
| US8203633B2 (en) | Four-channel color filter array pattern | |
| US8581995B2 (en) | Method and apparatus for parallax correction in fused array imaging systems | |
| US8253832B2 (en) | Interpolation for four-channel color filter array | |
| JP4253655B2 (en) | Color interpolation method for digital camera | |
| US10992878B2 (en) | Method of obtaining wide dynamic range image and image pickup device performing the same | |
| US9092881B2 (en) | Color interpolation apparatus | |
| TWI547169B (en) | Image processing method and module | |
| US10366478B2 (en) | Method and device for obtaining a HDR image by graph signal processing | |
| CN113170061B (en) | Image sensor, imaging device, electronic apparatus, image processing system, and signal processing method | |
| US8379977B2 (en) | Method for removing color fringe in digital image | |
| US20190139189A1 (en) | Image remosaicing | |
| CN113573030B (en) | Image generation method, device, electronic equipment and computer readable storage medium | |
| US20110032269A1 (en) | Automatically Resizing Demosaicked Full-Color Images Using Edge-Orientation Maps Formed In The Demosaicking Process | |
| WO2021174529A1 (en) | Image sensor, imaging apparatus, electronic device, image processing system and signal processing method | |
| US20100328504A1 (en) | Adaptive method for processing digital images, and an image processing device | |
| CN112752009B (en) | Image processing method, module, readable storage medium, and image sensor | |
| CN207251823U (en) | Imaging device and imaging system | |
| US9210386B2 (en) | Filter setup learning for binary sensor | |
| CN111050097B (en) | Infrared crosstalk compensation method and device | |
| US10887533B2 (en) | Infrared crosstalk compensation method and apparatus thereof | |
| WO2011076976A1 (en) | Determining color information using a binary sensor | |
| US20230066267A1 (en) | Image acquisition apparatus including a plurality of image sensors, and electronic apparatus including the image acquisition apparatus | |
| CN114882129A (en) | Image processing method, image processing device and chip | |
| CN117115593A (en) | Model training method, image processing method and device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |