WO2022199236A1 - Raw图像的处理方法、芯片和电子设备 - Google Patents

Raw图像的处理方法、芯片和电子设备 Download PDF

Info

Publication number
WO2022199236A1
WO2022199236A1 PCT/CN2022/072485 CN2022072485W WO2022199236A1 WO 2022199236 A1 WO2022199236 A1 WO 2022199236A1 CN 2022072485 W CN2022072485 W CN 2022072485W WO 2022199236 A1 WO2022199236 A1 WO 2022199236A1
Authority
WO
WIPO (PCT)
Prior art keywords
area
image
processing
block
raw image
Prior art date
Application number
PCT/CN2022/072485
Other languages
English (en)
French (fr)
Inventor
朱文波
Original Assignee
哲库科技(上海)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 哲库科技(上海)有限公司 filed Critical 哲库科技(上海)有限公司
Publication of WO2022199236A1 publication Critical patent/WO2022199236A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present disclosure relates to the technical field of image processing, and in particular, to a RAW image processing method, chip and electronic device.
  • LSC Lins Shading Correction
  • the purpose of the present disclosure is to provide a RAW image processing method, chip and electronic device.
  • an embodiment of the first aspect of the present disclosure proposes a method for processing a RAW image, including: identifying a first area and a second area of a single-frame RAW image; performing block processing on the first area and the second area , wherein the block processing of the first area and the second area is based on different block processing strategies; according to the block processing results, LUT calculations are performed on the first area and the second area respectively to obtain the LUT results; The first area and the second area are subjected to LSC processing to obtain the current frame image.
  • identifying the first area and the second area of the single-frame RAW image includes: performing area division on the single-frame RAW image according to focal plane data to obtain the first area and the second area, wherein the first area and the second area are obtained.
  • One area is the focus area
  • the second area is the image area other than the first area.
  • identifying the first area and the second area of the single-frame RAW image includes: performing content analysis on the single-frame RAW image to obtain the main area and the background area of the single-frame RAW image, wherein the main area is The first area, the background area is the second area.
  • identifying the first area and the second area of a single-frame RAW image includes: acquiring the first N frames of historical-frame RAW images of the single-frame RAW image, where N is an integer greater than 1; acquiring a single-frame RAW image The area where the image changes relative to the RAW image of the historical frame, wherein the changed area is the first area, and the unchanged area is the second area.
  • performing block processing on the first area and the second area includes: performing block processing on the RAW image based on different image block densities of the first area and the second area, wherein the first area is The image block density is greater than the image block density of the second region.
  • performing block processing on a RAW image based on different image block densities in the first area and the second area includes: setting a plurality of horizontal dividing lines along a pixel row direction, and setting a plurality of horizontal dividing lines along a pixel column direction Vertical dividing lines, wherein the horizontal dividing line and the vertical dividing line divide the RAW image into a plurality of image blocks; the distance between the horizontal dividing lines and the distance between the vertical dividing lines running through the first area is configured so that the first area
  • the density of the image blocks in the second area is greater than or equal to the first preset density value; the distance between the horizontal dividing lines and the distance between the vertical dividing lines running through the second area are configured, so that the density of the image blocks in the second area is smaller than the first preset density value; Set the density value.
  • the number of image blocks when performing block processing on a single-frame RAW image based on different image block densities in the first area and the second area is less than or equal to using the first preset density value to perform block processing on the single-frame RAW image.
  • performing block processing on the first area and the second area includes: uniformly dividing a single-frame RAW image to obtain multiple original image blocks;
  • the distance to the focal plane converts a single-frame RAW image into a curved patch image, where the distance from the pixel in the original image patch in the first area to the focal plane is smaller than the distance from the pixel in the original image patch in the second area to the focal plane.
  • the distance of the plane; the surface block image is projected along the direction perpendicular to the focal plane to obtain the projected image block, and the projected image block is used as the block processing result.
  • the method further includes: obtaining a historical frame image associated with the current frame image according to the current frame image; according to the historical frame image, Correct the same area as the historical frame image in the current frame image to obtain the corrected current frame image.
  • the method further includes: segmenting the corrected current frame image, and buffering the segmentation result.
  • the second aspect of the present disclosure provides a RAW image processing chip, including: a first chip, where the first chip is used to identify a first area and a second area of a single frame of raw image; The area and the second area are subjected to block processing, wherein the block processing of the first area and the second area is based on different block processing strategies, and the LUT is respectively performed on the first area and the second area according to the result of the block processing.
  • the second chip, the second chip is connected to the first chip, and the second chip is used to perform LSC processing on the first area and the second area respectively according to the LUT result to obtain the current frame image.
  • an embodiment of the third aspect of the present disclosure provides an electronic device, including a memory, a processor, and an image processing program stored in the memory and running on the processor.
  • the processor executes the image processing program, the The aforementioned RAW image processing method.
  • FIG. 1 is an application scene diagram of a method for processing a RAW image according to an embodiment of the present disclosure
  • FIG. 2 is a flowchart of a method for processing a RAW image according to an embodiment of the present disclosure
  • FIG. 3 is a schematic structural diagram of an image collector according to an embodiment of the present disclosure.
  • Fig. 4 is the image block strategy of the prior art
  • FIG. 5 is an image segmentation strategy according to an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of a curved image block converted from the original image block according to the distance between the pixel point and the focal plane in the original image block;
  • FIG. 7 is a schematic structural diagram of an image processing system according to an embodiment of the present disclosure.
  • FIG. 8 is a schematic structural diagram of a RAW image processing chip according to an embodiment of the present disclosure.
  • FIG. 9 is a schematic structural diagram of a RAW image processing chip according to yet another embodiment of the present disclosure.
  • the LSC Lins Shading Correction
  • the present disclosure provides a RAW image processing method, chip, and electronic device, which can identify regions of an image, and differentiate and process different regions based on the region identification results, so as to improve image processing effects.
  • the RAW image processing method provided by the present disclosure can be applied to the electronic device as shown in FIG. 1 .
  • the electronic device includes an image collector 11 , a front-end image signal processing chip 12 and a back-end application processing chip 13 , wherein the image collector 11 is used to acquire a single-frame RAW image.
  • the front-end image signal processing chip 12 is connected to the image collector 11, and is used for receiving a single-frame RAW image, identifying the first area and the second area of the single-frame RAW image, and performing block processing on the first area and the second area, Among them, the block processing of the first area and the second area is based on different block processing strategies, and then the LUT calculation is performed on the first area and the second area respectively according to the block processing result, and the LUT result is obtained; the back-end application processing chip 13 It is used to perform LSC processing on the first area and the second area respectively according to the LUT result to obtain the current frame image.
  • the electronic device may be a device with a photographing or video recording function, such as a mobile phone, a tablet computer, a personal computer, a smart camera, and a vehicle-mounted image acquisition device.
  • a method for processing a RAW image is provided.
  • the method can be applied to the electronic device shown in FIG. 1, and the method can include the following steps:
  • Step S101 identifying the first area and the second area of the single-frame RAW image.
  • the image collector of the electronic device is composed of a lens 31 and an image sensor 32 , wherein the lens 31 is used to collect an external light source signal and provide it to the image sensor, and the image sensor 32 should come from the lens 31
  • the light source signal is converted into digitized raw image data, that is, a RAW image.
  • RAW images are unprocessed and uncompressed image formats.
  • the image collector acquires a single-frame RAW image and sends the single-frame RAW image to the front-end image signal processing chip.
  • the front-end image signal processing chip performs region recognition on the single-frame RAW image to identify the first and second regions in the single-frame RAW image. area, where the first area is the image area that the user pays attention to, such as the area where the user manually focuses or the area where the target object is located in the image identified by the relevant image algorithm, and the second area is the image area outside the first area.
  • the front-end image signal processing chip can use an image content analysis algorithm, an image recognition algorithm, etc. to identify the target object in the image to determine the first area and the second area of the current single-frame RAW image.
  • identifying the first area and the second area of the single-frame RAW image includes: performing area division on the single-frame RAW image according to focal plane data to obtain the first area and the second area, wherein the first area is is the focus area, and the second area is the image area other than the first area.
  • the user when a user uses an electronic device to capture an image, the user usually focuses manually on an area of interest.
  • the focus area is the area that the user pays attention to, that is, the focus area may be used as the first area of the single-frame RAW image.
  • the focal plane data can be obtained.
  • the image recognition model can be used to identify the target object image corresponding to the focus area. After the target object image is identified, the boundary of the target object image can be divided. , take the area where the target object image is located as the first area.
  • the front-end image signal processing chip obtains the focal plane data. If the focal plane is located in the face area, the front-end image signal processing chip extracts the face contour, and determines the face contour area as the first area. The image area other than the area is the second area. It can be understood that the RAW image may include multiple faces, so that the front-end processing image signal chip can extract multiple face regions, and in this case, the first region may be multiple.
  • identifying the first area and the second area of the single-frame RAW image includes: performing content analysis on the single-frame RAW image to obtain the subject area and the background area of the single-frame RAW image, where the subject area is the first area, and the background area is the second area.
  • the RAW image in this embodiment includes a subject area and a background area, where the subject area is the area where the target object that the user pays attention to in the image is located, and the background area is the part other than the subject area, for example, in a portrait of a person, The person is the main area, and the image area other than the person area is the background area.
  • a trained image recognition model can be used to recognize people, plants, animals, vehicles, buildings, etc. in the image, and after the target object in the image is recognized, the contour of the target object is extracted, and the contour area of the target object is determined. is the subject area (ie, the first area), and the image area other than the subject area is determined as the background area (ie, the second area).
  • a foreground extraction algorithm can also be used to extract the foreground area of the image, wherein the foreground area is the main area of the image, and the image area other than the foreground area is the background area of the image.
  • the present disclosure may also use other methods to extract the subject area and the background area of the RAW image, which is not limited in this embodiment.
  • identifying the first area and the second area of the single-frame RAW image includes: obtaining the previous N frames of the single-frame RAW image historical frame RAW images, where N is an integer greater than 1; obtaining a single-frame RAW image The area where the image changes relative to the RAW image of the historical frame, wherein the changed area is the first area, and the unchanged area is the second area.
  • the area that users focus on is usually a dynamically changing area. Therefore, when performing LSC processing on an image in a video scene, the changing area in each frame of image can be used as the first area. Processing, give priority to ensuring the processing effect of the changed area.
  • the raw image of the previous frame or the previous N frames of the raw image of the current frame may be acquired, and the raw image of the current frame may be compared with the raw image of the previous frame or the previous N frames of the raw image of the current frame , to determine the area where the RAW image of the current frame has changed, and determine the changed area as the first area, and the unchanged area as the second area, where the value of N can be set according to the actual situation, this implementation The example does not limit this.
  • the method for processing a RAW image provided by the above-mentioned embodiment extracts the first area and the second area in the single-frame RAW image by performing area identification on the single-frame RAW image, so that the first area that the user pays attention to can be processed during image processing.
  • the LSC processing effect is enhanced. Since, for an image, the user's evaluation weight on the photo quality is also mainly reflected in the area that the user pays attention to, the user experience can be improved by enhancing the LSC processing effect of the first area that the user pays attention to.
  • this scheme provides a real-time differentiated block processing and LSC processing scheme. The difference between the frame and the frame will lead to the difference of the block density of different regions due to the difference of the region recognition results. Therefore, the image processing effect of the first region concerned by the user in each frame is enhanced, which can better meet the needs of the user and improve the user experience.
  • Step S102 performing block processing on the first area and the second area, wherein the block processing on the first area and the second area is based on different block processing strategies.
  • the block processing of the RAW image is usually based on the entire image to uniformly block, and the LUT (Look Up Table, LUT) of the entire image is calculated based on the result of the uniform block. Lookup table) result, when the LSC processing is performed subsequently according to the LUT result, the overall processing effect of the image can be guaranteed.
  • LUT Look Up Table
  • the processing standard of the image area that the user focuses on may be lowered, resulting in that although the overall processing effect of the entire image meets the requirements, the key areas that the user pays attention to.
  • the processing effect is biased, which reduces the user experience. Therefore, in this embodiment, after identifying the first area and the second area of the single-frame RAW image, different segmentation strategies are used to perform block processing on the first area and the second area based on the identification result, so as to realize the first area Differentiate processing from the second area, improve the processing accuracy of the first area, and improve the processing effect of the image area that the user pays attention to on the premise of not affecting the overall processing effect.
  • performing block processing on the first area and the second area includes: performing block processing on a single-frame RAW image based on different image block densities of the first area and the second area, wherein the first area The image block density is greater than the image block density of the second region.
  • Image block density is the number of image blocks per unit area. The area with higher density of image blocks, the smaller the area of each image block, and the fewer pixels in the image block. Since the subsequent calculation of LUT results is based on the results of image segmentation, sampling and calculation are performed. Each image block only calculates the LUT results of the sampling points. The LUT results of the pixels in the image block are determined by the sampling points using bilinear interpolation method calculate. Therefore, the larger the density of the image block, the smaller the image block, and the more accurate the calculation result of the pixel points in the image block calculated by the bilinear interpolation method.
  • the image block size of the first area is configured to be smaller than the image block size of the second area, so that the image block density of the first area is greater than that of the second area, thereby improving the pixel processing accuracy of the first area. , to improve the image processing effect of the first area.
  • performing block processing on the RAW image includes: setting a plurality of horizontal dividing lines along the pixel row direction, and setting a plurality of vertical dividing lines along the pixel column direction, wherein the horizontal dividing line and the vertical dividing line divide the RAW image
  • the image is divided into a plurality of image blocks; the distance between the horizontal dividing lines and the distance between the vertical dividing lines that run through the first area are configured, so that the density of the image blocks in the first area is greater than or equal to the first preset density value;
  • the distance between the horizontal dividing line and the vertical dividing line in the second area makes the density of the image blocks in the second area smaller than the first preset density value.
  • the size and density of the image blocks are determined by the distance between the dividing lines.
  • the first area is located in the middle area of the image, the distance between the horizontal dividing lines and the distance between the vertical dividing lines in the middle area are smaller, and the area dividing lines outside the middle area are The distance between them is large, so that the image blocks in the middle area of the image are small in size and high in density, and the image blocks in the areas outside the middle area are large in size and low in density.
  • the image block density of the first area can be set to be greater than or equal to the first preset density value
  • the image block density of the second area can be set to be greater than or equal to the first preset density value.
  • the density is smaller than the first preset density value, so that the image processing effect of the first area is the best, and the image processing effect of the second area is second, thereby enhancing the image processing effect of the first area.
  • a second preset density value may be used, and the second preset density value is smaller than the first preset density value.
  • the image block density reaches the first preset density value
  • the corresponding image processing effect is the best
  • the image block density reaches the second preset density value
  • the basic image processing effect can be achieved.
  • the density of image blocks in the first area is greater than or equal to the first preset density value
  • the density of image blocks in the second area is less than the first preset density value and greater than or equal to the first preset density value.
  • the present disclosure can also divide an image into multiple regions, and set pixel block densities in different regions according to the user's degree of attention to different regions.
  • the second area can be divided into sub-key areas and non-key areas according to the distance from the first area. , the image block density of the first area is the largest, the image block density of the sub-key area is the second, and the image block density of the non-key area is the smallest.
  • the number of image blocks when performing block processing on the RAW image based on different image block densities of the first area and the second area is less than or equal to using the first preset density value to uniformly block the single frame of the RAW image The number of image blocks to process.
  • the number of image blocks when the single-frame RAW image is uniformly divided by using the first preset density value is calculated.
  • the distance between the dividing lines after configuring the pixel block density of the first area to reach the first preset density value, reduce the image block density value of the second area, so that the processing effect of the second area can reach basic image processing. effect, and the number of image blocks in the entire RAW image is smaller than the number of image blocks when uniformly divided by the first preset density value, so that on the one hand, the number of blocks can be reduced, the amount of LUT result data in subsequent calculations can be reduced, and the algorithm can be reduced.
  • the processing pressure also reduces the power consumption of the system and improves the processing speed.
  • the first area achieves a better processing effect.
  • the processing of the entire image The effect is to meet the needs of the user, and the processing effect of the first area exceeds the needs of the user, which can improve the user experience.
  • performing block processing on the first area and the second area includes: uniformly dividing a single-frame RAW image to obtain multiple original image blocks;
  • the distance from the plane converts the single-frame RAW image into a curved patch image, wherein the distance from the pixel point in the original image block in the first area to the focal plane is smaller than the distance from the pixel point in the original image block in the second area to the focal plane.
  • the distance of the plane; the surface block image is projected along the direction perpendicular to the focal plane to obtain the projected image block, and the projected image block is used as the image segmentation result.
  • the solution of this example can be used to divide the image. Specifically, a single-frame RAW image is firstly divided into uniform blocks, and the density of the image blocks can be set as required, and after uniformly divided into blocks, multiple original image blocks are obtained. Obtain the distance from the pixel point on each original image block to the focal plane. In this embodiment, the distance between the point on each original image block and the center point of the focal plane can be calculated, and then the distance between each original image block can be calculated. The position coordinates of the pixel points are converted into the distance coordinates between the pixel points and the focal plane.
  • a single-frame RAW image can be converted into a surface block image, as shown in the figure 6 shown.
  • Each surface patch on the surface patch image corresponds to a raw image patch on the RAW image.
  • Step S103 perform LUT calculation according to the image block result to obtain the LUT result.
  • the correction of the lens shadow is to correct the four channels of the bayer respectively, and the correction process of each channel is a relatively independent process.
  • the LUT results of the entire image are usually not stored, but the entire image is divided and sampled, and the gain of the sampling points is stored. gain. Therefore, in this embodiment, sampling can be performed according to the image segmentation result, and the LUT result of the sampling point can be calculated.
  • the calculation method of the LUT is similar to the calculation method in the traditional technology, and details are not described herein again.
  • Step S104 perform LSC processing on the RAW image according to the LUT result to obtain the current frame image.
  • the front-end image signal processing chip After the front-end image signal processing chip calculates the LUT result, it sends the LUT result to the back-end application processing chip, so that the back-end application processing chip can perform lens shading correction on the RAW image according to the LUT result.
  • the back-end application processing chip uses bilinear interpolation to calculate the LUT results of each pixel in the image block according to the LUT results of the sampling points, and performs lens shading correction on the RAW image according to the LUT results of all pixels. , to obtain the preliminarily processed current frame image.
  • the LUT result can be obtained based on the information extraction module.
  • the front-end image processing chip 12 also performs LTM (Local Tone Mapping, local tone mapping) preprocessing on the image.
  • LTM Local Tone Mapping, local tone mapping
  • the characteristics are restored to the RAW image, and then the preprocessed RAW image and the LUT result are sent to the back-end application processing chip 13, so that the back-end application processing chip 13 performs LSC processing and other image algorithm processing on the image based on the LUT result to achieve Expected image effect.
  • the method further includes: according to the current frame image and the historical frame image associated with the current frame image; Correct the same area as the historical frame image to obtain the corrected current frame image.
  • the historical frame image refers to the historical frame image that is stored in the image database and has gone through the processing flow of steps S101 to S104.
  • a historical frame image most similar to the current frame image may be firstly found from the image database as the associated historical frame image of the current frame image.
  • searching the historical frame image most similar to the current frame image can be searched according to the feature points on the current frame image.
  • the historical frame image may be an entire frame image, or may be a block image of the entire frame image.
  • the block density of the first area is greater than that of the second area, and the processing accuracy of the first area is also greater than that of the second area. Therefore, when performing LSC After the processing, the second area may be modified according to the processed historical frame images, so as to improve the processing effect of the second area.
  • the pixel data of the processed historical frame image can be assigned to the pixel points of the same area in the current frame image, so as to complete the correction of the current frame image.
  • the first region can also be enhanced and fused according to the historical frame images, so as to further improve the image processing effect of the first region.
  • the number of the used historical frame images can be dynamically controlled according to the image effect of the current frame image, and it is not necessary to fix how many frames are used. For example, it can be dynamically adjusted according to the changes of the scene and the characteristics of system power consumption and temperature rise, so that the processing pressure of the system and the image effect can reach a relatively balanced level. Of course, the image effect can meet the user's requirements.
  • the frame number of the historical frame image may also be adjusted according to the change of the correction effect. As the number of historical frames used increases, the effect of image correction will gradually improve. Therefore, more historical frame data can be used at the beginning of correction, and the number of frames can be gradually reduced when a certain correction accuracy is achieved. In order to reduce the processing pressure of the system on the basis of ensuring the correction effect.
  • the current frame image is also segmented, and the segmented image segments are cached.
  • an equal division method may be adopted, for example, the image is divided into image segments equally in a 3*3 manner, and the image segments are cached so as to be used as historical frame images to provide a basis for subsequent image correction.
  • the image can also be segmented according to feature extraction, the subject area and the background area in the image can be extracted, and the subject area and the background area can be saved respectively, so as to be used as a historical frame image to provide a basis for subsequent image correction.
  • the method for processing a RAW image provided by the above-mentioned embodiment, by identifying the first area and the second area of a single-frame RAW image, and dividing the first area and the second area based on different segmentation strategies of the first area and the second area processing, perform LUT calculation according to the block processing result, obtain the LUT result, perform LSC processing on the first area and the second area according to the LUT result, and obtain the current frame image, so that the first area in the image that the user pays attention to can be identified, and Distinguishing processing is performed on the first area and the second area, so as to improve the image processing effect of the first area that the user pays attention to.
  • an embodiment of the present disclosure provides a RAW image processing chip, including a first chip 110 and a second chip 120 .
  • the first chip 110 is configured to identify the first area and the second area of the single-frame RAW image, and perform block processing on the first area and the second area, wherein the block processing on the first area and the second area is based on different According to the block processing strategy, LUT calculation is performed on the first area and the second area according to the block processing result, and the LUT result is obtained.
  • the second chip 120 is connected to the first chip 110, and the second chip 120 is configured to perform LSC processing on the first area and the second area according to the LUT result to obtain the current frame image.
  • the first chip 110 includes a region identification module 111 , and the region identification module 111 is configured to perform region identification on a single-frame RAW image to obtain the first region and the second region.
  • the area identification module 111 is specifically configured to perform area division on the RAW image according to the focal plane data to obtain a first area and a second area, wherein the first area is the focus area, and the second area is the first area area of the image outside the area.
  • the region identification module 111 is specifically configured to perform content analysis on a single-frame RAW image to obtain a subject region and a background region of the single-frame RAW image, wherein the subject region is the first region and the background region is the first region. Second area.
  • the region identification module 111 is specifically configured to acquire the first N frames of the raw images of the historical frames of the raw images of the single frame, where N is an integer greater than 1; the acquired raw images of the single frame are changed relative to the raw images of the historical frames area, wherein the area that has changed is the first area, and the area that has not changed is the second area.
  • the first chip 110 further includes an image segmentation module 112, and the image segmentation module 112 is configured to perform segmentation between the first area and the second area based on different segmentation strategies of the first area and the second area.
  • the second area is divided into blocks.
  • the image block module 112 is specifically configured to perform block processing on the RAW image based on different image block densities of the first area and the second area, wherein the image block density of the first area is greater than that of the second area. Image block density.
  • the image segmenting module 112 is configured to set a plurality of horizontal dividing lines along the pixel row direction, and set a plurality of vertical dividing lines along the pixel column direction, wherein the horizontal dividing line and the vertical dividing line divide a single frame of RAW image into Multiple image blocks.
  • the distance between the horizontal dividing lines and the distance between the vertical dividing lines running through the first area is configured, so that the density of the image blocks in the first area is greater than or equal to a first preset density value.
  • the distance between the horizontal dividing lines and the distance between the vertical dividing lines running through the second area is configured so that the density of the image blocks in the second area is smaller than the first preset density value.
  • the number of image blocks when performing block processing on a single-frame RAW image based on different image block densities in the first region and the second region is less than or equal to using the first preset density value to uniformize the single-frame RAW image.
  • the number of image blocks when chunking is less than or equal to using the first preset density value to uniformize the single-frame RAW image.
  • the image segmentation module 112 is specifically configured to perform uniform segmentation on a single frame of RAW image to obtain multiple original image blocks.
  • the single-frame RAW image is converted into a curved block image according to the distance from the pixel point on each original image block to the focal plane, wherein the distance from the pixel point in the original image block in the first area to the focal plane is smaller than that in the second area The distance from the pixel in the original image block to the focal plane.
  • the first chip 110 further includes a processing module 113, and the processing module 113 is configured to perform LUT calculation on the first area and the second area respectively according to the block processing result to obtain the LUT result.
  • the second chip 120 further includes a correction module 121, and the correction module 121 is specifically configured to obtain a historical frame image associated with the current frame image according to the current frame image, and according to the historical frame image, Correct the same area as the historical frame image in the current frame image to obtain the corrected current frame image.
  • the second chip 120 further includes a segmentation module 122 and a cache module 123 , wherein the segmentation module 122 is used to segment the corrected current frame image, and the cache module 123 is used to segment the corrected current frame image.
  • the segmentation result is cached.
  • the RAW image processing chip provided by the above-mentioned embodiment identifies the first area and the second area of the single-frame RAW image through the first chip, and separates the first area and the second area based on different segmentation strategies of the first area and the second area.
  • the area is subjected to block processing, and LUT calculation is performed on the first area and the second area respectively according to the block processing result to obtain the LUT result, and the first area and the second area are respectively LSC processed by the second chip according to the LUT result,
  • the current frame image is obtained, so that the first area in the image that the user pays attention to can be identified, the first area and the second area are differentiated, and the image processing effect of the first area that the user pays attention to is improved.
  • Yet another embodiment of the present disclosure provides an electronic device including a memory, a processor, and an image processing program stored on the memory and executable on the processor.
  • the processor executes the image processing program, the aforementioned RAW image processing method is implemented.
  • the above-mentioned electronic device can identify the first area in the image that the user pays attention to, and perform different processing on the first area and the second area, so as to improve the image processing effect of the first area that the user pays attention to.
  • a "computer-readable medium” can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus.
  • computer readable media include the following: electrical connections with one or more wiring (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM).
  • the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.
  • portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.
  • various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.
  • first and second are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with “first”, “second” may expressly or implicitly include at least one of that feature.
  • plurality means at least two, such as two, three, etc., unless expressly and specifically defined otherwise.
  • the terms “installed”, “connected”, “connected”, “fixed” and other terms should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection , or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between the two elements, unless otherwise specified limit.
  • installed may be a fixed connection or a detachable connection , or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between the two elements, unless otherwise specified limit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Image Processing (AREA)

Abstract

本公开涉及一种RAW图像的处理方法、芯片、计算机可读存储介质和电子设备。其中，图像的处理方法通过识别单帧RAW图像的第一区域和第二区域，基于第一区域和第二区域的不同分块策略对第一区域和第二区域进行分块处理，根据分块处理结果进行LUT计算，获得LUT结果，根据LUT结果对第一区域和第二区域进行LSC处理，获得当前帧图像。

Description

RAW图像的处理方法、芯片和电子设备
相关申请的交叉引用
本公开要求于2021年03月26日提交的申请号为202110328651.3，名称为“RAW图像的处理方法、芯片和电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及图像处理技术领域，尤其涉及一种RAW图像的处理方法、芯片和电子设备。
背景技术
目前，对于图像的LSC(Lens Shading Correction，镜头阴影校正)处理通常是基于整幅图像进行均匀处理。
公开内容
本公开的目的在于提出一种RAW图像的处理方法、芯片和电子设备。
为达到上述目的，本公开第一方面实施例提出了一种RAW图像的处理方法，包括：识别单帧RAW图像的第一区域和第二区域；对第一区域和第二区域进行分块处理，其中，对第一区域和第二区域的分块处理基于不同的分块处理策略；根据分块处理结果分别对第一区域和第二区域进行LUT计算，获得LUT结果；根据LUT结果分别对第一区域和第二区域进行LSC处理，获得当前帧图像。
根据本公开的一个实施例，识别单帧RAW图像的第一区域和第二区域，包括：根据焦平面数据对单帧RAW图像进行区域划分，以获得第一区域和第二区域，其中，第一区域为对焦区域，第二区域为第一区域以外的图像区域。
根据本公开的一个实施例，识别单帧RAW图像的第一区域和第二区域，包括：对单帧RAW图像进行内容分析以获得单帧RAW图像的主体区域和背景区域，其中，主体区域为第一区域，背景区域为第二区域。
根据本公开的一个实施例，识别单帧RAW图像的第一区域和第二区域，包括：获取单帧RAW图像的前N帧历史帧RAW图像，其中N为大于1的整数；获取单帧RAW图像相对于历史帧RAW图像发生变化的区域，其中，发生变化的区域为第一区域，未发生变化的区域为第二区域。
根据本公开的一个实施例，对第一区域和第二区域进行分块处理，包括：基于第一区域和第二区域的不同图像块密度对RAW图像进行分块处理，其中，第一区域的图像块密度大 于第二区域的图像块密度。
根据本公开的一个实施例，基于第一区域和第二区域的不同图像块密度对RAW图像进行分块处理，包括：沿像素行方向设置多条横向分割线，并沿像素列方向设置多条纵向分割线，其中，横向分割线和纵向分割线将RAW图像分割为多个图像块；配置贯穿第一区域的横向分割线之间的距离和纵向分割线之间的距离，以使第一区域内的图像块密度大于等于第一预设密度值；配置贯穿第二区域的横向分割线之间的距离和纵向分割线之间的距离，以使第二区域内的图像块密度小于第一预设密度值。
根据本公开的一个实施例，基于第一区域和第二区域的不同图像块密度对单帧RAW图像进行分块处理时的图像块数量小于等于采用第一预设密度值对单帧RAW图像进行均匀分块处理时的图像块数量。
根据本公开的一个实施例，对第一区域和第二区域进行分块处理，包括：对单帧RAW图像进行均匀分块，获取多个原始图像块；根据每个原始图像块上的像素点到焦平面的距离将单帧RAW图像转换为曲面块图像，其中，第一区域内的原始图像块中的像素点到焦平面的距离小于第二区域内的原始图像块中的像素点到焦平面的距离；将曲面块图像沿垂直于焦平面的方向进行投影，获得投影图像块，并将投影图像块作为分块处理结果。
根据本公开的一个实施例，在根据LUT结果分别对第一区域和第二区域进行LSC处理后，还包括：根据当前帧图像获取与当前帧图像相关联的历史帧图像；根据历史帧图像，对当前帧图像中与历史帧图像相同的区域进行修正，以获得修正后的当前帧图像。
根据本公开的一个实施例，该方法还包括：对修正后的当前帧图像进行分割，并对分割结果进行缓存。
为达到上述目的，本公开第二方面实施例提出了一种RAW图像的处理芯片，包括：第一芯片，第一芯片用于识别单帧raw图像的第一区域和第二区域；对第一区域和第二区域进行分块处理，其中，对第一区域和第二区域的分块处理基于不同的分块处理策略，并根据所分块处理结果分别对第一区域和第二区域进行LUT计算，获得LUT结果；第二芯片，第二芯片与第一芯片相连，第二芯片用于根据LUT结果分别对第一区域和第二区域进行LSC处理，获得当前帧图像。
为达到上述目的，本公开第三方面实施例提出了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的图像处理程序，处理器执行图像处理程序时，实现前述RAW图像的处理方法。
本公开附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本公开的实践了解到。
附图说明
图1为根据本公开实施例的RAW图像的处理方法的应用场景图；
图2为根据本公开实施例的RAW图像的处理方法的流程图；
图3为根据本公开实施例的图像采集器的结构示意图；
图4为现有技术的图像分块策略；
图5为根据本公开实施例的图像分块策略；
图6为根据原始图像块中像素点与焦平面的距离由原始图像块转换而成的曲面图像块示意图；
图7为根据本公开实施例的图像处理系统的结构示意图；
图8为根据本公开一个实施例的RAW图像的处理芯片的结构示意图；
图9为根据本公开又一实施例的RAW图像的处理芯片的结构示意图。
具体实施方式
下面详细描述本公开的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本公开，而不能理解为对本公开的限制。
目前，对于图像的LSC(Lens Shading Correction，镜头阴影校正)处理通常是基于整幅图像进行均匀处理，无法对用户关注的重点区域进行着重处理，使得重点区域的处理效果有所偏差。基于此，本公开了提供了一种RAW图像的处理方法、芯片和电子设备，能够识别图像的区域，并基于区域识别结果对不同的区域进行区分处理，以提高图像处理效果。
本公开提供的RAW图像的处理方法，可以应用于如图1所示的电子装置中。电子装置包括图像采集器11、前端图像信号处理芯片12和后端应用处理芯片13，其中，图像采集器11用于获取单帧RAW图像。前端图像信号处理芯片12与图像采集器11相连，用于接收单帧RAW图像，并识别单帧RAW图像的第一区域和第二区域，以及对第一区域和第二区域进行分块处理，其中，第一区域和第二区域的分块处理基于不同的分块处理策略，然后根据分块处理结果分别对第一区域和第二区域进行LUT计算，获得LUT结果；后端应用处理芯片13用于根据LUT结果分别对第一区域和第二区域进行LSC处理，获得当前帧图像。电子装置可以为手机、平板电脑、个人计算机、智能相机以及车载图像采集设备等具有拍照或摄像功能的设备。
在本公开中，参考图2所示，提供了一种RAW图像的处理方法，该方法可以应用于图 1所示的电子装置，该方法可包括以下步骤：
步骤S101，识别单帧RAW图像的第一区域和第二区域。
本实施例中，如图3所示，电子装置的图像采集器由透镜31和图像传感器32构成，其中透镜31用于采集外部的光源信号提供给图像传感器，图像传感器感32应来自于透镜31的光源信号，将其转换为数字化的原始图像数据，即RAW图像。其中，RAW图像是未经处理、也未经压缩的图像格式。
图像采集器获取单帧RAW图像后将单帧RAW图像发送至前端图像信号处理芯片，前端图像信号处理芯片对单帧RAW图像进行区域识别，以识别单帧RAW图像中的第一区域和第二区域，其中，第一区域为用户关注的图像区域，例如用户手动对焦的区域或采用相关的图像算法识别出的图像中目标物体所在的区域，第二区域为第一区域以外的图像区域。本公开中，前端图像信号处理芯片可以利用图像内容分析算法、图像识别算法等识别图像中的目标物体以确定当前单帧RAW图像的第一区域和第二区域。
作为第一种示例，识别单帧RAW图像的第一区域和第二区域，包括：根据焦平面数据对单帧RAW图像进行区域划分，以获得第一区域和第二区域，其中，第一区域为对焦区域，第二区域为第一区域以外的图像区域。
具体来说，当用户使用电子装置采集图像时，对于感兴趣的区域，用户通常会手动对焦。其中，对焦区域即为用户关注的区域，也即可以将对焦区域作为单帧RAW图像的第一区域。在对RAW图像进行区域识别时，可以获取焦平面数据，进一步地，可以利用图像识别模型，识别对焦区域对应的目标物体图像，在识别出目标物体图像后，可以将目标物体图像的边界划分出来，将目标物体图像所在的区域作为第一区域。
举例来说，当单帧RAW图像中包括人物图形时，通常用户会对人脸图像所在的区域或人体图像区域进行对焦。在用户执行对焦操作后，前端图像信号处理芯片获取焦平面数据，若焦平面位于人脸区域，则前端图像信号处理芯片提取人脸轮廓，并将人脸轮廓区域确定为第一区域，第一区域以外的图像区域为第二区域。可以理解的是，RAW图像中可包括多张人脸，从而前端处理图像信号芯片可以提取多个人脸区域，此时，第一区域可以为多个。
作为第二种示例，识别单帧RAW图像的第一区域和第二区域，包括：对单帧RAW图像进行内容分析以获得单帧RAW图像的主体区域和背景区域，其中，主体区域为第一区域，背景区域为第二区域。
具体来说，本实施例中的RAW图像包括主体区域和背景区域，其中，主体区域为图像中用户关注的目标对象所在的区域，背景区域为主体区域以外的部分，例如，在人物照中，人物为主体区域，人物区域以外的图像区域为背景区域。本实施例可以采用训练好的图像 识别模型识别图像中的人物、植物、动物、车辆、建筑等，并在识别到图像中的目标物体后，提取目标物体的轮廓，将目标物体的轮廓区域确定为主体区域(也即第一区域)，主体区域以外的图像区域确定为背景区域(也即第二区域)。或者，也可以采用前景提取算法提取图像的前景区域，其中，前景区域即为图像的主体区域，前景区域以外的图像区域即为图像的背景区域。当然，本公开也可采用其他方式提取RAW图像的主体区域和背景区域，本实施例对此不作限制。
作为第三种示例，识别单帧RAW图像的第一区域和第二区域，包括：获取与单帧RAW图像的前N帧历史帧RAW图像，其中，N为大于1的整数；获取单帧RAW图像相对于历史帧RAW图像发生变化的区域，其中，发生变化的区域为第一区域，未发生变化的区域为第二区域。
具体来说，在视频场景中，用户重点关注的区域通常为动态变化的区域，因此，在对视频场景中的图像进行LSC处理时，可以将每一帧图像中的变化区域作为第一区域进行处理，优先保证变化区域的处理效果。本实施例中，可以获取当前帧RAW图像的前一帧或前N帧历史帧RAW图像，并将当前帧RAW图像与当前帧RAW图像的前一帧或前N帧的历史帧RAW图像进行比较，以判断当前帧RAW图像发生变化的区域，并将发生变化的区域确定为第一区域，未发生变化的区域确定为第二区域，其中，N的取值可以根据实际情况设定，本实施例对此不作限制。
上述实施例提供的RAW图像的处理方法，通过对单帧RAW图像进行区域识别，提取单帧RAW图像中的第一区域和第二区域，从而在图像处理时，可以对用户关注的第一区域的LSC处理效果进行增强。由于对于一幅图像，用户对照片质量的评价权重也主要体现在用户关注的区域，因此通过增强用户关注的第一区域的LSC处理效果，可以提升用户体验。并且，对于预览场景和视频场景来说，本方案提供一种实时差异化的分块处理和LSC处理方案，帧与帧之间由于区域识别结果的不同就会导致不同区域分块密度的区别，从而使每一帧中用户关注的第一区域图像处理效果均得到增强，能更好的满足用户的需求，提升用户的体验。
步骤S102，对第一区域和第二区域进行分块处理，其中，对第一区域和第二区域的分块处理基于不同的分块处理策略。
如图4所示，目前，在进行LSC处理过程中，对于RAW图像的分块处理通常是基于整幅图像进行均匀分块，基于均匀分块的结果计算整幅图像的LUT(Look Up Table，查找表)结果，在后续根据LUT结果进行LSC处理时，能够保证图像整体的处理效果。但是，由于在分块时采用对整幅图像均匀分块的方式，导致图像的各个区域处理精度相同，处理结果 无法体现整幅图像的主次关系。并且，在一定的算力要求下，为了平衡整体图像的处理效果，可能会使得用户重点关注的图像区域处理标准降低，导致虽然整幅图像的整体处理效果满足要求，但是用户关注的重点区域的处理效果有所偏差，降低了用户体验。为此，本实施例在识别出单帧RAW图像的第一区域和第二区域后，基于识别结果采用不同分块策略对第一区域和第二区域进行分块处理，以实现对第一区域和第二区域的差异化处理，提高第一区域的处理精度，在不影响整体处理效果的前提下，提高用户关注的图像区域的处理效果。
作为第一种示例，对第一区域和第二区域进行分块处理，包括：基于第一区域和第二区域的不同图像块密度对单帧RAW图像进行分块处理，其中，第一区域的图像块密度大于第二区域的图像块密度。
具体地，如图5所示，本实施例在对第一区域和第二区域分块处理时，可以为不同的图像区域设置不同的图像块密度。图像块密度为单位面积内图像块的数量。图像块密度越大的区域，每个图像块的面积越小，进而图像块内的像素点越少。由于后续在进行LUT结果计算时，是基于图像分块的结果进行采样、计算，每个图像块只计算采样点的LUT结果，图像块内像素点的LUT结果由采样点利用双线性插值法计算。因此，图像块密度越大，图像块越小，利用双线性插值法计算的图像块内的像素点的计算结果越精确。本实施例中，配置第一区域的图像块尺寸小于第二区域的图像块尺寸，使得第一区域的图像块密度大于第二区域的图像块密度，进而可以提高第一区域的像素点处理精度，提升第一区域的图像处理效果。
在一个具体实施例中，对RAW图像进行分块处理包括：沿像素行方向设置多条横向分割线，并沿像素列方向设置多条纵向分割线，其中，横向分割线和纵向分割线将RAW图像分割为多个图像块；配置贯穿第一区域的横向分割线之间的距离和纵向分割线之间的距离，使第一区域内的图像块密度大于等于第一预设密度值；配置贯穿第二区域的横向分割线和纵向分割线之间的距离，使第二区域内的图像块密度小于第一预设密度值。
具体地，对RAW图像进行分块时，图像块的尺寸和图像块密度由分割线之间的距离决定。分割线之间的距离越小，图像块的尺寸越小，单位面积内图像块的密度越大，后续计算LUT结果时图像块内像素点的LUT计算结果越精确。因此，在设置分割线时，可以基于整幅图像的宽度和高度以及需要的图像块密度设置分割线之间的距离。
如图5所示，本实施例中，假设第一区域位于图像的中间区域，则中间区域的横向分割线之间的距离和纵向分割线之间的距离较小，中间区域以外的区域分割线之间的距离较大，从而使得图像中间区域的图像块尺寸小、密度大，中间区域以外的区域图像块尺寸大、密 度小。
当图像块密度达到第一预设密度值时，对应的图像处理效果最好，因此，本实施例可以设置第一区域的图像块密度大于等于第一预设密度值，第二区域的图像块密度小于第一预设密度值，从而使第一区域的图像处理效果最佳，第二区域的图像处理效果次之，以此增强第一区域的图像处理效果。
进一步地，还可以第二预设密度值，第二预设密度值小于第一预设密度值。当图像块密度达到第一预设密度值时，对应的图像处理效果最好，当图像块密度达到第二预设密度值时，可以达到基本的图像处理效果。在配置横向分割线和纵向分割线之间的距离时，使第一区域的图像块密度大于等于第一预设密度值，第二区域的图像块密度小于第一预设密度值且大于等于第二预设密度值，从而使第二区域能够达到基本的图像处理效果，第一区域达到较佳的图像处理效果，在保证整幅图像的处理效果的基础上，增强第一区域的处理效果。
当然，本公开也可以将图像划分为多个区域，按照用户对不同区域的关注程度设置不同区域的像素块密度。例如可以将第二区域按照距离第一区域的远近分为次重点区域、非重点区域等，次重点区域距离第一区域较近，非重点区域距离第一区域较远，在对图像进行分割时，第一区域的图像块密度最大，次重点区域的图像块密度次之，非重点区域的图像块密度最小。
在其中一个实施例中，基于第一区域和第二区域的不同图像块密度对RAW图像进行分块处理时的图像块数量小于等于采用第一预设密度值对单帧RAW图像进行均匀分块处理时的图像块数量。
具体来说，先根据图像的宽度和高度计算采用第一预设密度值对单帧RAW图像均匀分割时图像块的数量。在配置分割线之间的距离时，配置第一区域的像素块密度达到第一预设密度值后，降低第二区域的图像块密度值，使得第二区域的处理效果可以达到基本的图像处理效果，并且使整幅RAW图像中图像块的数量小于采用第一预设密度值均匀分割时的图像块数量，从而一方面可以减少分块数量，降低后续计算的LUT结果数据量，减少了算法处理压力，也减少了系统的功耗，提高了处理速度，另一方面，由于第一区域的图像块密度使第一区域达到了较好的处理效果，对用户来说，整幅图像的处理效果是满足用户需求的，且第一区域的处理效果超出了用户需求，可以提升用户体验。
作为第二种示例，对第一区域和第二区域进行分块处理，包括：对单帧RAW图像进行均匀分块，获取多个原始图像块；根据每个原始图像块上的像素点到焦平面的距离将单帧RAW图像转换为曲面块图像，其中，第一区域内的原始图像块中的像素点到焦平面的距离 小于第二区域内的原始图像块中的像素点到所述焦平面的距离；将曲面块图像沿垂直于焦平面的方向进行投影，获得投影图像块，并将投影图像块作为图像分块结果。
当第一区域和第二区域是根据焦平面数据划分的时，可以采用本示例的方案对图像进行分块。具体地，先对单帧RAW图像进行均匀分块，图像块密度可以根据需要设置，均匀分块后获得多个原始图像块。获取每个原始图像块上像素点到焦平面的距离，本实施例中，可以计算每个原始图像块上的点到焦平面的中心点之间的距离，进而可以将每个原始图像块上的像素点的位置坐标转换为像素点与焦平面之间的距离坐标，由于图像块上各个像素点到焦平面的距离均不相同，从而可以将单帧RAW图像转换为曲面块图像，如图6所示。曲面块图像上的每一个曲面块对应一个RAW图像上的原始图像块。将曲面块图像沿垂直于焦平面的方向进行投影，获得投影图像块，投影图像块即为图像分块结果。可以知道的是，距焦平面越近的原始图像块，其上的像素点与焦平面中心点的距离越小，对应的投影图像块的尺寸越小，进而使得焦平面所在区域的图像块尺寸较小，距离焦平面越远的图像块尺寸越大，从而可以提高焦平面及其附近区域的数据处理精度，以提高图像处理效果。
步骤S103，根据图像分块结果进行LUT计算，获得LUT结果。
目前对于镜头阴影的校正是分别对于bayer的四个通道进行校正，每个通道的校正过程是相对独立的过程。考虑到芯片设计以及数据传输和处理的成本，通常不会存储整幅图像的LUT结果，而是对整幅图像进行分割、采样，存储采样点的增益，例如，存储128*128个采样点的增益。因此，本实施例可以根据图像分块结果进行采样，计算采样点的LUT结果。LUT计算方式与传统技术中的计算方式类似，在此不再赘述。
步骤S104，根据LUT结果对RAW图像进行LSC处理，获得当前帧图像。
前端图像信号处理芯片计算出LUT结果后，将LUT结果发送至后端应用处理芯片，以便后端应用处理芯片根据LUT结果对RAW图像进行镜头阴影校正。后端应用处理芯片在进行校正处理时，根据采样点的LUT结果采用双线性插值法计算图像块中每个像素点的LUT结果，并根据所有像素点的LUT结果对RAW图像进行镜头阴影校正，获得初步处理后的当前帧图像。
如图7所示，在前端图像处理芯片12侧，在进行图像分块后，可以基于信息抽取模块获得LUT结果。前端图像处理芯片12在获得LUT结果后，还对图像进行LTM(Local Tone Mapping，局部色调映射)预处理。需要说明的是，前端图像信号处理芯片20在进行LTM前，会利用LUT结果提前进行LSC图像亮度预处理，然后使用LTM算法对图像亮度及对比度进行调整，在调整后依据LUT结果将图像的LSC特性还原至RAW图像中，然后将预处理后的RAW图像及LUT结果发送至后端应用处理芯片13，使得后端应用处理芯片13 基于LUT结果对图像进行LSC处理及其他图像算法处理，以达到预期的图像效果。
在其中一个实施例中，根据LUT结果对第一区域和第二区域进行LSC处理后，还包括：根据当前帧图像与当前帧图像相关联的历史帧图像；根据历史帧图像，对当前帧图像与历史帧图像相同的区域进行修正，以获得修正后的当前帧图像。
本实施例中，历史帧图像是指存储在图像数据库中、且已经经过步骤S101～步骤S104的处理流程的历史帧图像。在对当前帧图像进行修正时，可以先从图像数据库中找到与当前帧图像最相似的历史帧图像作为当前帧图像的关联历史帧图像。在查找时，可以根据当前帧图像上的特征点查找与当前帧图像最相似的历史帧图像。其中，历史帧图像可以是整帧图像，也可以是整帧图像的分块图像。
在前面的步骤中，由于对RAW图像分块时，第一区域的分块密度大于第二区域的分块密度，第一区域的处理精度也大于第二区域的处理精度，因此，在进行LSC处理之后，可以根据已经处理过的历史帧图像，对第二区域进行修正，以提高第二区域的处理效果。其中，可以将已经处理过的历史帧图像的像素数据赋值于当前帧图像中相同区域的像素点，以完成对当前帧图像的修正。当然，除了对第二区域进行修正，还可以根据历史帧图像，对第一区域进行加强融合，进一步提高第一区域的图像处理效果。
基于历史帧图像进行修正时，可以动态的根据当前帧图像的图像效果控制所使用的历史帧图像的数量，而不必须固定使用多少帧。例如，可以根据场景的变化以及系统功耗和温升的特点动态的进行调整，使得系统的处理压力和图像效果达到一个较为平衡的水平，当然，图像效果是满足用户要求的。或者，也可以根据修正效果的变化进行历史帧图像帧数的调整。随着使用的历史帧图像帧数的增加，图像修正的效果会逐步提高，因此可以在开始修正时使用更多的历史帧帧数据，在达到一定的修正精度时再将帧数逐步降低下来，以便在保证修正效果的基础上减少系统的处理压力。
在对当前帧图像修正完成后，还对当前帧图像进行分割，并将分割获得的图像片段进行缓存。本实施例中，可以采用等分的方式，例如将图像以3*3的方式等分为图像片段，并对图像片段进行缓存，以便作为历史帧图像，为后续图像修正提供依据。或者，也可以将图像按照特征提取进行分割，提取图像中的主体区域和背景区域，并分别保存主体区域和背景区域，以便作为历史帧图像，为后续图像修正提供依据。当然，也可以不对修正后的图像进行分割，直接缓存整帧图像。
上述实施例提供的RAW图像的处理方法，通过识别单帧RAW图像的第一区域和第二区域，基于第一区域和第二区域的不同分块策略对第一区域和第二区域进行分块处理，根据分块处理结果进行LUT计算，获得LUT结果，根据LUT结果对第一区域和第二区域进 行LSC处理，获得当前帧图像，从而可以识别出图像中的用户关注的第一区域，并对第一区域和第二区域进行区分处理，提高用户关注的第一区域的图像处理效果。
此外，如图8所示，本公开的一个实施例提供一种RAW图像的处理芯片，包括第一芯片110和第二芯片120。第一芯片110用于识别单帧RAW图像的第一区域和第二区域，以及对第一区域和第二区域进行分块处理，其中，对第一区域和第二区域的分块处理基于不同的分块处理策略，并根据分块处理结果对第一区域和第二区域进行LUT计算，获得LUT结果。第二芯片120与第一芯片110相连，第二芯片120用于根据LUT结果对第一区域和第二区域进行LSC处理，获得当前帧图像。
如图9所示，在其中一个实施例中，第一芯片110包括区域识别模块111，区域识别模块111用于对单帧RAW图像进行区域识别，获得第一区域和第二区域。
在一种实施方式中，区域识别模块111具体用于根据焦平面数据对RAW图像进行区域划分，以获得第一区域和第二区域，其中，第一区域为对焦区域，第二区域为第一区域以外的图像区域。
在另一种实施方式中，区域识别模块111具体用于对单帧RAW图像进行内容分析以获得单帧RAW图像的主体区域和背景区域，其中，所主体区域为第一区域，背景区域为第二区域。
在又一种实施方式中，区域识别模块111具体用于获取单帧RAW图像的前N帧历史帧RAW图像，其中N为大于1的整数；获取单帧RAW图像相对于历史帧RAW图像发生变化的区域，其中，发生变化的区域为第一区域，未发生变化的区域为第二区域。
如图9所示，在其中一个实施例中，第一芯片110还包括图像分块模块112，图像分块模块112用于基于第一区域和第二区域的不同分块策略对第一区域和第二区域进行分块处理。
在一种实施方式中，图像分块模块112具体用于基于第一区域和第二区域的不同图像块密度对RAW图像进行分块处理，其中，第一区域的图像块密度大于第二区域的图像块密度。
具体来说，图像分块模块112用于沿像素行方向设置多条横向分割线，并沿像素列方向设置多条纵向分割线，其中，横向分割线和纵向分割线将单帧RAW图像分割为多个图像块。配置贯穿第一区域的横向分割线之间的距离和纵向分割线之间的距离，以使第一区域内的图像块密度大于等于第一预设密度值。配置贯穿第二区域的横向分割线之间的距离和纵向分割线之间的距离，以使第二区域内的图像块密度小于第一预设密度值。
在其中一个实施例中，基于第一区域和第二区域的不同图像块密度对单帧RAW图像进行分块处理时的图像块数量小于等于采用第一预设密度值对单帧RAW图像进行均匀分块 处理时的图像块数量。
在另一种实施方式中，图像分块模块112具体用于对单帧RAW图像进行均匀分块，获取多个原始图像块。根据每个原始图像块上的像素点到焦平面的距离将单帧RAW图像转换为曲面块图像，其中，第一区域内的原始图像块中的像素点到焦平面的距离小于第二区域内的原始图像块中的像素点到焦平面的距离。将曲面块图像沿垂直于焦平面的方向进行投影，获得投影图像块，并将投影图像块作为图像分块结果。
如图9所示，在其中一个实施例中，第一芯片110还包括处理模块113，处理模块113用于根据分块处理结果分别对第一区域和第二区域进行LUT计算，获得LUT结果。
如图9所示，在其中一个实施例中，第二芯片120还包括修正模块121，修正模块121具体用于根据当前帧图像获取与当前帧图像相关联的历史帧图像，根据历史帧图像，对当前帧图像中与历史帧图像相同的区域进行修正，以获得修正后的当前帧图像。
如图9所示，在其中一个实施例中，第二芯片120还包括分割模块122和缓存模块123，其中，分割模块122用于对修正后的当前帧图像进行分割，缓存模块123用于对分割结果进行缓存。
需要说明的是，关于本公开中的RAW图像的处理芯片的描述，请参考本公开中关于ARW图像的处理方法的描述，具体这里不再赘述。
上述实施例提供的RAW图像的处理芯片，通过第一芯片识别单帧RAW图像的第一区域和第二区域，以及基于第一区域和第二区域的不同分块策略对第一区域和第二区域进行分块处理，并根据分块处理结果对第一区域和第二区域分别进行LUT计算，获得LUT结果，以及通过第二芯片根据LUT结果分别对第一区域和第二区域进行LSC处理，获得当前帧图像，从而可以识别图像中用户关注的第一区域，对第一区域和第二区域进行区分处理，提高用户关注的第一区域的图像处理效果。
本公开的又一实施例提供一种电子设备，包括存储器、处理器以及存储在存储器上并可在处理器上运行的图像处理程序。处理器在执行图像处理程序时，实现前述RAW图像的处理方法。
上述电子设备，通过前述RAW图像的处理方法，可以识别图像中用户关注的第一区域，对第一区域和第二区域进行区分处理，提高用户关注的第一区域的图像处理效果。
需要说明的是，在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行 系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(电子装置)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。
应当理解，本公开的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本公开的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本公开的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。
在本公开中，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”、“固定”等术语应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或成一体；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通或两个元件的相互作用关系，除非另有明确的限定。对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本公开中的具体含义。
尽管上面已经示出和描述了本公开的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本公开的限制，本领域的普通技术人员在本公开的范围内可以对上述实施例 进行变化、修改、替换和变型。

Claims (12)

  1. 一种RAW图像的处理方法，包括：
    识别单帧RAW图像的第一区域和第二区域；
    对所述第一区域和所述第二区域进行分块处理，其中，对所述第一区域和所述第二区域的分块处理基于不同的分块处理策略；
    根据分块处理结果分别对所述第一区域和所述第二区域进行LUT计算，获得LUT结果；
    根据所述LUT结果分别对所述第一区域和所述第二区域进行LSC处理，获得当前帧图像。
  2. 根据权利要求1所述的RAW图像的处理方法，其中，所述识别单帧RAW图像的第一区域和第二区域，包括：
    根据焦平面数据对所述单帧RAW图像进行区域划分，以获得所述第一区域和所述第二区域，其中，所述第一区域为对焦区域，所述第二区域为所述第一区域以外的图像区域。
  3. 根据权利要求1所述的RAW图像的处理方法，其中，所述识别单帧RAW图像的第一区域和第二区域，包括：
    对所述单帧RAW图像进行内容分析以获得所述单帧RAW图像的主体区域和背景区域，其中，所述主体区域为所述第一区域，所述背景区域为所述第二区域。
  4. 根据权利要求1所述的RAW图像的处理方法，其中，所述识别单帧RAW图像的第一区域和第二区域，包括：
    获取所述单帧RAW图像的前N帧历史帧RAW图像，其中N为大于1的整数；
    获取所述单帧RAW图像相对于所述历史帧RAW图像发生变化的区域，其中，发生变化的区域为所述第一区域，未发生变化的区域为所述第二区域。
  5. 根据权利要求1-4中任一项所述的RAW图像的处理方法，其中，对所述第一区域和所述第二区域进行分块处理，包括：
    基于所述第一区域和所述第二区域的不同图像块密度对所述RAW图像进行分块处理，其中，所述第一区域的图像块密度大于所述第二区域的图像块密度。
  6. 根据权利要求5所述的RAW图像的处理方法，其中，基于所述第一区域和所述第二区域的不同图像块密度对所述RAW图像进行分块处理，包括：
    沿像素行方向设置多条横向分割线，并沿像素列方向设置多条纵向分割线，其中，所述横向分割线和所述纵向分割线将所述RAW图像分割为多个图像块；
    配置贯穿所述第一区域的所述横向分割线之间的距离和所述纵向分割线之间的距离，以 使所述第一区域内的图像块密度大于等于第一预设密度值；
    配置贯穿所述第二区域的所述横向分割线之间的距离和所述纵向分割线之间的距离，以使所述第二区域内的图像块密度小于第一预设密度值。
  7. 根据权利要求6所述的RAW图像的处理方法，其中，基于所述第一区域和所述第二区域的不同图像块密度对所述单帧RAW图像进行分块处理时的图像块数量小于等于采用所述第一预设密度值对所述单帧RAW图像进行均匀分块处理时的图像块数量。
  8. 根据权利要求1-4中任一项所述的RAW图像的处理方法，其中，对所述第一区域和所述第二区域进行分块处理，包括：
    对所述单帧RAW图像进行均匀分块，获取多个原始图像块；
    根据每个所述原始图像块上的像素点到焦平面的距离将所述单帧RAW图像转换为曲面块图像，其中，所述第一区域内的原始图像块中的像素点到所述焦平面的距离小于所述第二区域内的原始图像块中的像素点到所述焦平面的距离；
    将所述曲面块图像沿垂直于所述焦平面的方向进行投影，获得投影图像块，并将所述投影图像块作为所述分块处理结果。
  9. 根据权利要求1所述的RAW图像的处理方法，其中，在根据所述LUT结果分别对所述第一区域和所述第二区域进行LSC处理后，还包括：
    根据所述当前帧图像获取与所述当前帧图像相关联的历史帧图像；
    根据所述历史帧图像，对所述当前帧图像中与所述历史帧图像相同的区域进行修正，以获得修正后的当前帧图像。
  10. 根据权利要求9所述的RAW图像的处理方法，其中，还包括：
    对所述修正后的当前帧图像进行分割，并对分割结果进行缓存。
  11. 一种RAW图像的处理芯片，包括：
    第一芯片，所述第一芯片用于识别单帧raw图像的第一区域和第二区域；对所述第一区域和所述第二区域进行分块处理，其中，对所述第一区域和所述第二区域的分块处理基于不同的分块处理策略，并根据所分块处理结果分别对所述第一区域和所述第二区域进行LUT计算，获得LUT结果；
    第二芯片，所述第二芯片与所述第一芯片相连，所述第二芯片用于根据所述LUT结果分别对所述第一区域和所述第二区域进行LSC处理，获得当前帧图像。
  12. 一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的图像处理程序，所述处理器执行所述图像处理程序时，实现如权利要求1-10中任一项所述的RAW图像的处理方法。
PCT/CN2022/072485 2021-03-26 2022-01-18 Raw图像的处理方法、芯片和电子设备 WO2022199236A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110328651.3A CN115131271A (zh) 2021-03-26 2021-03-26 Raw图像的处理方法、芯片和电子设备
CN202110328651.3 2021-03-26

Publications (1)

Publication Number Publication Date
WO2022199236A1 true WO2022199236A1 (zh) 2022-09-29

Family

ID=83374936

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/072485 WO2022199236A1 (zh) 2021-03-26 2022-01-18 Raw图像的处理方法、芯片和电子设备

Country Status (2)

Country Link
CN (1) CN115131271A (zh)
WO (1) WO2022199236A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117011179A (zh) * 2023-08-09 2023-11-07 北京精英路通科技有限公司 一种图像转换方法、装置、电子设备及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070285552A1 (en) * 2006-06-05 2007-12-13 Core Logic Inc. Lens shading correction device and method in image sensor
US20150097969A1 (en) * 2013-10-03 2015-04-09 Amlogic Co., Ltd. Methods and Systems for Lens Shading Correction
CN106506905A (zh) * 2016-10-20 2017-03-15 湖南国科微电子股份有限公司 镜头阴影校正方法
US20170078596A1 (en) * 2015-09-14 2017-03-16 Apical Ltd. Adaptive shading correction
CN107590840A (zh) * 2017-09-21 2018-01-16 长沙全度影像科技有限公司 基于网格划分的颜色阴影校正方法及其校正系统
CN107613192A (zh) * 2017-08-09 2018-01-19 深圳市巨龙创视科技有限公司 一种基于摄像机模组的数字图像处理算法
CN111818239A (zh) * 2020-03-12 2020-10-23 成都微光集电科技有限公司 一种图像传感器中镜头阴影校正方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070285552A1 (en) * 2006-06-05 2007-12-13 Core Logic Inc. Lens shading correction device and method in image sensor
US20150097969A1 (en) * 2013-10-03 2015-04-09 Amlogic Co., Ltd. Methods and Systems for Lens Shading Correction
US20170078596A1 (en) * 2015-09-14 2017-03-16 Apical Ltd. Adaptive shading correction
CN106506905A (zh) * 2016-10-20 2017-03-15 湖南国科微电子股份有限公司 镜头阴影校正方法
CN107613192A (zh) * 2017-08-09 2018-01-19 深圳市巨龙创视科技有限公司 一种基于摄像机模组的数字图像处理算法
CN107590840A (zh) * 2017-09-21 2018-01-16 长沙全度影像科技有限公司 基于网格划分的颜色阴影校正方法及其校正系统
CN111818239A (zh) * 2020-03-12 2020-10-23 成都微光集电科技有限公司 一种图像传感器中镜头阴影校正方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117011179A (zh) * 2023-08-09 2023-11-07 北京精英路通科技有限公司 一种图像转换方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN115131271A (zh) 2022-09-30

Similar Documents

Publication Publication Date Title
US11457138B2 (en) Method and device for image processing, method for training object detection model
US11178324B2 (en) Focusing method and device, electronic device and computer-readable storage medium
WO2019085792A1 (en) Image processing method and device, readable storage medium and electronic device
US8306262B2 (en) Face tracking method for electronic camera device
US8605955B2 (en) Methods and apparatuses for half-face detection
US11205278B2 (en) Depth image processing method and apparatus, and electronic device
JP4539597B2 (ja) 画像処理装置、画像処理方法および撮像装置
US20050276481A1 (en) Particular-region detection method and apparatus, and program therefor
CN108537782B (zh) 一种基于轮廓提取的建筑物图像匹配与融合的方法
US7868915B2 (en) Photographing apparatus, method and computer program product
US20060280362A1 (en) Image processing apparatus, image processing method, computer program, and storage medium
WO2019105297A1 (zh) 图像虚化处理方法、装置、移动设备及存储介质
JP7024736B2 (ja) 画像処理装置、および画像処理方法、並びにプログラム
JP2005303991A (ja) 撮像装置、撮像方法、及び撮像プログラム
CN111784620B (zh) 空间信息引导角度信息的光场相机全聚焦图像融合算法
EP2238744A1 (en) Automatic redeye detection
WO2019105298A1 (zh) 图像虚化处理方法、装置、移动设备及存储介质
CN113674303B (zh) 图像处理方法、装置、电子设备及存储介质
WO2022199236A1 (zh) Raw图像的处理方法、芯片和电子设备
Rahman et al. Real-time face-priority auto focus for digital and cell-phone cameras
CN107277348A (zh) 对焦方法、装置、计算机可读存储介质和移动终端
JP4348028B2 (ja) 画像処理方法、画像処理装置、撮像装置及びコンピュータプログラム
US9077926B2 (en) Image processing method and image processing apparatus
JP4039858B2 (ja) 画像マッチング方法と装置、および画像符号化方法と装置
US20150186373A1 (en) Method for sorting a group of images of a database and method for color correcting an image, corresponding devices, computer program and non-transitory computer readable medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22773898

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22773898

Country of ref document: EP

Kind code of ref document: A1