WO2023056926A1

WO2023056926A1 - Image processing method, electronic device, and storage medium

Info

Publication number: WO2023056926A1
Application number: PCT/CN2022/123779
Authority: WO
Inventors: 陈士奇; 周骥; 冯歆鹏
Original assignee: 上海肇观电子科技有限公司
Priority date: 2021-10-08
Filing date: 2022-10-08
Publication date: 2023-04-13
Also published as: CN113808222A

Abstract

The present disclosure provides an image processing method, an electronic device, and a storage medium. The image processing method comprises: obtaining a first image and a second image that are image captured for the same scene, a first exposure amount of the first image being greater than a second exposure amount of the second image, and the first image and the second image having the same image area; on the basis of a first response curve corresponding to the first exposure amount and a pixel value of at least one first pixel in the first image, determining a grayscale value of a first object in a scene corresponding to the pixel position of the at least one first pixel in the first image; on the basis of a second response curve corresponding to the second exposure amount and a pixel value of at least one second pixel in the second image, determining a grayscale value of a second object in a scene corresponding to the pixel position of the at least one second pixel in the second image; and determining a target image on the basis of the grayscale value of the first object and the grayscale value of the second object.

Description

Image processing method, electronic device and storage medium

technical field

The present disclosure relates to the technical field of image processing, and in particular to an image processing method, a method for calibrating a response curve of an image sensor, an electronic device, and a storage medium.

Background technique

The dynamic range (Dynamic Range, DR) of the brightness of the real scene is usually greater than the dynamic range of the camera. When the camera captures images, one exposure can only record a part of the dynamic range of the real scene, so there are often overexposed areas or underexposed areas in the captured images, resulting in the loss of scene details and poor visual effects of the images .

The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any approaches described in this section are admitted to be prior art solely by virtue of their inclusion in this section. Similarly, issues mentioned in this section should not be considered to have been recognized in any prior art unless otherwise indicated.

Contents of the invention

The present disclosure provides an image processing method, an electronic device and a storage medium, so as to realize high-quality and efficient image fusion.

According to an aspect of the present disclosure, there is provided an image processing method, including: acquiring a first image and a second image taken for the same scene, wherein the first exposure of the first image is greater than the second exposure of the second image exposure, the first image and the second image have the same image area; based on the first response curve corresponding to the first exposure and the pixel value of at least one first pixel in the first image, determining a grayscale value of a first object in the scene corresponding to a pixel position of at least one first pixel in the first image; based on a second response curve corresponding to the second exposure and in the second image a pixel value of at least one second pixel, determining a gray scale value of a second object in the scene corresponding to the pixel position of the at least one second pixel in the second image; and based on the gray scale value of the first object and The grayscale value of the second object determines the target image.

According to another aspect of the present disclosure, there is provided a method for calibrating a response curve of an image sensor, comprising: acquiring a first calibration image including a plurality of calibration grayscales in a first calibration grayscale group with a first calibration exposure; based on In the first calibration image, pixel values at multiple first calibration positions respectively corresponding to multiple calibration grayscales in the first calibration grayscale group determine a first calibration curve, and the first calibration curve indicates the The mapping relationship between the pixel values collected by the image sensor under the first calibration exposure and the grayscale values of multiple calibration grayscales in the first calibration grayscale group; different from the first calibration exposure Acquiring a second calibration image including a plurality of calibration grayscales in the second calibration grayscale group based on the second calibration exposure; The pixel values at multiple second calibration positions of the gray scale determine a second calibration curve, and the second calibration curve indicates that the pixel values collected by the image sensor under the second calibration exposure are different from those in the second calibration gray scale group. The mapping relationship between the gray scale values of multiple calibration gray scales; and determining the response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the response curve indicates the image sensor collected between the pixel value and the exposure amount used when capturing the image, and the grayscale values within the grayscale range of multiple calibrated grayscales including at least one of the first calibrated grayscale group and the second calibrated grayscale group mapping relationship.

According to another aspect of the present disclosure, an electronic device is provided. The electronic device comprises: a processor; and a memory storing a program comprising instructions which, when executed by the processor, cause the processor to perform the method according to the above.

According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a program is provided. The program includes instructions which, when executed by a processor of the electronic device, cause the electronic device to perform the method according to the above.

According to another aspect of the present disclosure, a computer program product is provided. The computer program product comprises a computer program which, when executed by a processor, implements the method described above.

According to the embodiments of the present disclosure, by using the response curve to determine the grayscale value corresponding to the pixel value of the image in the real environment under the current exposure, the real brightness information of the object in the image can be obtained conveniently. With response curves corresponding to different exposure levels, all dynamic ranges in the scene can be covered, enabling images with high dynamic range to be acquired.

These and other aspects of the disclosure will be apparent from and will be elucidated with reference to the embodiments described hereinafter.

Description of drawings

The drawings exemplarily illustrate the embodiment and constitute a part of the specification, and together with the text description of the specification, serve to explain the exemplary implementation of the embodiment. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, like reference numbers designate similar, but not necessarily identical, elements.

FIG. 1 shows a flowchart of an exemplary process of an image processing method according to an embodiment of the present disclosure;

Figure 2A shows an example of a first response curve and a second response curve according to an embodiment of the present disclosure;

Figure 2B shows an example of first and second response curves and inverse mapping curves according to an embodiment of the present disclosure;

FIG. 3 shows a flowchart of an exemplary process of a method for calibrating a response curve according to an embodiment of the present disclosure;

Figure 4A shows an example of a calibration curve according to an embodiment of the present disclosure;

Figure 4B shows an example of an extended calibration curve according to an embodiment of the present disclosure;

FIG. 5A shows an example of an image processing process according to an embodiment of the present disclosure;

5B shows an example of an image captured by an image sensor according to an embodiment of the present disclosure;

FIG. 5C shows an example of a target image according to an example of the present disclosure; and

FIG. 6 is a block diagram illustrating an example of an electronic device according to an exemplary embodiment of the present disclosure.

Detailed ways

In the present disclosure, unless otherwise stated, using the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship or importance relationship of these elements, and such terms are only used for Distinguishes one element from another. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on contextual description.

The terminology used in describing the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, there may be one or more elements. In addition, the term "and/or" used in the present disclosure covers any one and all possible combinations of the listed items.

In related art, the Debevec method can be utilized to recover high dynamic range radiance maps from photos taken by image sensors based on the physical properties of the imaging system (ie, photochemical and electronic reciprocity). In this method, multiple photos of the scene are taken with different exposures. The algorithm uses these differently exposed photographs to recover the response function of the imaging process, in terms of scaling factors, using the assumption of reciprocity. Using a known response function, Debevec's method fuses multiple photos into a single high dynamic range luminance map with pixel values proportional to the true luminance radiance values in the scene.

As with film, the most pronounced non-linearity in the response curve is at the saturation point, where any pixel with irradiance above a certain level maps to the same maximum image value, so the dynamic range of a single picture is limited. To obtain a high dynamic range response curve, the range of luminance radiance values of interest can be selected and an appropriate exposure time determined. Cover the entire dynamic range in a scene by taking and blending a series of photos with different exposures.

Using the Debevec method, it is possible to obtain a response curve describing the relationship between the pixel values in the image and the exposure value with which the image was acquired, from a series of differently exposed photographs of the scene. Therefore, based on the Debevec method, the real brightness of the scene is not introduced to calibrate the response curve, and the relative brightness between pixels can only be estimated. When the picture is too dark or too bright, the estimated relative brightness also has a large error compared with the real brightness. Furthermore, the Debevec method does not take into account the parameter of exposure gain of current complementary metal-oxide-semiconductor (CMOS) sensors.

In order to solve the above-mentioned problems in the related art, the present disclosure provides a new method for acquiring a target image with a high dynamic range by using a response curve. Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

FIG. 1 shows a flowchart of an exemplary process of an image processing method according to an embodiment of the present disclosure.

As shown in FIG. 1, in step S102, a first image and a second image taken for the same scene may be acquired. Wherein, the first exposure amount of the first image may be greater than the second exposure amount of the second image, and the first image and the second image may have the same image area.

In step S104, based on the first response curve corresponding to the first exposure and the pixel value of at least one first pixel in the first image, the scene corresponding to the pixel position of at least one first pixel in the first image can be determined The grayscale value of the first object.

In step S106, based on the second response curve corresponding to the second exposure and the pixel value of at least one second pixel in the second image, the scene corresponding to the pixel position of at least one second pixel in the second image can be determined The grayscale value of the second object.

In step S108, the target image may be determined based on the grayscale value of the first object and the grayscale value of the second object.

Using the image processing method provided by the embodiments of the present disclosure, the response curves corresponding to different exposures can be used to determine the real brightness of the object corresponding to each pixel in the scene in the image obtained based on different exposures, so that the scene can be further utilized. The true brightness of the object in the scene determines a target image containing the object in the scene, wherein the dynamic range of the target image is greater than that of the first image and the second image.

Each step of the method 100 is described in detail below.

In step S102, the first image and the second image taken for the same scene may be acquired. Wherein, the first exposure amount of the first image may be greater than the second exposure amount of the second image, and the first image and the second image may have the same image area.

In some embodiments, nearly identical objects are included in the first image and the second image. Since the first exposure used when acquiring the first image is different from the second exposure used when acquiring the second image, the first image and the second image have different dynamic ranges. In the case where the first exposure amount is greater than the second exposure amount, for an area with higher real brightness in the captured scene, this area may be overexposed in the image acquired by using the first exposure amount.

In some implementations, the first exposure amount may be determined based on a first exposure time used when acquiring the first image, and the second exposure amount may be determined based on a second exposure time used when acquiring the second image. In other implementations, the first exposure amount may be determined based on the first exposure time and the first exposure gain used when acquiring the first image, and the second exposure amount is determined based on the second exposure amount used when acquiring the second image Time and the second exposure gain are determined. For example, the first exposure amount may be the product of the first exposure time and the first exposure gain, and the second exposure amount may be the product of the second exposure time and the second exposure gain. By controlling the exposure time and exposure gain of the image sensor, the exposure used by the image sensor when acquiring images can be adjusted, so that the first image and the second image with different dynamic ranges can be acquired.

Wherein, the first response curve used in step S104 may correspond to the pixel value of at least one first pixel in the first image at the first exposure amount indicated by the calibration response curve of the image sensor and the pixel value of at least one first pixel in the scene corresponding to the first pixel. A first mapping relationship between grayscale values of the first object. The second response curve used in step S106 corresponds to the pixel value of at least one second pixel in the second image at the second exposure indicated by the calibration response curve and the gray scale of the second object in the scene corresponding to the second pixel A second mapping between values. Wherein, the gray scale value indicates the real brightness of the object in the scene.

Using the above-mentioned first response curve and second response curve, it is possible to determine the true brightness of objects in the captured scene based on the pixel values in the first image and the pixel values in the second image. For any pixel in the first image, the gray scale value for the pixel can be determined using the first response curve. Similarly, for any pixel in the second image, the grayscale value for that pixel can be determined using the second response curve. Since both the first response curve and the second response curve indicate the relationship between the pixel value and the real brightness of the object in the scene at the corresponding exposure, the first response curve can be used to determine the corresponding For the first grayscale image of all pixels, the second grayscale image corresponding to all pixels in the image area in the second image can be determined by using the second response curve. In the absence of overexposure, the first grayscale image and the second grayscale image should be the same, that is, both reflect the real brightness of objects in the image area in the scene.

However, if there is overexposure in the image, pixels above a certain brightness level all correspond to the same saturated pixel value in the image, so the response curve cannot be used to determine the grayscale value corresponding to the pixel in the overexposed area. In this case, grayscale values corresponding to pixels in the image area can be obtained by fusing grayscale images determined from images acquired with different exposures.

The grayscale value corresponding to the pixel in the overexposed area may be determined by using an image acquired with a smaller exposure amount (eg, the second image). In some embodiments, if overexposure also exists in the second image, the image information of the overexposure area in the second image may be acquired using a third image with a third exposure. Wherein the third exposure amount may be smaller than the second exposure amount. Without departing from the principles of embodiments of the present disclosure, the pixel value information and corresponding response curves of two or more images may be used to obtain the true brightness of an object in the scene corresponding to a pixel within an image region. In the present disclosure, the principle of the present disclosure is described by taking no overexposed area in the second image as an example, but those skilled in the art can understand that the embodiments of the present disclosure are not limited thereto.

Overexposed areas in the first image can be determined using a pixel threshold. For example, a set of pixels above a pixel threshold in the first image may be determined as an overexposed area. That is to say, a set of at least one first pixel in the first image with a pixel threshold value not higher than the pixel threshold forms a non-overexposed area in the first image. Wherein, the positions of the over-exposure area and the non-over-exposure area in the image area are different.

In some embodiments, at least one first pixel in the first image has a pixel value not higher than the pixel threshold, and at least one second pixel in the second image is located in a different position in the image area than the at least one first pixel in position in the image area. Wherein, the pixel threshold may be smaller than the saturation pixel value of the image sensor. The saturated pixel value of the image sensor may be determined based on the data bit width of the image sensor. For an image sensor with a data bit width of k bits, the saturated pixel value may be 2 to the k power. For example, for an image sensor with a data width of 12 bits, the saturated pixel value may be 2 ¹² =4096 (ie, 2 to the 12th power).

In some implementation manners, the aforementioned pixel threshold may be determined based on the saturated pixel value and the overexposure coefficient of the image sensor. In some examples, the overexposure factor may be a factor greater than zero and less than one. Thus the pixel threshold can be smaller than the saturation pixel value of the image sensor. In other implementation manners, the pixel threshold may also be any pixel value pre-specified by the user that is smaller than the saturation pixel value.

Using the grayscale value of the first object (corresponding to the non-overexposed area in the first image) and the grayscale value of the second object (corresponding to the overexposed area in the first image) determined in step S104 and step S106 can be Determine the pixel value for each pixel in the target image.

In some embodiments, the gray scale value of the first object and the gray scale value of the second object may be reverse mapped using an inverse mapping curve to obtain the pixel position of the first object in the target image and the pixel position of the second object Target pixel value. Wherein, the inverse mapping curve may indicate the mapping relationship between the target pixel value and the grayscale value at each pixel position in the target image. In the reverse mapping curve, the larger the grayscale value, the larger the target pixel value. The specific form of the inverse mapping curve is not limited here. In some other embodiments, any other data processing method can be used to process the grayscale value of the first object and the grayscale value of the second object to obtain image pixel values corresponding to each grayscale value, as long as the grayscale value The larger is the constraint condition that the target pixel value is larger.

For a color target image, the method provided by the present disclosure can be used to process the pixel values of the image of each color channel, so as to obtain the target pixel value of each color channel. A colored target image can be obtained by fusing the target pixel values of each color channel.

Figure 2A shows an example of a first response curve and a second response curve according to an embodiment of the disclosure.

As shown in FIG. 2A, the first response curve 201 indicates the mapping relationship between the image pixel value at the first exposure level and the grayscale value of the object in the scene, and the second response curve 202 indicates the image pixel value at the second exposure level. The mapping relationship with the grayscale value of objects in the scene.

The saturation cutoff point 203 indicates the cutoff point between the overexposed area and the non-overexposed area in the first image. In some examples, for an image sensor having a data width of 12 bits, the pixel value of the saturation cutoff point 203 may be a pixel threshold value slightly smaller than the saturation pixel value of 4096 of the image sensor. For example, the pixel threshold may be determined to be 4090 or any other suitable value.

It can be seen from FIG. 2A that within the range of pixel values not higher than the saturation cut-off point 203 , the first response curve 201 can well represent the mapping relationship between image pixel values and grayscale values. In the range of pixel values higher than the saturation cut-off point 203, the first response curve 201 begins to enter the saturation region, and the gray scale value corresponding to the pixel value higher than the pixel threshold cannot be determined by using the first response curve 201, because any Gray scale values above the saturation cut-off point 203 are all mapped to the same saturated pixel value.

Therefore, for the overexposed area in the first image, the grayscale value corresponding to the pixel in the overexposed area can be determined by using the second response curve 202 . It can be seen that since the second exposure amount is smaller than the first exposure amount, the second response curve enters the saturation region later than the first response curve. Pixels located within the overexposed region in the first image are not overexposed in the second image. Therefore, the grayscale value of the object in the scene corresponding to the pixel in the overexposed area can be determined using the second image and the second response curve 202 for the second image.

2B shows an example of first and second response curves and an inverse mapping curve according to an embodiment of the disclosure.

As shown in FIG. 2B , the curve 206 and the curve 207 correspond to the first response curve of the first image at the first exposure level and the second response curve of the second image at the second exposure level, respectively. Curve 204 corresponds to the inverse mapping curve of an embodiment of the present disclosure.

After using the first response curve 206 and the second response curve 207 to determine the grayscale value for each pixel in the image area, the inverse mapping curve 204 can be used to determine the pixel value of each pixel in the target image. The higher the grayscale value, the larger the pixel value of the pixel.

In some embodiments, as shown in FIG. 2B , at least a portion of the inverse mapping curve 204 may be determined based on a portion of the first response curve 206 . In the range where the gray scale value is smaller than the critical point 205, the inverse mapping curve and the first response curve 206 may overlap. Using this method, for an object whose gray scale value is smaller than the critical point 205, the pixel value in the target image obtained by the inverse mapping curve can be the same as the pixel value actually acquired by the image sensor, so that the pixel value close to the real acquisition can be obtained. Image way to generate part of the target image. For objects whose gray scale value is not less than the critical point 205, any mathematical tool can be used to generate the specific form of the inverse mapping curve, as long as the constraint condition that the higher the gray scale value is, the larger the pixel value of the pixel is satisfied. The inverse mapping curve can be implemented as a linear function, a polynomial function, or any possible monotonically increasing function. In this way, in the target image, objects with higher real brightness in the environment are also reflected in the image with higher pixel values, so that the target image can correctly reflect the brightness difference of objects in the real environment , and thus a target image with a larger dynamic range can be generated based on the first image and the second image.

In order to obtain the response curve describing the mapping relationship between the image pixel value and the gray scale value of the object in the environment involved in the embodiment described in conjunction with FIG. 1 , FIG. 2A and FIG. 2B , the present disclosure also provides a method for calibrating the image The sensor response curve method.

FIG. 3 shows a flowchart of an exemplary process of a method for calibrating a response curve according to an embodiment of the present disclosure.

As shown in FIG. 3 , in step S302 , a first calibration image including a plurality of calibration grayscales in the first calibration grayscale group may be acquired with a first calibration exposure.

In step S304, a first calibration curve may be determined based on pixel values at multiple first calibration positions in the first calibration image corresponding to multiple calibration grayscales in the first calibration grayscale group. Wherein, the first calibration curve may indicate the mapping relationship between the pixel values collected by the image sensor at the first calibration exposure and the grayscale values of the multiple calibration grayscales in the first calibration grayscale group.

In step S306, a second calibration image including a plurality of calibration grayscales in the second calibration grayscale group may be acquired with a second calibration exposure different from the first calibration exposure.

In step S308, a second calibration curve may be determined based on pixel values at multiple second calibration positions respectively corresponding to multiple calibration gray levels in the second calibration gray scale group in the second calibration image. Wherein, the second calibration curve may indicate the mapping relationship between the pixel value collected by the image sensor at the second calibration exposure and the grayscale values of multiple calibration grayscales in the second calibration grayscale group.

In step S310, fitting may be performed based on the first calibration curve and the second calibration curve to obtain a response curve of the image sensor. The response curve can indicate the mapping relationship between the pixel value collected by the image sensor, the exposure used when capturing the image, and the grayscale value in the grayscale range including multiple calibrated grayscales. Wherein, the above-mentioned grayscale range may include at least one of the first calibrated grayscale group and the second calibrated grayscale group.

Using the method for calibrating the response curve of the image sensor provided by the embodiments of the present disclosure, the difference between the calibration gray scale (that is, the real brightness in the environment) and the pixel value in the collected image can be determined based on the images collected under different exposures. The mapping relationship between them, so that the mapping relationship between the real brightness in the environment in the predetermined gray scale range and the pixel value in the collected image can be determined by fusing the image information under different exposures.

Each step of the method 300 is described in detail below.

In step S302, a first calibration image including a plurality of calibration grayscales in the first calibration grayscale group is acquired with a first calibration exposure. Wherein the first calibration image is the raw data (raw image) acquired by the image sensor.

In some embodiments, an image of a predetermined grayscale card may be acquired at a first calibration exposure as the first calibration image. Wherein, the predetermined gray scale card may include 20 gray scales that change with a predetermined gray scale variation. In some examples, the gray scales in the predetermined gray scale card may vary uniformly.

In step S304, a first calibration curve may be determined based on pixel values at multiple first calibration positions in the first calibration image corresponding to multiple calibration grayscales in the first calibration grayscale group.

Based on the first calibration image acquired in step S302, the pixel value in the first calibration image for each calibration gray scale can be calibrated. By determining each first calibration position corresponding to each calibration gray scale in the first calibration image and reading the pixel value at each first calibration position, the pixel value of each calibration gray scale in the first calibration image can be determined. The first calibration curve can be obtained by fitting the pixel values corresponding to each calibration gray scale, wherein the first calibration curve indicates the pixel value collected by the image sensor under the first calibration exposure and the multiple calibration values in the first calibration gray scale group The mapping relationship between the grayscale values of the grayscale.

In step S306, a second calibration image including a plurality of calibration grayscales in the second calibration grayscale group in the second calibration grayscale group may be acquired with a second calibration exposure different from the first calibration exposure. Wherein the second calibration image is the original data acquired by the image sensor.

In some implementations, the first calibration exposure and the second calibration exposure used in step S304 and step S306 may be determined based on the exposure time used when acquiring the image. Among them, taking a CMOS image at most 30 frames per second as an example, the longest exposure time can be 33ms. The first calibrated exposure amount may be greater than the second calibrated exposure amount, or may be smaller than the second calibrated exposure amount. In some exemplary calibration processes, the minimum calibration exposure time can be determined as 1 ms, and the exposure time used when acquiring images is gradually increased in increments of 1 ms. For example, the exposure time of the first calibration exposure amount may be 1 ms, and the exposure time of the second calibration exposure amount may be 2 ms. For another example, the exposure time of the first calibration exposure amount may be 20 ms, and the exposure time of the second calibration exposure amount may be 3 ms. The specific exposure time of the first calibration exposure amount and the second calibration exposure amount is not limited here, and those skilled in the art can select an appropriate exposure time according to the actual situation.

In some other implementation manners, the first calibration exposure amount and the second calibration exposure amount used in step S304 and step S306 may be based on the exposure time and exposure gain used when acquiring the image. For example, the first calibrated exposure amount may be the product of the first calibrated exposure time and the first calibrated exposure gain, and the second calibrated exposure amount may be the product of the second calibrated exposure time and the second calibrated exposure gain. In some exemplary calibration processes, under the condition that the exposure gain is kept at 1, the exposure used when acquiring the image can be increased by first increasing the exposure time. In the case of reaching the maximum exposure time, keep the exposure time constant while gradually changing the exposure gain to the maximum exposure gain.

Similar to step S304, each second calibration position corresponding to each calibration grayscale in the second calibration image can be determined and the pixel value at each second calibration position can be read, and it can be determined that each calibration grayscale is in the second calibration image pixel value. The second calibration curve can be obtained by fitting the pixel values corresponding to each calibration gray scale, wherein the second calibration curve indicates the pixel value collected by the image sensor under the second calibration exposure and the multiple calibration values in the second calibration gray scale group. The mapping relationship between the grayscale values of the grayscale.

In step S310, fitting may be performed based on the first calibration curve and the second calibration curve to obtain a response curve of the image sensor. The response curve may indicate the mapping relationship between the pixel values collected by the image sensor, the exposure used when capturing the image, and the grayscale values within the grayscale range including multiple calibrated grayscales.

In some embodiments, the multiple calibration gray scales in the first calibration gray scale group and the multiple calibration gray scales in the second calibration gray scale group may be the same. In some implementations, the first calibration grayscale group may include a plurality of calibration grayscales that start from a reference grayscale value with a predetermined grayscale variation, and the second calibration grayscale group includes a plurality of calibration grayscales that change from a reference grayscale value with the same predetermined Multiple calibrated grayscales with the same amount of grayscale variation. Since the calibration gray scales contained in the first calibration image and the second calibration image are the same, the first calibration curve and the second calibration curve can be used to determine the corresponding different pixel values.

In some other embodiments, the multiple calibration gray scales in the first calibration gray scale group and the multiple calibration gray scales in the second calibration gray scale group may be different. For example, the first calibration grayscale group may include a first number of multiple calibration grayscales that change with a first predetermined grayscale variation from the first reference grayscale value. The second calibration grayscale group may include a second number of multiple calibration grayscales varying by a second predetermined grayscale variation from the second reference grayscale value. The various first reference gray scale values and the second reference gray scale values may be the same or different. The first predetermined gray scale change amount and the second predetermined gray scale change amount may be the same or different. The first quantity and the second quantity may be the same or different.

In some embodiments, the grayscale range corresponding to the response curve may include multiple calibration grayscales in the first calibration grayscale group and/or multiple calibration grayscales in the second calibration grayscale group. In some implementation manners, the grayscale range corresponding to the response curve may be greater than the grayscale range of the first calibration grayscale group and/or the second calibration grayscale group. That is to say, the maximum grayscale value of the grayscale range corresponding to the response curve may be greater than the maximum standardized grayscale value in the first calibration grayscale group and the maximum calibration grayscale value in the second calibration grayscale group. In some examples, the gray scale range corresponding to the response curve may include 60 gray scales based on a predetermined gray scale change amount. In some other examples, the grayscale range corresponding to the response curve also includes any other number of grayscales based on the predetermined grayscale variation, as long as the grayscale range corresponding to the response curve is not smaller than the first calibration grayscale group and/or the second Just calibrate the gray scale range of the gray scale group.

In step S310, based on the first calibration curve and the second calibration curve determined in step S306 and step S308, based on the assumption of physical reciprocity (reciprocity) of the image sensor, the calibration curve obtained by translation measurement can be translated into The grayscale range covered by the calibration curve is extended, so that a calibration curve covering a grayscale range larger than the predetermined calibration grayscale group can be obtained without actually photographing a larger range of grayscale values.

Figure 4A shows an example of a calibration curve according to an embodiment of the present disclosure.

As shown in FIG. 4A , by continuously changing the exposure used by the image sensor to acquire images, a series of calibration curves indicating the mapping relationship between image pixel values and gray scale values can be obtained. Wherein, each curve shown in FIG. 4A is measured under different exposure amounts. For the same grayscale value, the higher the exposure, the higher the pixel value corresponding to the grayscale value in the image captured by the image sensor. In the example shown in Figure 4A. The exposure amount corresponding to the calibration curve 401 is greater than the exposure amount corresponding to the calibration curve 402 , and the exposure amount corresponding to the calibration curve 402 is greater than the exposure amount corresponding to the calibration curve 403 .

In the example shown in FIG. 4A , for the 20 gray scales that can be directly acquired in the calibration image, each calibration curve within the range of 20 gray scales can be directly obtained through the image captured by the image sensor.

In some embodiments, based on the assumption of the physical reciprocity of the image sensor, on the basis of each calibration curve obtained in FIG. 4A , the calibration curve can be theoretically extended by shifting the curve, so as to obtain a larger gray scale Calibration curves available in the range.

Figure 4B shows an example of an extended calibration curve according to an embodiment of the disclosure.

As shown in Figure 4B, for the low exposure curve, even when the brightest grayscale value in the calibration grayscale is 20, the pixel value corresponding to the grayscale 20 in the image is still low and the distance The saturation region is farther away. In this case, by shifting part of the higher exposure calibration curve and stitching it with the lower exposure calibration curve, you can get the calibration curve from gray scale 20 to higher gray at the lower exposure. order (up to the saturation region) curve.

In some embodiments, the first calibration curve can be used to extend the second calibration curve.

The maximum pixel value corresponding to the maximum calibrated grayscale value of the second calibrated grayscale group under the second calibrated exposure amount may be determined based on the second calibration curve. Based on the corresponding maximum pixel value on the second calibration curve, the curve portion of the first calibration curve higher than the maximum pixel value of the second calibration curve is spliced with the second calibration curve to obtain an adjusted second calibration curve, wherein the adjusted The second calibration curve indicates the mapping relationship between at least one grayscale value higher than the maximum calibrated grayscale value in the grayscale range and the pixel value collected by the image sensor under the second calibrated exposure. A response curve of the image sensor is determined based on the first calibration curve and the second calibration curve.

As shown in FIG. 4B , for the calibration curve 407, the maximum pixel value of the calibration curve 407 at the gray scale 20 can be determined, and the part of the curve on the calibration curve 404 that is greater than the maximum pixel value of the calibration curve 407 can be translated to the gray scale value 20 To form the calibration curve 404', by splicing the calibration curve 404' and the calibration curve 407 at the gray scale value 20, the calibration curve 407 can be extended to the gray scale range with a gray scale value greater than 20 and the saturation of the calibration curve 407 can be obtained area. Similarly, the calibration curve 405' and the calibration curve 408 obtained after the calibration curve 405 is shifted can be spliced at the gray scale value of 20, and the calibration curve 406' and the calibration curve 409 obtained after the calibration curve 406 is translated at the gray scale value of 20 splicing. According to the physical reciprocity of the image sensor, the first calibration curve used to extend the second calibration curve can be arbitrary, as long as the first calibration exposure corresponding to the first calibration curve is greater than the second calibration corresponding to the second calibration curve Exposure is fine. In some cases, more than two calibration curves corresponding to different exposures can also be spliced.

After each calibration curve corresponding to the gray scale range including the calibration gray scale is determined, each calibration curve within the gray scale range can be used for fitting to obtain a response curve of the image sensor. For example, the adjusted second calibration curve can be fitted with the first calibration curve to obtain the response curve of the image sensor.

In some embodiments, the response curve of the image sensor can be expressed as a function of the image pixel value with respect to the exposure when the image was captured and the gray scale value of the captured object. Multiple calibration curves including the first calibration curve and the adjusted second calibration curve can be used to fit the parameters required by the response curve.

The response curve of the image sensor can be represented by formula (1):

Among them, I represents the pixel value collected by the image sensor, S represents the grayscale value, E represents the exposure amount, and a, γ, b, δ, and w are constants.

FIG. 5A shows an example of an image processing procedure according to an embodiment of the present disclosure.

As shown in FIG. 5A , in step S501 , the image sensor 510 may be used to capture multiple calibration images with multiple calibration exposures. A calibration response curve 530 may be determined according to a predetermined relationship between calibration gray scales and image pixel values in a plurality of calibration images. The calibration response curve 530 indicates the mapping relationship between the pixel values captured by the image sensor, the exposure used when capturing the image, and the gray scale values within the gray scale range including multiple calibration gray scales.

In step S502 , the image sensor 510 may capture the first image 520 - 1 , the second image 520 - 2 . . . and the nth image 502 - n under different exposures, where n is an integer greater than 2.

In step S503 , the first image 520 - 1 , the second image 520 - 2 . . . and the nth image 502 - n can be processed respectively by using the calibration response curve 530 . According to the exposure amount corresponding to the first image 520-1, the second image 520-2 ... and the nth image 520-n and the pixel value of each pixel in the image, the calibration response curve 530 can be used to determine the -1, the second image 520-2...and the first brightness-mapped image 540-1 of the nth image 520-n, the second brightness-mapped image 540-2...and the n-th brightness-mapped image 540-n, wherein A luminance mapping image 540-1, a second luminance mapping image 540-2... and an n th luminance mapping image 540-n indicate the grayscale value of the object in the scene corresponding to each pixel in the image area, that is, the real luminance.

FIG. 5B shows an example of an image captured by an image sensor according to an embodiment of the present disclosure. It can be seen that there is an overexposed area in FIG. 5B , and image details in the overexposed area cannot be obtained through the original image.

In step S504, based on the pixel values of the first image 520-1, the second image 520-2... and the nth image 520-n, the first brightness mapping image 540-1, the second brightness mapping image 540-2 ... and the nth brightness map image 540 - n are fused to obtain a fused brightness map 550 . Among them, the processing can start from the image with the highest exposure. Taking the first image with the highest exposure as an example, the first brightness mapping image 540-1 corresponding to at least one first pixel (that is, the non-overexposed area) in the first image 520-1 that is not higher than the pixel threshold can be The grayscale value of is determined as the grayscale value of the corresponding pixel in the fused brightness map 550 . For the overexposed area in the first image 520-1, grayscale values of at least some of the pixels in the overexposed area may be determined using a brightness mapping image corresponding to an image with a smaller exposure (such as the second image). By analogy, until the grayscale values of all pixels in the fused brightness map 550 are determined.

In step S505 , the target image 560 may be determined by using the grayscale values of the pixels indicated in the fused brightness map 550 . Wherein, pixels with higher grayscale values in the fused brightness map 550 have higher pixel values in the target image.

FIG. 5C illustrates an example of a target image according to an example of the present disclosure. Wherein, the image shown in FIG. 5C is a high dynamic range target image determined based on the image shown in FIG. 5B . As shown in Figure 5C, by fusing information from other images, the image details in the overexposed area in Figure 5B are restored. Therefore, Fig. 5C has a better visual effect than Fig. 5B.

According to an embodiment of the present disclosure, an image processing device is also provided. The image processing device may include an image acquisition unit, a first grayscale determination unit, a second grayscale determination unit, and a target image determination unit, wherein the image acquisition unit may be configured to acquire the first image and the second image taken for the same scene, Wherein the first exposure amount of the first image is greater than the second exposure amount of the second image, and the first image and the second image have the same image area. The first grayscale determination unit may be configured to determine the pixel value corresponding to at least one first pixel in the first image based on the first response curve corresponding to the first exposure amount and the pixel value of at least one first pixel in the first image. The grayscale value of the first object in the scene at the pixel position of the first pixel. The second grayscale determination unit may be configured to determine the pixel value corresponding to at least one second pixel in the second image based on the second response curve corresponding to the second exposure amount and the pixel value of at least one second pixel in the second image. The grayscale value of the second object in the scene at the pixel position of the second pixel. The target image determining unit may be configured to determine the target image based on the grayscale value of the first object and the grayscale value of the second object.

Here, the operations of the above units of the image processing device are similar to the operations of steps S102-S108 described above, and will not be repeated here.

According to an embodiment of the present disclosure, a device for calibrating a response curve of an image sensor is also provided. The apparatus for calibrating a response curve of an image sensor may include a first calibration image acquisition unit, a first calibration curve determination unit, a second calibration image acquisition unit, a second calibration curve determination unit, and a response curve determination unit. Wherein, the first calibration image acquiring unit may be configured to acquire a first calibration image including a plurality of calibration grayscales in the first calibration grayscale group with a first calibration exposure. The first calibration curve determining unit may be configured to determine the first calibration based on pixel values at multiple first calibration positions in the first calibration image respectively corresponding to multiple calibration grayscales in the first calibration grayscale group Curve, the first calibration curve indicates the mapping relationship between the pixel value collected by the image sensor under the first calibration exposure and the grayscale values of multiple calibration grayscales in the first calibration grayscale group. The second calibration image acquisition unit may be configured to acquire a second calibration image including a plurality of calibration grayscales in the second calibration grayscale group with a second calibration exposure different from the first calibration exposure. The second calibration curve determination unit may be configured to determine the second calibration based on pixel values at multiple second calibration positions in the second calibration image corresponding to multiple calibration grayscales in the second calibration grayscale group. Curve, the second calibration curve indicates the mapping relationship between the pixel value collected by the image sensor at the second calibration exposure and the grayscale values of multiple calibration grayscales in the second calibration grayscale group. The response curve determination unit may be configured to determine the response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the response curve indicates the pixel values captured by the image sensor and the exposure used when capturing the image and a mapping relationship between grayscale values within a grayscale range of multiple calibrated grayscales including at least one of the first calibrated grayscale group and the second calibrated grayscale group.

Here, the operations of the above units of the apparatus for calibrating the response curve of the image sensor are similar to the operations of steps S302-S310 described above, and will not be repeated here.

According to another aspect of the present disclosure, there is also provided an electronic device, including: a processor; and a memory storing a program, the program including instructions, which when executed by the processor cause the processor to perform the above-mentioned Methods.

According to another aspect of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing a program, the program includes instructions, and the instructions, when executed by a processor of an electronic device, cause the electronic device to perform the above-mentioned Methods.

According to another aspect of the present disclosure, there is also provided a computer program product, including a computer program, and when the computer program is executed by a processor, the above method is implemented.

Referring to FIG. 6 , an electronic device 600 will now be described, which is an example of a hardware device (electronic device) that can be applied to aspects of the present disclosure. Electronic device 600 may be any machine configured to perform processing and/or computation, which may be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal digital assistant, robot, smartphone, vehicle-mounted computer, or any combination thereof. The above-mentioned image processing method 100 and the method 300 for calibrating the response curve may be fully or at least partially implemented by the electronic device 600 or similar devices or systems.

Electronic device 600 may include elements connected to or in communication with bus 602 (possibly via one or more interfaces). For example, electronic device 600 may include a bus 602 , one or more processors 604 , one or more input devices 606 , and one or more output devices 608 . Processor(s) 604 may be any type of processor and may include, but is not limited to, one or more general purpose processors and/or one or more special purpose processors (eg, special processing chips). The input device 606 may be any type of device capable of inputting information into the electronic device 600, and may include, but is not limited to, a mouse, keyboard, touch screen, microphone, and/or remote control. Output devices 608 may be any type of device capable of presenting information, and may include, but are not limited to, displays, speakers, video/audio output terminals, vibrators, and/or printers. The electronic device 600 may also include a non-transitory storage device 610, which may be any storage device that is non-transitory and that enables data storage, including but not limited to disk drives, optical storage devices, solid-state memory, floppy disks, flexible disk, hard disk, tape or any other magnetic medium, optical disk or any other optical medium, ROM (read only memory), RAM (random access memory), cache memory and/or any other memory chips or cartridges, and/or computer Any other medium from which data, instructions and/or code can be read. The non-transitory storage device 610 is detachable from the interface. The non-transitory storage device 610 may have data/programs (including instructions)/codes for implementing the above methods and steps. The electronic device 600 may also include a communication device 612 . The communication device 612 may be any type of device or system that enables communication with external devices and/or with a network, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication devices, and/or chipsets, such as Bluetooth ^™ device, 1302.11 device, Wi-Fi device, Wi-Max device, cellular communication device, and/or the like.

Electronic device 600 may also include working memory 614, which may be any type of working memory that may store programs (including instructions) and/or data useful for the work of processor 604, and may include, but is not limited to, random access memory and and/or read-only memory devices.

Software elements (programs) may be located in working memory 614, including but not limited to operating system 616, one or more application programs 618, drivers, and/or other data and code. Instructions for performing the above methods and steps may be included in one or more application programs 618, and the above image processing method 100 and the method 300 for calibrating response curves may be read and executed by the processor 604 by one or more The instructions of an application program 618 are implemented. More specifically, in the above-mentioned image processing method 100, steps S102-S108 can be implemented, for example, by the processor 604 executing the application program 618 having the instructions of steps S102-S108. In the above-mentioned method 300 for calibrating the response curve, steps S302-S310 can be realized, for example, by the processor 604 executing the application program 618 having the instructions of steps S302-S310. In addition, other steps in the above-mentioned image processing method 100 and the method 300 for calibrating response curves can be realized, for example, by the processor 604 executing the application program 618 having instructions for executing the corresponding steps. The executable code or source code of the instructions of the software elements (programs) may be stored in a non-transitory computer-readable storage medium (such as the above-mentioned storage device 610), and when executed, may be stored in the working memory 614 (possibly compiled and/or install). The executable code or source code of the instructions of the software element (program) may also be downloaded from a remote location.

It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware could also be used, and/or particular elements could be implemented in hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. For example, some or all of the disclosed methods and devices can be implemented by programming hardware (e.g., including Field Programmable Gate Arrays) in assembly language or hardware programming languages (such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure. (FPGA) and/or Programmable Logic Circuits of Programmable Logic Array (PLA)) to implement programming.

It should also be understood that the aforementioned methods can be implemented in a server-client mode. For example, a client may receive user-entered data and send the data to a server. The client may also receive the data input by the user, perform part of the processing in the aforementioned method, and send the processed data to the server. The server may receive data from the client, execute the aforementioned method or another part of the aforementioned method, and return the execution result to the client. The client can receive the execution result of the method from the server, and can present it to the user, for example, through an output device.

It should also be understood that components of electronic device 600 may be distributed across a network. For example, some processing may be performed using one processor while other processing may be performed by another processor remote from the one processor. Other components of computing system 600 may be similarly distributed. As such, electronic device 600 may be interpreted as a distributed computing system that performs processing at multiple locations.

Some exemplary aspects of the disclosure are described below.

Aspect 1. An image processing method, comprising:

acquiring a first image and a second image taken for the same scene, wherein the first exposure of the first image is greater than the second exposure of the second image, and the first image and the second image have the same image area;

Based on the first response curve corresponding to the first exposure and the pixel value of the at least one first pixel in the first image, determine the scene corresponding to the pixel position of the at least one first pixel in the first image The grayscale value of the first object of ;

Based on the second response curve corresponding to the second exposure and the pixel value of at least one second pixel in the second image, determine the scene corresponding to the pixel position of at least one second pixel in the second image The grayscale value of the second object of ; and

A target image is determined based on the gray scale value of the first object and the gray scale value of the second object.

Aspect 2. The image processing method according to aspect 1, wherein at least one first pixel in the first image has a pixel value not higher than a pixel threshold, and at least one second pixel in the second image is at The location in the image area is different than the location of the at least one first pixel in the image area.

Aspect 3. The image processing method according to aspect 2, wherein the pixel threshold is smaller than a saturation pixel value of the image sensor.

Aspect 4. The image processing method according to aspect 3, wherein the pixel threshold is determined based on a saturated pixel value of the image sensor and an overexposure coefficient, and the overexposure coefficient is a coefficient greater than 0 and less than 1.

Aspect 5. The image processing method according to aspect 1, wherein determining the target image based on the grayscale value of the first object and the grayscale value of the second object comprises:

Using an inverse mapping curve to perform inverse mapping on the grayscale value of the first object and the grayscale value of the second object, so as to obtain the pixel position of the first object and the pixel position of the second object in the target image A target pixel value at a pixel position, wherein the inverse mapping curve indicates a mapping relationship between the target pixel value at each pixel position in the target image and the gray scale value.

Aspect 6. The image processing method according to aspect 5, in the inverse mapping curve, the larger the gray scale value is, the larger the target pixel value is.

Aspect 7. The image processing method according to aspect 1, wherein the first exposure amount is determined based on the first exposure time and the first exposure gain used when acquiring the first image, and the second exposure The amount is determined based on a second exposure time and a second exposure gain used when acquiring the second image.

Aspect 8. The image processing method according to aspect 1, wherein the first response curve corresponds to at least one first pixel in the first image at the first exposure amount indicated by a calibration response curve of the image sensor The first mapping relationship between the pixel value of the first pixel and the grayscale value of the first object in the scene corresponding to the first pixel, the second response curve corresponds to the second response curve indicated by the calibration response curve in the second A second mapping relationship between the pixel value of at least one second pixel in the second image and the grayscale value of a second object in the scene corresponding to the second pixel under the exposure.

Aspect 9. The image processing method according to aspect 8, wherein the calibration response curve is determined by the following steps:

Acquiring a first calibration image including a plurality of calibration grayscales in the first calibration grayscale group with a first calibration exposure;

A first calibration curve is determined based on pixel values at multiple first calibration positions in the first calibration image corresponding to multiple calibration grayscales in the first calibration grayscale group, and the first calibration curve indicates The mapping relationship between the pixel values collected by the image sensor under the first calibration exposure and the grayscale values of multiple calibration grayscales in the first calibration grayscale group;

acquiring a second calibration image including a plurality of calibration grayscales in a second calibration grayscale group with a second calibration exposure different from the first calibration exposure;

Determine a second calibration curve based on pixel values at multiple second calibration positions in the second calibration image corresponding to multiple calibration grayscales in the second calibration grayscale group, the second calibration curve indicates A mapping relationship between the pixel values collected by the image sensor under the second calibration exposure and the grayscale values of multiple calibration grayscales in the second calibration grayscale group; and

Determining a calibration response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the calibration response curve indicates the pixel value collected by the image sensor and the exposure used when capturing the image and includes the first calibration curve A mapping relationship between grayscale values within a grayscale range of a plurality of calibrated grayscales in a calibrated grayscale group and at least one of the second calibrated grayscale group.

Aspect 10. The image processing method according to aspect 9, wherein determining the calibration response curve of the image sensor based on the first calibration curve and the second calibration curve comprises:

Based on the second calibration curve, determine the maximum pixel value corresponding to the maximum calibration gray scale value among the multiple calibration gray scales in the second calibration gray scale group under the second calibration exposure;

Based on the maximum pixel value, splicing the curve part of the first calibration curve higher than the maximum pixel value of the second calibration curve with the second calibration curve to obtain an adjusted second calibration curve, wherein the The adjusted second calibration curve indicates a mapping relationship between at least one grayscale value higher than the maximum calibrated grayscale value in the grayscale range and pixel values collected by the image sensor under a second calibrated exposure; and

Fitting the adjusted second calibration curve to the first calibration curve to obtain the calibration response curve.

Aspect 11. The image processing method according to aspect 9, wherein the second calibrated exposure amount is smaller than the first calibrated exposure amount.

Aspect 12. The image processing method according to aspect 9, wherein the first calibration grayscale group includes a plurality of calibration grayscales that change with a predetermined grayscale variation from the reference grayscale value, and the second calibration grayscale The group includes the same number of calibration grayscales varying by the same predetermined grayscale variation from the reference grayscale value.

Aspect 13. The image processing method according to any one of aspects 9-12, wherein the calibration response curve is represented by the following formula:

Aspect 14. A method for calibrating a response curve of an image sensor, comprising:

Determining a response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the response curve indicates the pixel values collected by the image sensor and the exposure used when capturing the image and includes the first calibration A mapping relationship between the gray scale group and the gray scale values within the gray scale range of at least one of the multiple calibration gray scales in the second calibration gray scale group.

Aspect 15. The method of aspect 14, wherein the exposure amount is determined based on an exposure time and an exposure gain used when acquiring an image with the image sensor.

Aspect 16. The method of aspect 14, wherein the second nominal exposure level is less than the first nominal exposure level.

Aspect 17. The method according to aspect 14, wherein the first set of calibration grayscales includes a plurality of calibration grayscales that change from a reference grayscale value by a predetermined amount of grayscale variation, and the second set of calibration grayscales includes A plurality of calibration gray scales of the same number varying with the same predetermined gray scale variation from the reference gray scale value.

Aspect 18. The method according to aspect 14, wherein the plurality of calibrated gray scales change with a predetermined gray scale variation, and the maximum gray scale value including the gray scale range is greater than the first set of calibrated gray scales and the maximum calibrated gray scale value of the second calibrated gray scale group.

Aspect 19. The method of aspect 14, wherein determining a response curve of the image sensor based on the first calibration curve and the second calibration curve comprises:

determining the maximum pixel value corresponding to the maximum calibrated grayscale value of the second calibrated grayscale group under the second calibrated exposure amount based on the second calibration curve;

Fitting the adjusted second calibration curve to the first calibration curve to obtain the response curve.

Aspect 20. The method of any one of aspects 14-19, wherein the response curve is represented by the following formula:

Aspect 21. An electronic device comprising:

processor; and

A memory storing a program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of aspects 1-20.

Aspect 22. A non-transitory computer-readable storage medium storing a program comprising instructions which, when executed by a processor of an electronic device, cause the electronic device to perform any of aspects 1-20. method described in the item.

Aspect 23. A computer program product comprising a computer program, wherein said computer program, when executed by a processor, implements the method according to any one of aspects 1-20.

Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but It is limited only by the appended claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by equivalent elements thereof. Also, steps may be performed in an order different from that described in the present disclosure. Further, various elements in the embodiments or examples can be combined in various ways. Importantly, as technology advances, many of the elements described herein may be replaced by equivalent elements appearing after this disclosure.

Claims

An image processing method, comprising:

acquiring a first image and a second image taken for the same scene, wherein the first exposure of the first image is greater than the second exposure of the second image, and the first image and the second image have the same image area;

Based on the first response curve corresponding to the first exposure and the pixel value of the at least one first pixel in the first image, determine the scene corresponding to the pixel position of the at least one first pixel in the first image The grayscale value of the first object of ;

Based on the second response curve corresponding to the second exposure and the pixel value of at least one second pixel in the second image, determine the scene corresponding to the pixel position of at least one second pixel in the second image The grayscale value of the second object of ; and

A target image is determined based on the gray scale value of the first object and the gray scale value of the second object.
The image processing method according to claim 1, wherein at least one first pixel in the first image has a pixel value not higher than a pixel threshold, and at least one second pixel in the second image is in the The position in the image area is different from the position of the at least one first pixel in the image area.
The image processing method according to claim 2, wherein the pixel threshold is smaller than a saturation pixel value of the image sensor.
The image processing method according to claim 3, wherein the pixel threshold is determined based on a saturated pixel value of the image sensor and an overexposure coefficient, and the overexposure coefficient is a coefficient greater than 0 and less than 1.
The image processing method according to claim 1, wherein determining the target image based on the grayscale value of the first object and the grayscale value of the second object comprises:

Using an inverse mapping curve to perform inverse mapping on the grayscale value of the first object and the grayscale value of the second object, so as to obtain the pixel position of the first object and the pixel position of the second object in the target image A target pixel value at a pixel position, wherein the inverse mapping curve indicates a mapping relationship between the target pixel value at each pixel position in the target image and the gray scale value.
The image processing method according to claim 5, wherein, in the inverse mapping curve, the larger the gray scale value is, the larger the target pixel value is.
The image processing method according to claim 1, wherein the first exposure amount is determined based on the first exposure time and the first exposure gain used when acquiring the first image, and the second exposure amount is Determined based on the second exposure time and the second exposure gain used when acquiring the second image.
The image processing method according to claim 1, wherein the first response curve corresponds to the pixel of at least one first pixel in the first image at the first exposure amount indicated by the calibration response curve of the image sensor value and the grayscale value of the first object in the scene corresponding to the first pixel, the second response curve corresponds to the calibration response curve indicated at the second exposure A second mapping relationship between the pixel value of at least one second pixel in the second image and the gray scale value of the second object in the scene corresponding to the second pixel is described below.
The image processing method according to claim 8, wherein said calibration response curve is determined by the following steps:

Acquiring a first calibration image including a plurality of calibration grayscales in the first calibration grayscale group with a first calibration exposure;

A first calibration curve is determined based on pixel values at multiple first calibration positions in the first calibration image corresponding to multiple calibration grayscales in the first calibration grayscale group, and the first calibration curve indicates The mapping relationship between the pixel values collected by the image sensor under the first calibration exposure and the grayscale values of multiple calibration grayscales in the first calibration grayscale group;

acquiring a second calibration image including a plurality of calibration grayscales in a second calibration grayscale group with a second calibration exposure different from the first calibration exposure;

Determine a second calibration curve based on pixel values at multiple second calibration positions in the second calibration image corresponding to multiple calibration grayscales in the second calibration grayscale group, the second calibration curve indicates A mapping relationship between the pixel values collected by the image sensor under the second calibration exposure and the grayscale values of multiple calibration grayscales in the second calibration grayscale group; and

Determining a calibration response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the calibration response curve indicates the pixel value collected by the image sensor and the exposure used when capturing the image and includes the first calibration curve A mapping relationship between grayscale values within a grayscale range of a plurality of calibrated grayscales in a calibrated grayscale group and at least one of the second calibrated grayscale group.
The image processing method according to claim 9, wherein determining the calibration response curve of the image sensor based on the first calibration curve and the second calibration curve comprises:

Based on the second calibration curve, determine the maximum pixel value corresponding to the maximum calibration gray scale value among the multiple calibration gray scales in the second calibration gray scale group under the second calibration exposure;

Based on the maximum pixel value, splicing the curve part of the first calibration curve higher than the maximum pixel value of the second calibration curve with the second calibration curve to obtain an adjusted second calibration curve, wherein the The adjusted second calibration curve indicates a mapping relationship between at least one grayscale value higher than the maximum calibrated grayscale value in the grayscale range and pixel values collected by the image sensor under a second calibrated exposure; and

Fitting the adjusted second calibration curve to the first calibration curve to obtain the calibration response curve.
The image processing method according to claim 9, wherein said second calibration exposure is smaller than said first calibration exposure.
The image processing method according to claim 9, wherein said first calibration grayscale group includes a plurality of calibration grayscales that change with a predetermined grayscale variation from a reference grayscale value, and said second calibration grayscale group includes A plurality of calibration gray scales of the same number varying with the same predetermined gray scale variation from the reference gray scale value.
The image processing method according to any one of claims 9-12, wherein the calibration response curve is represented by the following formula:

Among them, I represents the pixel value collected by the image sensor, S represents the grayscale value, E represents the exposure amount, and a, γ, b, δ, and w are constants.
A method for calibrating a response curve of an image sensor, comprising:

Acquiring a first calibration image including a plurality of calibration grayscales in the first calibration grayscale group with a first calibration exposure;

A first calibration curve is determined based on pixel values at multiple first calibration positions in the first calibration image corresponding to multiple calibration grayscales in the first calibration grayscale group, and the first calibration curve indicates The mapping relationship between the pixel values collected by the image sensor under the first calibration exposure and the grayscale values of multiple calibration grayscales in the first calibration grayscale group;

acquiring a second calibration image including a plurality of calibration grayscales in a second calibration grayscale group with a second calibration exposure different from the first calibration exposure;

Determine a second calibration curve based on pixel values at multiple second calibration positions in the second calibration image corresponding to multiple calibration grayscales in the second calibration grayscale group, the second calibration curve indicates A mapping relationship between the pixel values collected by the image sensor under the second calibration exposure and the grayscale values of multiple calibration grayscales in the second calibration grayscale group; and

Determining a response curve of the image sensor based on the first calibration curve and the second calibration curve, wherein the response curve indicates the pixel values collected by the image sensor and the exposure used when capturing the image and includes the first calibration A mapping relationship between the gray scale group and the gray scale values within the gray scale range of at least one of the multiple calibration gray scales in the second calibration gray scale group.
14. The method of claim 14, wherein the exposure amount is determined based on an exposure time and an exposure gain used in acquiring an image with the image sensor.
The method of claim 14, wherein said second nominal exposure level is less than said first nominal exposure level.
The method according to claim 14, wherein said first set of calibration gray scales comprises a plurality of calibration gray scales varying by a predetermined gray scale variation from a reference gray scale value, said second calibration gray scale set comprises The same number of nominal grayscales whose grayscale values start to vary by the same predetermined grayscale variation.
The method according to claim 14, wherein the plurality of calibration gray scales change with a predetermined gray scale variation, and the maximum gray scale value including the gray scale range is greater than the maximum value of the first calibration gray scale group The calibrated grayscale value and the maximum calibrated grayscale value of the second calibrated grayscale group.
The method of claim 14, wherein determining a response curve of the image sensor based on the first calibration curve and the second calibration curve comprises:

determining the maximum pixel value corresponding to the maximum calibrated grayscale value of the second calibrated grayscale group under the second calibrated exposure amount based on the second calibration curve;

Based on the maximum pixel value, splicing the curve part of the first calibration curve higher than the maximum pixel value of the second calibration curve with the second calibration curve to obtain an adjusted second calibration curve, wherein the The adjusted second calibration curve indicates a mapping relationship between at least one grayscale value higher than the maximum calibrated grayscale value in the grayscale range and pixel values collected by the image sensor under a second calibrated exposure; and

Fitting the adjusted second calibration curve to the first calibration curve to obtain the response curve.
The method according to any one of claims 14-19, wherein the response curve is represented by the following formula:

Among them, I represents the pixel value collected by the image sensor, S represents the grayscale value, E represents the exposure amount, and a, γ, b, δ, and w are constants.
An electronic device comprising:

processor; and

A memory storing a program comprising instructions which when executed by the processor causes the processor to perform the method according to any one of claims 1-20.
A non-transitory computer-readable storage medium storing a program, the program comprising instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the operation described in any one of claims 1-20. described method.
A computer program product comprising a computer program, wherein said computer program implements the method according to any one of claims 1-20 when executed by a processor.