WO2023000868A1 - Image processing method and apparatus, device, and storage medium - Google Patents

Abstract

An image processing method and apparatus, a device, and a storage medium. The method comprises: segmenting an image to be processed on the basis of a first visual feature of a pixel unit of an image to be processed so as to obtain multiple image areas, the first visual feature comprising at least one of brightness and color temperature; determining a first gamma curve corresponding to the first image area, the first image area being one of the multiple image areas; and performing gamma correction on the first image area at least partly on the basis of the first gamma curve.

Description

Image processing method and device, equipment, storage medium

Cross References to Related Applications

This application is based on a Chinese patent application with application number 202110836393.X and a filing date of July 23, 2021, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated in its entirety this application.

technical field

This application relates to image processing technology, but not limited to image processing methods, devices, equipment, and storage media.

Background technique

For an image, brightness is an important factor affecting the display effect of the image. In order to improve the final display effect of the image, gamma correction is usually performed on the image before display. The so-called gamma correction is to edit the gamma curve of the image, and use the method of non-linear tone editing to the image to determine the dark part and light part of the image signal, and increase the ratio of the two, thereby improving the image quality. The contrast effect. In the field of computer graphics, it is customary to use the conversion relationship curve between the screen output voltage and the corresponding brightness, which is called the gamma curve. However, for some images, the gamma-corrected image quality is poor.

Contents of the invention

In view of this, the image processing method, device, device, and storage medium provided in the embodiments of the present application can improve image quality and improve user's visual experience. The image processing method, device, equipment, and storage medium provided in the embodiments of the present application are implemented as follows:

The image processing method provided in the embodiment of the present application includes: based on the first visual feature of the pixel unit of the image to be processed, segmenting the image to be processed to obtain multiple image regions; wherein the first visual feature includes the following At least one of: brightness, color temperature; determining a first gamma curve corresponding to a first image area; wherein, the first image area is one of the plurality of image areas; based at least in part on the first A gamma curve, performing gamma correction on the first image area.

The image processing chip provided in the embodiment of the present application includes a processor unit configured to execute the image processing method described in the embodiment of the present application.

The image processing device provided in the embodiment of the present application includes: a segmentation module configured to segment the image to be processed based on the first visual feature of the pixel unit of the image to be processed to obtain a plurality of image regions; wherein the first A visual feature includes at least one of the following: brightness, color temperature; a determination module configured to determine a first gamma curve corresponding to a first image area; wherein the first image area is one of the plurality of image areas 1. A first correction module configured to perform gamma correction on the first image region based at least in part on the first gamma curve.

The electronic device provided by the embodiment of the present application includes a memory and a processor, the memory stores a computer program that can run on the processor, and the processor implements the method described in the embodiment of the present application when executing the program.

The computer-readable storage medium provided by the embodiment of the present application has a computer program stored thereon, and when the computer program is executed by a processor, the method described in the embodiment of the present application is implemented.

In the embodiment of the present application, the image to be processed is segmented based on the first visual feature of the pixel unit of the image to be processed to obtain multiple image regions; then, the first gamma curve corresponding to the first image region is determined, at least partially Based on the first gamma curve, gamma correction is performed on the first image region; in this way, the corresponding gamma curve is used for correction for image regions with different visual characteristics, so that the quality of the image region can be improved in a targeted manner, and then Improve the user's visual experience.

Description of drawings

The accompanying drawings here are incorporated into the specification and constitute a part of the specification. These drawings show embodiments consistent with the application, and are used together with the description to describe the technical solution of the application.

FIG. 1 is a schematic diagram of an implementation flow of an image processing method provided by the present application;

Figure 2 shows the gamma curves of the human eye in different environments;

FIG. 3 is a schematic diagram of an implementation flow of another image processing method provided by the present application;

FIG. 4 is a schematic diagram of a transitional image area provided by the present application;

FIG. 5 is a schematic diagram of an implementation flow of another image processing method provided by the present application;

6 is a schematic diagram of the relationship between brightness in nature and the corresponding brightness value perceived by human eyes;

Figure 7 is a comparison of image display effects before and after gamma correction;

FIG. 8 is a schematic diagram of an implementation flow of another image processing method provided by the present application;

FIG. 9 is a schematic diagram of the brightness of picture a before gamma correction;

FIG. 10 is a schematic diagram of gamma curves corresponding to scene a1 and scene b1 of the present application;

Fig. 11 is a schematic diagram of the transitional image area of the picture and the gamma correction of the image area in the present application;

12 is a schematic diagram of the weight relationship between the first weight function and the second weight function of the present application at the same pixel position;

FIG. 13 is a schematic diagram of the overall brightness of picture a after gamma correction in the present application;

FIG. 14 is a schematic structural diagram of an image processing device of the present application;

FIG. 15 is a schematic structural diagram of an electronic device provided by the present application.

detailed description

In order to make the purpose, technical solutions and advantages of the present application clearer, the specific technical solutions of the present application will be further described in detail below in conjunction with the drawings in the present application. The following examples are used to illustrate the present application, but not to limit the scope of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terminology used herein is for the purpose of describing the application only and is not intended to limit the application.

In the following description, references to "some embodiments" describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or a different subset of all possible embodiments, and Can be combined with each other without conflict.

It should be pointed out that the term "first\second\third" involved in this application does not represent a specific ordering of objects. It is understandable that "first\second\third" can be interchanged with specific sequence or sequence such that the application described herein can be practiced in sequences other than those illustrated or described herein.

An embodiment of the present application provides an image processing method, which is applied to an electronic device, and the electronic device may be various types of devices with information processing capabilities during implementation, for example, the electronic device may include a mobile phone, a tablet computer , laptop computer, projector, desktop computer, personal digital assistant, navigator, digital phone, video phone, television or sensory equipment, etc. The functions realized by the method can be realized by calling the program codes by the processor in the electronic device, and of course the program codes can be stored in the computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

Figure 1 is a schematic diagram of the implementation flow of the image processing method provided by the embodiment of the present application. As shown in Figure 1, the method may include the following steps 101 to 103:

Step 101: Segment the image to be processed based on the first visual feature of the pixel unit of the image to be processed to obtain a plurality of image regions; wherein the first visual feature includes at least one of the following: brightness and color temperature.

In this embodiment of the present application, the images to be processed may be various types of images. For example, the image to be processed is an image to be corrected by the image sensor; another example, the image to be processed is an image to be displayed on the image display terminal. In a word, in the embodiment of the present application, the image processing method described in the present application can be executed on the images generated in different stages.

In some embodiments, the image to be processed may be segmented in the following manner: the pixel units in which the first visual feature falls within the same specific interval are divided into one image area, so as to obtain multiple image areas.

For example, if the first visual feature is brightness, the pixel units whose brightness falls in the first brightness range can be divided into one image area, and the pixel units whose brightness falls in the second brightness range can be divided into another image area, and so on. For another example, if the first visual feature is color temperature, pixel units whose color temperature falls within the first color temperature range can be divided into one image area, and pixel units whose color temperature falls within the second color temperature range can be divided into another image area, and so on. For another example, the first visual feature is brightness and color temperature, divide the pixel units whose brightness falls into the third brightness range and whose color temperature falls into the third color temperature range into an image area, and divide the pixel units whose brightness falls into the fourth brightness range and whose color temperature falls into the third color temperature range The pixel unit of the four color temperature intervals is divided into another image area, and so on. It should be noted that a specific interval has at least one boundary, for example, a specific interval is [m,∞), it can be seen that it has only one boundary m; another example, a specific interval is [A, B], it can be seen that it has Two boundaries, namely A and B.

It should also be noted that a pixel unit may include one pixel or multiple pixels. When multiple pixels are included, the first visual feature of the pixel unit may be a representative feature of the visual feature of each pixel, and the representative feature may be an average value or a median value.

In the embodiment of the present application, whether to segment the image to be processed may be unconditional, and in some embodiments, the segmentation may also be conditional. For example, if the overall second visual feature of the image to be processed satisfies the segmentation condition, execute steps 101 to 103; otherwise, if the second visual feature does not meet the segmentation condition, the image to be processed is not segmented, but determined The fourth gamma curve corresponding to the second visual feature, and performing gamma correction on the image to be processed according to the fourth gamma curve; in this way, under the premise of ensuring that each image conforms to the human eye's perception of brightness, Reduce unnecessary segmentation and other processing, thereby saving power consumption and computing resources.

Wherein, the electronic device may determine the overall second visual feature of the image to be processed based on the first visual feature of the pixel units of the image to be processed.

In some embodiments, the second visual feature may be a value used to characterize changes in the first visual feature of the pixel units of the image to be processed. For example, the second visual feature includes the variance and/or standard deviation of brightness of each pixel unit of the image to be processed, and/or the variance and/or standard deviation of color temperature of each pixel unit of the image to be processed. Correspondingly, the segmentation condition is that the variance and/or the standard deviation is greater than a corresponding threshold.

In some other embodiments, the second visual feature may also be used to characterize whether the image to be processed includes a light source, the number of light sources and/or the position of the light source, and the like. Correspondingly, the segmentation condition is: the image to be processed includes a light source, the number of light sources is greater than a corresponding threshold, and/or the position of the light source is at a specific position, etc. For example, the specific location is an edge area or a central area of the image to be processed.

In some other embodiments, the segmentation condition may also be: the variance and/or the standard deviation are greater than a corresponding threshold, and the image to be processed includes light sources, the number of light sources is greater than the corresponding threshold, and/or the position of the light source is at a specific position.

Step 102, determining a first gamma curve corresponding to a first image area; wherein, the first image area is one of the plurality of image areas.

In the case that the image to be processed is the image to be corrected at the image sensor end, the first gamma curve is obtained by calibration at the image sensor end, based on this, the first image region is calculated based on the first gamma curve at the image sensor end Perform gamma correction. In addition, the image sensor side also calibrates other different gamma curves, and different gamma curves are applicable to different image areas. In some embodiments, different gamma curves have a mapping relationship with a third visual feature characterizing the scene. The electronic device may determine the overall fourth visual feature of the first image area, and then determine a third visual feature that matches the fourth visual feature from different third visual features, and then obtain and determine the fourth visual feature. The gamma curve corresponding to the three visual features (namely the first gamma curve); wherein, different third visual features correspond to different scenes, in other words, different scenes include different light sources and light intensities. The electronic device may pre-acquire the images taken of the same scene under each candidate gamma curve, and then select a target image whose image quality meets the conditions from the captured images, and use the candidate gamma curve corresponding to the target image as the The final gamma curve for the scene.

For example, the electronic device captures a scene under candidate gamma curve 1 to obtain image 1, captures the scene under candidate gamma curve 2 to obtain image 2, and captures the scene under candidate gamma curve 3 to obtain In image 3, the tester can select the image that is most in line with human perception from image 1 to image 3, and use the candidate gamma curve corresponding to the image as the final gamma curve of the scene; then, the electronic device calibrates the final gamma curve of the scene. gamma curve. In this way, the final gamma curves of different scenes are selected through testing, and the mapping relationship between the third visual feature of each scene and the final gamma curves of the corresponding scenes is recorded.

For the case where the image to be processed is the image to be displayed on the image display terminal, it is similar to the solution of determining the corresponding first gamma curve in the above solution of the image to be corrected, and will not be repeated here.

In the embodiment of the present application, the fourth visual feature represents the overall feature of the first image region, for example, the overall feature is an average brightness value, an average color temperature value, a median color temperature value, and/or a median brightness value.

In some embodiments, the multiple gamma curves calibrated by the image sensor end and/or the image display end correspond to different visual feature intervals. For example, its mapping relationship is shown in Table 1: gamma curve 10 corresponding to brightness interval 10, gamma curve 20 corresponding to brightness interval 20, ..., gamma curve N0 corresponding to brightness interval N0; wherein, gamma curve 10, 20 , . . . , N0 are different, and the brightness interval 10 to the brightness interval N0 include different brightness.

Table 1

Brightness range 10	Gamma Curve 10
Brightness range 20	Gamma Curve 20
…	…
Brightness interval N0	Gamma curve N0

The electronic device may determine the gamma curve corresponding to the interval from Table 1 according to the brightness interval in which the average brightness value or the median brightness value of the first image area falls, and use the curve as the first gamma curve corresponding to the first image area. For example, if the brightness mean value or brightness median value of the first image area falls into the brightness interval 20, then the gamma curve 20 is used as the first gamma curve.

As another example, as shown in Table 2: the gamma curve 11 corresponding to the brightness interval 11 and the color temperature interval 11, the gamma curve 21 corresponding to the brightness interval 21 and the color temperature interval 21, ..., the gamma curve corresponding to the brightness interval N1 and the color temperature interval N1 N1; wherein, the gamma curves 11, 21, .

Table 2

The electronic device may determine the corresponding first gamma curve according to the interval in which the average brightness value and the average color temperature value of the first image region fall. For example, if the average brightness falls into the brightness interval 11, and the average color temperature falls into the color temperature interval 11, the gamma curve 11 is taken as the first gamma curve; for another example, if the average brightness falls into the brightness interval 11, the average color temperature falls into If the color temperature range is 12, the gamma curve 11 or the gamma curve 21 can be used as the first gamma curve, or the fusion of the gamma curve 11 and the gamma curve 21 can be used as the first gamma curve.

Step 103: Perform gamma correction on the first image region based at least in part on the first gamma curve.

It is understandable that for an image to be processed, especially an image with obvious changes in light and dark, if the same gamma curve is used to correct it, the result may make the brighter area brighter, or even appear white, while the darker Dark areas are even darker, and even appear black.

In view of this, in the embodiment of the present application, the image to be processed is segmented based on the first visual feature of the pixel unit of the image to be processed to obtain multiple image regions; then, the first gamma curve corresponding to the first image region is determined , performing gamma correction on the first image region based at least in part on the first gamma curve; in this way, for image regions with different visual characteristics, the corresponding gamma curve is used for correction, so that the gamma correction of the image region can be improved in a targeted manner. quality, thereby enhancing the user's visual experience.

In some embodiments, gamma correction may be performed on the first image region based on the first gamma curve; gamma correction may also be performed on the first image region based on the third gamma curve and the first gamma curve; wherein, The third gamma curve is obtained by calibration at the image display end, and the first gamma curve is obtained by calibration at the image sensor end.

In some embodiments, different third gamma curves correspond to different light characteristics, and the electronic device may determine the corresponding third gamma curve according to the light characteristics of the current physical environment.

Understandably, under different light intensities, the corresponding gamma curves of human eyes are different. As shown in Figure 2, in a scene where the contrast between light and dark is particularly obvious, the gamma value corresponding to the human eye will adapt to the overall brightness. Compress the dark places and brighten the dark places. Therefore, in the embodiment of the present application, when gamma correction is performed on the first image area, not only the influence of the visual characteristics of the first image area itself on the gamma curve of the human eye is considered, but also the current location of the electronic device is considered. The influence of the light characteristics of the physical environment on the gamma curve of the human eye; in this way, it helps to simulate the human eye's perception of light and shade in different light environments, thereby eliminating image mutations and making the first image area after gamma correction more accurate. Conforms to the human eye's perception of light and dark.

The present application further provides an image processing method. FIG. 3 is a schematic diagram of the implementation flow of another image processing method provided by the present application. As shown in FIG. 3 , the method may include the following steps 301 to 305:

Step 301, segment the image to be processed based on the first visual feature of the pixel unit of the image to be processed to obtain multiple image regions; wherein the first visual feature includes at least one of the following: brightness, color temperature;

Step 302, determining a first gamma curve corresponding to a first image area; wherein, the first image area is one of the plurality of image areas;

Step 303, performing gamma correction on the first image region based at least in part on the first gamma curve;

Step 304, determining a second gamma curve corresponding to a second image area; wherein, the second image area is another image area adjacent to the first image area among the plurality of image areas;

Step 305: Perform gamma correction on the second image region based at least in part on the second gamma curve.

It should be noted that each image area of the image to be processed adopts the processing method for the first image area or the second image area described in the embodiment of the present application; in this way, when the electronic device performs gamma correction on the image to be processed , instead of using only one gamma curve to perform gamma correction on the entire image, the image is divided into multiple image regions, so as to select the gamma curve that matches the image region in a targeted manner, and based on this, the corresponding image region Perform gamma correction to improve the gamma correction effect of the image to be processed and improve image quality.

In some embodiments, the method further includes: performing gamma correction on a transitional image region based at least in part on the first gamma curve and/or the second gamma curve, wherein the transitional image region includes A boundary line between the first image area and the second image area.

For the determination method of the transitional image area, in some embodiments, the boundary line between the first image area and the second image area can be determined first; An area enclosed by the second image area offset by a specific pixel distance is used as the transition image area.

For example, as shown in FIG. 4, 401 is the boundary line between the first image area 402 and the second image area 403, then, the boundary line 401 can be offset upward by the first pixel distance to obtain a boundary 404, and the boundary line 401 is shifted down by a second pixel distance, resulting in border 405 . The area enclosed by the boundary 404 and the boundary 405 is the transition image area between the first image area 402 and the second image area.

In the embodiment of the present application, the first pixel distance and the second pixel distance can be the same or different, and there is no limit to the size of the first pixel distance and the second pixel distance. In a word, to eliminate the image mutation near the dividing line, should. For example, the first pixel distance and the second pixel distance are set specific pixel distances, and the value is a length of 3 to 5 pixel points.

Further, in some embodiments, the electronic device may perform gamma correction on the transitional image region in the following way: based on the first weight function and the second weight function, the first gamma curve and the second gamma curve The weighted sum is used as a fifth gamma curve; performing gamma correction on the transition image region based at least in part on the fifth gamma curve; thus, the brightness between different regions of the gamma-corrected image can be further improved continuity, the image effect looks more natural.

Wherein, the first weight function is the coefficient of the first gamma curve, the second weight function is the coefficient of the second gamma curve, and the value of the first weight function corresponding to the same pixel unit is the same as The sum of the values of the second weight function is 1; the value of the first weight function is negatively correlated with the distance from the first pixel unit to the first boundary of the first image area, wherein the first pixel A unit is a pixel unit belonging to a first image area in the transition image area, and the first boundary is located in the first image area.

For example, the expression of the gamma curve fun3(light _{(x, y)} ) of the transition image region shown in the following formula (1):

fun3(light _(x,y) )＝weight1*fun1(light _(x,y) )+weight2*fun2(light _(x,y) ) formula (1);

Among them, light _{(x, y)} represents the brightness of the pixel unit (x, y), fun1 (light _{(x, y)} ) represents the first gamma curve of the first image area, weight1 refers to the first weight function, the function The independent variable of is the pixel distance from the pixel unit of the transition image area to the first boundary; weight2 refers to the second weight function, and the sum of the first weight function is 1; fun2(light _(x,y) ) indicates the second image The second gamma curve for the region.

In some embodiments, the electronic device may determine the gamma value of the pixel unit (x, y) in the transition image area according to the above formula (1), and perform gamma correction on the brightness of the pixel unit according to the gamma value.

The present application further provides an image processing method. FIG. 5 is a schematic diagram of the implementation flow of another image processing method provided by the present application. As shown in FIG. 5 , the method may include the following steps 501 to 505:

Step 501, segment the image to be processed based on the first visual feature of the pixel unit of the image to be processed to obtain multiple image regions; wherein the first visual feature includes at least one of the following: brightness, color temperature;

Step 502, determine the first gamma curve corresponding to the first image area; wherein, the first image area is one of the plurality of image areas, and the first gamma curve is calibrated at the image sensor end obtained;

Step 503: Perform gamma correction on the first image region based at least in part on the first gamma curve.

Step 504, determining a third gamma curve corresponding to the first image area; wherein, the third gamma curve is obtained by performing calibration on the image display terminal.

In some embodiments, the electronic device may perform gamma correction on the first image area based on the first gamma curve at the image sensor end to obtain the corrected first image area; then, at the image display end based on the third A gamma curve, performing gamma correction on the corrected first image area to obtain a target image area.

In some other embodiments, the electronic device may firstly fuse the first gamma curve and the third gamma curve to obtain the fused gamma curve; then, based on the fused gamma curve, the first image region Perform gamma correction to obtain the target image area. In a word, this application does not limit how the electronic device corrects the first image region based on the first gamma curve and the third gamma curve.

Further, in some embodiments, the electronic device may implement step 404 as follows: determine the light characteristics of the physical environment where the electronic device is located; and determine the corresponding third gamma curve according to the light characteristics.

As mentioned above, under different light intensities, the gamma curves corresponding to the human eye are different. Therefore, in some embodiments, the electronic device can calibrate different light characteristics on the image display end to correspond to the corresponding gamma curves, so that the electronic device can determine the corresponding gamma curve according to the light characteristics of the current physical environment before actually displaying the image to be processed. The third gamma curve, so that the display effect of the first image area corrected based on the third gamma curve is more in line with the current perception of light and shade by human eyes.

In some embodiments, the light characteristics of the physical environment where the electronic device is located may be collected through a sensor on the electronic device. Further, in some embodiments, the light characteristics may include light intensity and/or color temperature.

Step 505: Perform gamma correction on the first image region according to the third gamma curve.

It should be noted that, in the embodiment where both the image sensor end and the image display end perform gamma correction on the image to be processed, the two ends may or may not use the same image segmentation result. When not in use, the electronic device can re-segment the image to be displayed on the image display end. The segmentation method is the same as the solution described above, and will not be repeated here. Determine the sixth gamma curve corresponding to the image area of the image to be displayed, and perform gamma correction on the image area; wherein, the sixth gamma curve is obtained through calibration at the image display end. The processing method for the transitional image area of the adjacent image area of the image to be displayed is the same as the above solution, and will not be repeated here.

Gamma correction is used to solve the nonlinear output problem of CRT monitors, and can also help to "improve" the output image quality. Figure 6 shows the brightness in nature and the corresponding brightness value felt by the human eye. It can be seen from the figure , the human eye is more sensitive to darker (closer to 0) brightness values, and less sensitive to brighter (closer to 1) brightness values. It can be understood that the human eye can better distinguish changes in darker brightness values, so the color is in When storing, the color value of the darker part should be preserved more.

In some embodiments, the gamma curve is set according to the characteristics of the image sensor (sensor) and lens (lens), lighting environment, and even user preference. When the Gamma value is large, the image will be very bright, but the contrast and saturation will decrease, as shown in Figure 7, it feels intuitively that there is a layer of fog covering the picture, and the noise in the dark will be amplified. When the Gamma value is small, the brightness will decrease, the overall contrast of the image will increase, the color will be more vivid, the noise in the dark place will be smaller, and the picture in the dark place will be darker, making it difficult to see clearly.

An exemplary application of the present application in a practical application scenario will be described below.

In some embodiments, a gamma curve is used to perform gamma correction on the image to be processed during shooting by the mobile phone. Although this can improve the quality of the image to be processed, the gamma curves corresponding to the actual human eyes are not exactly the same in scenes with different brightness, especially in environments with obvious changes in light and dark and in environments where light and dark changes are not obvious. For example, in an environment with obvious changes in light and shade, the gamma corresponding to the human eye will adapt to the overall brightness. The higher the overall brightness, the steeper the gamma, compressing bright areas and brightening dark areas. That is to say, the human eye does not perceive the actual darker area as darker. For the brighter area, the gamma of the human eye will be automatically compressed, and the human eye will not feel that it is particularly bright. For such an environment with obvious changes in light and shade, if the same gamma curve is used for correction, the result will be that the brighter area will be brighter, or even appear white, while the darker area will be darker, or even appear black.

Therefore, further, in some embodiments, a gamma curve parameter that can adaptively adjust gamma curve parameters for different shooting scenes is provided, so as to simulate the human eye's perception of lightness and darkness in different environments as much as possible, and output the corresponding Image.

As shown in Figure 8, the overall process includes three parts:

The first part: gamma parameter calibration stage: a set of gamma curves divided and calibrated according to the scene;

This part calibrates the gamma curves in different scenes, that is, the gamma curves corresponding to different visual feature intervals.

The second part, the image capture and generation stage: calculate the gamma correction table corresponding to the image;

In this part, first determine the degree of brightness change of the captured image (that is, an example of an image to be processed), and based on the degree of brightness change, determine whether the captured image needs to be segmented; if necessary, perform regional segmentation on the captured image , to obtain image regions corresponding to different scenes, and then, based on the brightness of the image region, search for a corresponding gamma curve from the first part of the calibrated gamma curve set, and use the gamma curve to perform gamma correction on the corresponding image region. And for the situation that multiple scenes appear in the same image, interpolation is performed on the adjacent areas of the corresponding multiple image areas (ie transition image areas), so as to ensure the continuity of the adjacent image areas after gamma correction.

The third part, the image display stage, corrects the gamma correction according to the display environment and finally displays it;

Since the display effect of the actual picture is not only related to the brightness and darkness of the captured image itself, but also related to the physical environment where the display device is located, therefore, the gamma curve obtained in the second part needs to be adjusted according to the actual physical environment.

The following describes each step accordingly:

For the first part, you can test in advance, select different scenes, calibrate the gamma curves in different scenes, such scenes include different light sources and light intensities, and select images under different gamma curves captured in this scene and The image that is most consistent with the perception of the person being tested is used to select the gamma curve (that is, the gamma correction parameter) that is most in line with the perception of the human eye in different scenes. (There is no uniform standard for the shape of the gamma curve, and there are many human-influenced factors. A reference method for setting the shape of the gamma curve is to shoot 24 color cards in a light box environment, and make the brightness data Y of the 6 color blocks at the bottom of the color card as much as possible showing a linear proportional relationship.)

In the actual ISP design, the method of segmented linear interpolation is generally used, and SRAM can also be used to realize point-by-point mapping, so that the shape of the curve can be flexibly configured. After the first part is completed, the mapping relationship between multiple sets of gamma curves and scenes can be obtained.

For the second part, scene recognition and scene segmentation: in actual shooting, due to the complexity of the scene, the scene in the picture often contains not exactly the same environment when shooting, and the acquired image is down-sampled at the front end to obtain RGB condensed Outline, use the front-end segmentation module to segment different image areas, judge the environment of different image areas, and determine the gamma curve under the environment according to the judged environment.

Then, corresponding corrections are made to the gamma curve at the edge of the region to ensure that the change of the gamma curve between the divided regions is relatively uniform, so as to ensure that the brightness and darkness of the picture will not jump.

As an example to illustrate the overall solution process, the brightness of a certain picture a before gamma correction is shown in Figure 9, where x and y represent the pixel position respectively, and the z value represents the brightness of the pixel position. After the scene is segmented, the image is divided into two image areas, as shown in FIG. 4 , one image area 302 includes the scene a1, and the other image area 303 includes the scene b1.

Among them, the horizontal and vertical coordinates correspond to the positions of the pixels in the picture a, and the curve represents the divided boundary line 301, which divides the whole picture into scene a1 and scene b1, and the gamma curve corresponding to the pixel in scene a1 is The gamma curve a1, the gamma curve corresponding to the pixel in the scene b1 is the gamma curve b1.

Due to the difference in gamma correction between different scenes, a sudden change will occur near the boundary line. By setting a transitional image area at the boundary line and performing interpolation, the continuity of brightness changes is ensured, and the boundary line 301 is offset by a certain number of pixels on both sides. The distance p is used to obtain the transition image area c, and the gamma curve corresponding to the pixels in the transition image area c adopts the weighted average of the former two, and sets the weight according to the distance of the offset.

According to scene a1 and scene b1, compare the values obtained in the first part to find the corresponding gamma curve. For example, as shown in Figure 10, the obtained gamma curves gamma=2.1 and gamma=2.4 are just examples here, and the actual situation is not the case . Among them, x and y represent pixel positions respectively, and different gamma curves are used according to different scenes. In order to ensure the continuity, the curve of the transition image area is interpolated to ensure the continuity of the brightness change after mapping. The method used here is linear interpolation, as shown in the following formula (2), the obtained gamma curve fun3 of the transition image area The expression for (light _(x,y) ) is:

fun3(light _(x,y) )＝weight1*fun1(light _(x,y) )+weight2*fun2(light _(x,y) ) formula (2);

Among them, light _{(x, y)} represents the brightness of the pixel unit (x, y), fun1(light _{(x, y)} ) represents the target gamma curve of the image area, weight1 refers to the first weight function, and the argument of the function is the pixel distance from the pixel unit of the transition image area to the boundary of the transition image area in the image area; weight2 refers to the second weight function, and the sum of the first weight function is 1; fun2(light _(x,y) ) means Target gamma curves of image regions adjacent to the image region.

For example, FIG. 11 is a schematic diagram of the transitional image area of the above picture a and the gamma correction of the image area. As shown in FIG. 11, the area [0.0.5] represents the gamma correction result in the image area a1, [ 0.5,1.5] represents the gamma correction result in the transition image region, and [1.5,2] represents the gamma correction result in the image region b1. It should be noted that 0.5, 1.5, 2, etc. here have no practical meaning, and are only examples, and do not limit the scope of the technical solution of the present application.

When gamma correction is performed, as shown in the following formula (3), the gamma correction of pixel values at different positions is different according to the area divided by it, and different gamma curves are used:

Among them, (x, y) represents the position of the pixel in the picture, light represents the brightness of the pixel at the x, y position, weight represents the weight, where the pixel width is p, when the pixel is on the upper boundary (that is, the transition image area is in The boundary of the image area of a1) weight1 is 1, weight2 is 0; when the pixel point is at the lower boundary (that is, the transition image area is at the boundary of the image area of b1), weight1 is 0, weight2 is 1, and other areas change as shown in Figure 12 As shown, the abscissa represents the proportion of the distance between the pixel point in the transition image area and the upper boundary (the maximum value is 1, and the distance from the upper boundary is 1*p, which is the lower boundary).

After corresponding gamma correction is performed on picture a, the overall brightness of picture a is shown in FIG. 13 .

The corresponding gamma curves in different areas of the picture are saved and bound to the picture ID, and stored locally. When actually displayed, the light sensor of the mobile phone is used to obtain the light intensity and color temperature of the current external environment, combined with the currently displayed ambient brightness. Combined with the gamma curve in the first part, the data in the second part is subjected to corresponding light and dark calibration processing, and the corresponding image is output.

For example, it can be found from the gamma curve collection in the first part that the gamma curve corresponding to the currently displayed scene is fun2(), combined with the gamma curves in different areas of the shooting scene picture in the second part is fun(), Then the gamma curve used in the final display is: Weignt2*fun()+Weignt3*fun2().

Among them, Weignt2 and Weignt3 are both constants, using the weighted average of the two functions, Weignt2+Weignt3=1, Weignt3 represents the correction range, which can be determined according to empirical values, so as to obtain the brightness of the final displayed image.

It is understandable that the gamma curves corresponding to the human eye are not completely consistent in different environments, especially the corresponding gamma curves are not completely consistent under environments with obvious changes in brightness and darkness and in environments with insignificant changes in brightness and darkness. In this application, different gamma curves are used according to the shooting scene and the display scene, and the gamma curve is adaptively adjusted for different shooting scenes, so as to simulate the human eye's perception of light and shade in different environments as much as possible, and the output corresponding image.

In this application, multiple sets of different gamma curves are used in comparison with different scenes in the same image. For the color temperature of multiple light sources in the same scene, the front-end chip is used to divide different regions according to the color temperature and light intensity to select the gamma curve. And interpolation is performed in the transition image area of the adjacent image area to ensure the continuity of the brightness change.

In this application, the determination of the gamma curve during actual display is adjusted again according to the light intensity and color temperature of the current display environment, the corresponding gamma curve is obtained according to the current display scene, and the gamma curve corresponding to the existing shooting environment is carried out Weighted to get the final gamma curve.

In some embodiments, the gamma curve sets generated by human eye calibration under different color temperature and light intensity scenes are pre-calibrated. When taking pictures, the original image is first divided into image regions according to the overall brightness difference of the picture in the front-end chip; The principle of image segmentation is based on scene areas with obviously different color temperatures and light intensities in the overall picture. By setting threshold ladders (multiple sets of thresholds), the color temperature or brightness difference of different areas falls within a certain threshold range to divide into different areas and perform segmentation. For these areas, according to the pre-calibrated gamma curve set, use different gamma curves to perform gamma correction on these areas; then, in order to ensure that the gamma parameters between adjacent image areas in the image change with the pixel position The continuity of the gamma parameter in the whole image is two-dimensional cubic polynomial interpolation; at the same time, the obtained information (including color temperature distribution, light intensity and gamma parameter distribution G1(x,y)) is saved in the image information.

When the picture is previewed and displayed, use the light sensor of the mobile phone to obtain the light intensity and color temperature value of the current external environment, compare it with the existing scene information during shooting, and perform gamut according to the similarity between the current environment and the shooting environment saved in the original picture Secondary adjustment of the horse curve; for example, find the corresponding overall gamma correction parameter G2 based on the environmental information obtained when the picture is currently displayed (because the current display scene is fixed, the gamma parameter is a fixed value), and the gamma generated by the old interpolation The horse parameters G1(x, y) are summed and averaged to obtain the final parameters.

Based on the foregoing embodiments, the present application provides an image processing device, which includes each module included and each unit included in each module, which can be realized by a processor; of course, it can also be realized by a specific logic circuit; During implementation, the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an image signal processing chip (ISP).

FIG. 14 is a schematic structural diagram of the image processing device of the present application. As shown in FIG. 14, the image processing device 140 includes:

The segmentation module 141 is configured to segment the image to be processed based on the first visual feature of the pixel unit of the image to be processed to obtain a plurality of image regions; wherein the first visual feature includes at least one of the following: brightness, color temperature;

The determination module 142 is configured to determine a first gamma curve corresponding to a first image area; wherein the first image area is one of the plurality of image areas;

The first correction module 143 is configured to perform gamma correction on the first image region based at least in part on the first gamma curve.

In some embodiments, the determining module 142 is further configured to: determine a second gamma curve corresponding to a second image area; wherein, the second image area is one of the plurality of image areas that is similar to the first image Another image region adjacent to the region; the first gamma curve is different from the second gamma curve; the first correction module 143 is also configured to: based on the second gamma curve, correct the second gamma curve Image areas are gamma corrected.

In some embodiments, the first correction module 143 is further configured to perform gamma correction on the transitional image region based at least in part on the first gamma curve and/or the second gamma curve, wherein the The transition image area includes a boundary between the first image area and the second image area.

In some embodiments, the determination module 142 is further configured to: determine the boundary line between the first image area and the second image area; The area enclosed by the area and the second image area offset by a specific pixel distance is used as the transition image area.

In some embodiments, the first correction module 143 is configured to: based on the first weight function and the second weight function, use the weighted sum of the first gamma curve and the second gamma curve as the fifth gamma curve; performing gamma correction on the transition image region based at least in part on the fifth gamma curve; wherein the first weight function is a coefficient of the first gamma curve, and the second weight function is the coefficient of the second gamma curve, the sum of the value of the first weight function corresponding to the same pixel unit and the value of the second weight function is 1; the value of the first weight function and the value of the first pixel The distance between the unit and the first boundary of the first image area is negatively correlated, wherein the first pixel unit is a pixel unit belonging to the first image area in the transition image area, and the first boundary is located in the first image area.

In some embodiments, the image processing device 140 further includes a second correction module; wherein, the determination module 142 is further configured to: determine a third gamma curve corresponding to the first image region; wherein, the third gamma curve The horse curve is obtained by calibration at the image display end, and the first gamma curve is obtained by calibration at the image sensor end; the second correction module is also configured to, according to the third gamma curve, correct the Perform gamma correction on the first image area.

In some embodiments, the determining module 142 is configured to: determine light characteristics of the physical environment; and determine a corresponding third gamma curve according to the light characteristics.

In some embodiments, the segmentation module 141 is configured to: segment the pixel units in which the first visual feature falls into the same specific interval into one image area, so as to obtain multiple image areas.

In some embodiments, the determining module 142 is further configured to: determine an overall second visual feature of the image to be processed based on the first visual feature of the pixel units of the image to be processed; if the second visual feature Satisfy the segmentation condition, trigger the segmentation module 141 to segment the image to be processed; if the second visual feature does not meet the segmentation condition, determine the fourth gamma curve corresponding to the second visual feature; and trigger the first correction module 143 according to The fourth gamma curve performs gamma correction on the image to be processed.

In some embodiments, the second visual feature includes at least one of the following: the change of the first visual feature of the pixel unit of the image to be processed; whether the image to be processed includes a light source; the number of the light source ; the position of the light source.

In some embodiments, the segmentation condition includes at least one of the following: the parameter characterizing the change is greater than a corresponding threshold; the image to be processed includes a light source; the number of the light source is greater than a corresponding threshold; the position of the light source at a specific location.

In some embodiments, the determination module 142 is further configured to: acquire the mapping relationship between different gamma curves and the third visual feature representing the scene; determine the overall fourth visual feature of the first image region; from the Determining a third visual feature that matches the fourth visual feature from the third visual features that characterize the scene; determining a gamma curve corresponding to the matched third visual feature as the first gamma curve .

The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

It should be noted that the division of modules by the device in the present application is schematic, and is only a logical function division, and there may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or physically exist separately, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. It can also be implemented in the form of a combination of software and hardware.

It should be noted that, in this application, if the above-mentioned method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of this application or the part that contributes to the related technology can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to make the electronic device Execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes such as U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), magnetic disk or optical disk. As such, the present application is not limited to any specific combination of hardware and software.

The present application provides an image processing chip, including a processor unit configured to: execute the image processing method as described above.

The present application provides an electronic device. FIG. 15 is a schematic diagram of the hardware entity of the electronic device of the present application. As shown in FIG. A computer program running on the processor 152, and the processor 152 implements the steps in the methods provided in the above-mentioned embodiments when executing the program.

It should be noted that the memory 151 is configured to store instructions and applications executable by the processor 152, and may also cache data to be processed or processed by each module in the processor 152 and the electronic device 150 (for example, image data, audio data, etc. , voice communication data and video communication data), can be realized by flash memory (FLASH) or random access memory (Random Access Memory, RAM).

The present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the method provided in the above-mentioned embodiments are implemented.

The present application provides a computer program product containing instructions, which, when run on a computer, causes the computer to execute the steps in the methods provided by the above method embodiments.

It should be pointed out here that: the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to those of the method embodiments. For technical details not disclosed in the storage medium, storage medium, and device embodiments of the present application, please refer to the description of the method embodiment of the present application for understanding.

It should be understood that reference throughout this specification to "one embodiment" or "an embodiment" or "some embodiments" means that a particular feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application . Thus, appearances of "in one embodiment" or "in an embodiment" or "in some embodiments" in various places throughout the specification are not necessarily referring to the same embodiments. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the serial numbers of the above-mentioned processes do not mean the order of execution, and the order of execution of the processes should be determined by their functions and internal logic, and should not be used in the implementation of the present application. process constitutes any qualification. The serial numbers of the above application are for description only, and do not represent the advantages and disadvantages of the embodiments. The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, the same or similar points can be referred to each other, and for the sake of brevity, details are not repeated herein.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, such as object A and/or object B, which can mean: object A exists alone, and object A and object exist at the same time B, there are three situations of object B alone.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article or apparatus comprising that element.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. The above-described embodiments are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or can be Integrate into another system, or some features may be ignored, or not implemented. In addition, the mutual coupling, or direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms of.

The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed to multiple network units; Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can be used as a single unit, or two or more modules can be integrated into one unit; the above-mentioned integration The modules can be implemented in the form of hardware, or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps to realize the above method embodiments can be completed by hardware related to program instructions, and the aforementioned programs can be stored in computer-readable storage media. When the program is executed, the execution includes The steps of the foregoing method embodiments; and the foregoing storage media include: removable storage devices, read-only memory (Read Only Memory, ROM), magnetic disks or optical disks and other media that can store program codes.

Alternatively, if the above-mentioned integrated units of the present application are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of this application or the part that contributes to the related technology can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to make the electronic device Execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media capable of storing program codes such as removable storage devices, ROMs, magnetic disks or optical disks.

The methods disclosed in several method embodiments provided in this application can be combined arbitrarily to obtain new method embodiments under the condition of no conflict.

The features disclosed in several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

The above is only the embodiment of the present application, but the scope of protection of the present application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, and should covered within the scope of protection of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (20)

1. An image processing method, comprising:

   Based on the first visual feature of the pixel unit of the image to be processed, the image to be processed is segmented to obtain a plurality of image regions; wherein the first visual feature includes at least one of the following: brightness, color temperature;

   determining a first gamma curve corresponding to a first image area; wherein the first image area is one of the plurality of image areas;

   The first image region is gamma corrected based at least in part on the first gamma curve.
2. The method according to claim 1, wherein the method further comprises:

   Determining a second gamma curve corresponding to a second image area; wherein, the second image area is another image area adjacent to the first image area among the plurality of image areas; the first gamma the horse curve is different from said second gamma curve;

   Perform gamma correction on the second image area based on the second gamma curve.
3. The method according to claim 2, wherein the method further comprises:

   performing gamma correction on the transitional image region based at least in part on the first gamma curve and/or the second gamma curve; wherein the transitional image region includes the first image region and the second image The dividing line between areas.
4. The method according to claim 3, wherein the method further comprises:

   determining a boundary between the first image area and the second image area;

   The boundary line and the area enclosed by the boundary line offset from the first image area and the second image area by a specific pixel distance are used as the transition image area.
5. The method according to claim 3, wherein performing gamma correction on the transition image region based at least in part on the first gamma curve and/or the second gamma curve comprises:

   Based on the first weight function and the second weight function, using the weighted sum of the first gamma curve and the second gamma curve as a fifth gamma curve;

   gamma correcting the transition image region based at least in part on the fifth gamma curve;

   Wherein, the first weight function is the coefficient of the first gamma curve, the second weight function is the coefficient of the second gamma curve, and the value of the first weight function corresponding to the same pixel unit is the same as The sum of the values of the second weight function is 1; the value of the first weight function is negatively correlated with the distance from the first pixel unit to the first boundary of the first image area, wherein the first pixel A unit is a pixel unit belonging to a first image area in the transition image area, and the first boundary is located in the first image area.
6. The method according to any one of claims 1 to 5, wherein the method further comprises:

   Determining a third gamma curve corresponding to the first image area; wherein, the third gamma curve is obtained by performing calibration on the image display end, and the first gamma curve is obtained by performing calibration on the image sensor end obtained by calibration;

   Perform gamma correction on the first image area according to the third gamma curve.
7. The method according to claim 6, wherein said determining a third gamma curve corresponding to said first image region comprises:

   Determine the light characteristics of the physical environment;

   According to the light characteristics, a corresponding third gamma curve is determined.
8. The method according to any one of claims 1 to 5, wherein the segmentation of the image to be processed based on the first visual feature of the image to be processed to obtain a plurality of image regions includes:

   The pixel units in which the first visual feature falls into the same specific interval are divided into one image area, so as to obtain multiple image areas.
9. The method according to any one of claims 1 to 5, wherein the method further comprises:

   determining an overall second visual feature of the image to be processed based on the first visual feature of the pixel unit of the image to be processed;

   If the second visual feature satisfies the segmentation condition, execute the step of segmenting the image to be processed.
10. The method according to claim 9, wherein the method further comprises:

    If the second visual feature does not satisfy the segmentation condition, determine a fourth gamma curve corresponding to the second visual feature; and perform gamma correction on the image to be processed according to the fourth gamma curve .
11. The method of claim 10, wherein the second visual characteristic comprises at least one of the following:

    The change of the first visual feature of the pixel unit of the image to be processed;

    Whether the image to be processed includes a light source;

    the number of said light sources;

    The location of the light source.
12. The method according to claim 11, wherein the segmentation condition comprises at least one of the following:

    The parameter characterizing the change is greater than the corresponding threshold;

    The image to be processed includes a light source;

    The number of the light sources is greater than a corresponding threshold;

    The position of the light source is at a specific position.
13. The method according to claim 1, wherein said determining the first gamma curve corresponding to the first image region comprises:

    Obtain the mapping relationship between different gamma curves and the third visual feature representing the scene;

    determining an overall fourth visual characteristic of the first image region;

    determining a third visual feature matching the fourth visual feature from the third visual features representing the scene;

    A gamma curve corresponding to the matched third visual feature is determined as the first gamma curve.
14. An image processing chip, comprising a processor unit configured to:

    Executing the image processing method according to any one of claims 1 to 13.
15. An image processing device, comprising:

    The segmentation module is configured to segment the image to be processed based on the first visual feature of the pixel unit of the image to be processed to obtain a plurality of image regions; wherein the first visual feature includes at least one of the following: brightness, color temperature ;

    A determination module configured to determine a first gamma curve corresponding to a first image area; wherein the first image area is one of the plurality of image areas;

    A first correction module configured to perform gamma correction on the first image region based at least in part on the first gamma curve.
16. The device according to claim 15, wherein the determining module is configured to:

    Obtain the mapping relationship between different gamma curves and the third visual feature representing the scene;

    determining an overall fourth visual characteristic of the first image region;

    determining a third visual feature matching the fourth visual feature from the third visual features representing the scene;

    A gamma curve corresponding to the matched third visual feature is determined as the first gamma curve.
17. The device according to claim 15, further comprising a second calibration module;

    The determining module is further configured to determine a third gamma curve corresponding to the first image area; wherein, the third gamma curve is obtained by calibration at the image display terminal, and the first gamma curve The curve is obtained by calibration at the image sensor end;

    The second correction module is configured to perform gamma correction on the first image area according to the third gamma curve.
18. The apparatus of claim 15, wherein,

    The determination module is further configured to determine a second gamma curve corresponding to a second image area; wherein the second image area is another one of the plurality of image areas that is adjacent to the first image area an image area; the first gamma curve is different from the second gamma curve;

    The first correction module is further configured to perform gamma correction on the second image region based on the second gamma curve.
19. An electronic device, comprising a memory and a processor, the memory stores a computer program that can run on the processor, and the processor implements the method according to any one of claims 1 to 13 when executing the program.
20. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images