CN113724144A - Image processing method and image signal processor on terminal equipment - Google Patents
Image processing method and image signal processor on terminal equipment Download PDFInfo
- Publication number
- CN113724144A CN113724144A CN202110379572.5A CN202110379572A CN113724144A CN 113724144 A CN113724144 A CN 113724144A CN 202110379572 A CN202110379572 A CN 202110379572A CN 113724144 A CN113724144 A CN 113724144A
- Authority
- CN
- China
- Prior art keywords
- image
- layer
- pyramid
- tone mapping
- layers
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 8
- 238000013507 mapping Methods 0.000 claims abstract description 67
- 238000000034 method Methods 0.000 claims description 38
- 230000003044 adaptive effect Effects 0.000 claims description 3
- 230000007613 environmental effect Effects 0.000 claims description 3
- 230000000694 effects Effects 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000012937 correction Methods 0.000 description 2
- 238000002679 ablation Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
Images
Classifications
-
- G06T5/90—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
Abstract
The present disclosure discloses an image processing method, comprising the steps of: acquiring initial image data from an image sensor; deconstructing the image data into N image pyramid layers, wherein N is an integer greater than 2; reconstructing an image pyramid layer with k-N and an image pyramid layer with k-N-1, wherein k is N. Reconstructing the intermediate layer and the subsequent image pyramid layer to create a new intermediate layer, and repeating the steps until the last intermediate layer is reconstructed with the image pyramid layer with k being 1 to generate a final image; wherein a tone mapping operator is applied to at least one intermediate layer and/or image pyramid layer.
Description
Technical Field
The present disclosure relates to an image processing method, in particular, a tone mapping process of an image, an Image Signal Processor (ISP) for performing the method, and a terminal apparatus including the ISP.
Background
Today's digital cameras and image sensors can only capture a limited range of dynamic ranges in real life. Furthermore, the viewing environment of the captured images, such as mobile device displays, computer displays, televisions, etc., may even support a narrower dynamic range than can be captured by digital cameras and image sensors.
To alleviate this problem, a tone mapping process is applied to the acquired image data. Tone mapping is the process of mapping image pixels representing a relatively high dynamic range to a viewing environment (i.e., a display medium) having a relatively low dynamic range. In this process, the tone mapping process is responsible for rendering the image as close as possible to the real-world scene. Where tone mapping is one of the key modules of image processing from the acquisition of image data to the presentation of the final image to the viewer, it is responsible for changing the contrast and brightness of the image to successfully convert/map the original high dynamic range of the real world to the image displayed on the low dynamic range display device.
In particular, it is well known that pyramid-based tone mapping algorithms can be used to enhance the dynamic range of an image. Thus, the image data is deconstructed into multiple (N) levels or layers. For example, in a Gaussian-pyramid algorithm, the first stage is a Gaussian (Gaussian) filtered image of the original image data, which has a lower resolution than the original image data. The second stage is a gaussian filtered image of the first stage, with a lower resolution relative to the first stage, and so on to the top stage. Other filters, such as Laplacian filters (Laplacian) filters, may be used instead of gaussian filters to deconstruct the initial image data into stages. Subsequently, the contrast of one or more of the stages is adjusted accordingly, and then from the top stage, the stages are reconstructed to form the final image with enhanced dynamic range. However, due to the limited dynamic range of the digital imaging sensor and the image viewing medium (i.e., the display device), the scene is typically underexposed, at the expense of underexposed shadow areas to avoid ablation or saturation of the highlight. Therefore, known mapping algorithms lead to final images with unacceptable consequences of loss of detail and insufficient dynamic range.
Disclosure of Invention
It is an object of the present disclosure to provide an image processing method, in particular a tone mapping method that improves the dynamic range of the final image.
The above problem is solved by the image processing method of claim 1, the ISP of claim 14 and the terminal device of claim 15.
The image processing method according to the present disclosure, in particular an image tone mapping algorithm, comprises the steps of:
acquiring initial image data from an image sensor;
decomposing the image data into N image pyramid layers, wherein N is an integer greater than 2;
reconstructing an image pyramid layer with k-N (i.e., a top layer) and an image pyramid layer with k-N-1, wherein k is N.
The intermediate layer and subsequent image pyramid layers are reconstructed to create a new intermediate layer, and this step is repeated until the last intermediate layer is reconstructed with the image pyramid layer with k 1 to generate the final image. Where the image pyramid layer with k-N is also denoted as the top layer, and the image pyramid layer with k-1 is denoted as the bottom layer. At the beginning of reconstruction, the image pyramid layer with k equal to N (i.e., the top layer) is reconstructed with the subsequent image pyramid layer with k equal to N-1. By reconfiguring the two layers, an intermediate layer is created. This intermediate layer is the starting point for the next reconstruction step, and is reconstructed with the subsequent image pyramid layer of k-N-2 to create a new intermediate layer, which is the starting layer for the next reconstruction step, and so on. This process continues until the image pyramid layer (i.e., the bottom layer) with k ═ 1 is reached. In this step, the latest intermediate layer is reconstructed with the image pyramid layer (i.e., bottom layer) with k ═ 1 to generate the final image. Wherein N is an integer greater than 2.
According to the present disclosure, a tone mapping operator is applied to at least one intermediate layer. In this manner, the hue of at least one intermediate layer produced by the above-described step of reconstructing the N image pyramid layers is changed or manipulated. By applying a tone mapping operator to the intermediate layer, the display of the final image can be customized in an intuitive and user-pleasing manner in the final image by influencing the desired tone (brightness, intermediate tone or shading) of the initial image data. Therefore, shadow areas can be brightened by the tone mapping operator of the present disclosure, dark objects in relatively bright areas can be unaffected, neither artifacts nor loss of contrast can be created.
Optionally, the tone mapping operator is applied to at least two or more intermediate layers, even more optionally to each intermediate layer. Thus, tone mapping is distributed over the intermediate layers to achieve the desired result. Among other things, tone mapping may be assigned according to a control algorithm that may analyze the content of an image through image statistics (e.g., a histogram of the image), or may perform AI-based image classification/analysis.
Optionally, a tone mapping operator is applied to the initial image data and/or the final image to further adjust the display of the final image. Additionally or alternatively, the tone mapping operator is applied to the top layer, i.e., the image pyramid layer with k ═ N.
Optionally, contrast manipulation is applied to at least one, optionally to more than one, more optionally to each of the N image pyramid layers. Additionally or alternatively, contrast manipulation is applied to at least one, optionally to more than one, and more optionally to each of the intermediate layers. Thus, by applying contrast manipulation to at least one image pyramid layer or intermediate layer, the contrast of the final image may be manipulated and/or the dynamic range of the final image may be further enhanced.
Optionally, a tone mapping operator is applied locally to the sub-regions of each intermediate layer. Wherein a sub-region of the image data is a region smaller than the area of the complete image. In particular, the tone mapping operator is applied to more than one sub-region of one of the respective intermediate layers. Alternatively, the tone mapping operator is applied globally to the complete respective middle layer. Thus, either local or global tone mapping is feasible in order to specifically control the effect of the final image. Wherein the local tone mapping may be applied to one or more intermediate layers while the global tone mapping is applied to at least one or more other intermediate layers. Thus, the local tone mapping and the global tone mapping can be freely selected and applied to different intermediate layers.
Optionally, contrast manipulation is applied locally to sub-regions of the image pyramid layer and/or to sub-regions of the intermediate layer. Wherein the sub-regions of the image pyramid layer and/or the sub-regions of the intermediate layer are regions having a smaller area than the complete image pyramid layer and/or the intermediate layer. In particular, the contrast manipulation is applied to more than one sub-region of the respective image pyramid layer and/or more than one sub-region of the respective intermediate layer. Alternatively, contrast manipulation is applied globally to the complete individual image pyramid layers and/or intermediate layers. Thus, local or global contrast manipulation is possible in order to specifically control the effect of the final image. Wherein local contrast manipulation may be applied to one or more intermediate layers and/or one or more image pyramid layers while global contrast manipulation is applied to at least one or more other intermediate layers and/or image pyramid layers. Thus, the local contrast adjustment and the global contrast adjustment can be freely chosen, and preferably in addition to the above-described brightness manipulation, the contrast adjustment is applied to different intermediate layers and/or image pyramid layers.
Optionally, a tone mapping operator is additionally applied to the final image. Additionally or alternatively, a tone mapping operator is applied to the initial image data.
Optionally, contrast manipulation is additionally applied to the final image. Additionally or alternatively, contrast manipulation is applied to the initial image data. Additionally or alternatively, contrast manipulation is applied to the top layer, i.e., the image pyramid layer with k ═ N.
Optionally, the reconstructing is performed by:
Imgk(i，j)＝Lk(i, j), wherein k is Top _ level
Where k is N-1, 1.,
for the nth layer:
Imgk(i，j)＝Upscale(Imgk+1(i，j))+Lk(i，j)，
wherein L isk(i, j) is an image pyramid layer of k layers, UPSCALE is a resolution adaptive function between the k +1 layer and the k layer, IMGk(i, j) are the intermediate layers, IMGk＝1Is the final image. Here, i, j is used to describe the position of each pixel in the image.
Alternatively, tone-mapping (tone-mapping) is applied as follows:
Imgk(i，j)＝ToneMapping(Imgk(i，j))。
IMG if tone mapping is applied to one of the intermediate layerskIf tone mapping is applied directly to one of the image pyramid layers Lk(i, j) then, tone mapping will be applied as follows: l isk(i，j)＝ToneMapping(Lk(i，j))。
Optionally, the image data is deconstructed into Laplacian-pyramids (Laplacian-pyramids). Here, for Laplacian-Pyramid, alternatively, the image Pyramid layer with k ═ 1 is a Laplacian filtered image with the resolution of the original image data reduced, and each image Pyramid layer subsequent to the top layer is a Laplacian filtered image with the resolution of the previous layer reduced. Of course, different filters may be used to deconstruct the initial image data into image pyramids. In particular, different filters may be used in combination during deconstruction of the initial image data.
Optionally, the image is deconstructed into multiple image pyramid layers, where the top layer, i.e., the image pyramid layer with k equal to N in the pyramid, is a low resolution representation of the original image, while the other layers below carry edge information. Some examples may be pyramids based on Laplacian (Laplacian) transforms and Wavelet (Wavelet) transforms.
Optionally, the tone mapping operator is implemented as a luminance manipulation.
Optionally, the brightness manipulation is realized by a functional relationship given by one of the following predefined functions: e.g. a Gamma function, e.g. out inGamma(also known as Gamma correction); or a contrast enhanced Sigmoid function, as given by the following equation:
where p is a given or preset parameter. Here, "in" denotes an input pixel, and "out" denotes an output pixel of each functional relationship.
Optionally, the determination of the applied tone mapping operator, in particular the applied luminance manipulation, depends on one or more parameters of the scene content, such as high/low contrast scenes, portrait or landscape scenes, indoor scenes, light sources, moving objects, etc. Thus, the tone mapping operator or brightness manipulation may be adapted to specific user requirements and/or given circumstances to achieve the desired effect of the final image.
Optionally, the applied tone mapping operator, in particular the applied luminance manipulation, is determined by one or more parameters of the environmental properties, such as low/high dynamic range, low/high lux, total gain of the image sensor, noise of the image, luminance histogram and user preferences. Thus, the tone mapping operator or brightness manipulation may be adapted to specific user requirements and/or given circumstances to achieve the desired effect of the final image.
Optionally, the brightness manipulation is implemented with a Look-up Table (LUT). The lookup table may be a predefined functional relationship, or may be generated according to one or more of scene content, environmental attributes, overall gain of the image sensor, noise of the image sensor, luminance histogram, and user preferences. Thus, by means of the look-up table, a fast method is provided for applying the brightness manipulation to the respective intermediate layer or the respective image pyramid layer.
Optionally, an overall tone mapping, in particular an overall luminance manipulation, between the initial image data and the final image is assigned between one or more intermediate layer tone mappings. Thus, the overall tone mapping, in particular the overall luminance manipulation, may be distributed between at least two intermediate layers. Here, in particular the overall tone mapping and in particular the overall luminance manipulation may be evenly distributed such that the same tone mapping/luminance manipulation is applied to each intermediate layer. Alternatively, at least two tone mappings or luminance manipulations may be unequal, then the total luminance manipulation will be unevenly distributed to meet further requirements and provide a higher degree of freedom to achieve the desired effect of the final image. Here, the assignment may be determined according to scene content, such as overall brightness of the initial image data. Alternatively, for brighter images, the brightness manipulation may be evenly distributed; whereas for darker images the brightness manipulation may be unevenly distributed between different intermediate layers. Here, the total tone mapping may be assigned according to a control algorithm that may analyze the contents of an image through image statistics such as a histogram of the image, or may perform AI-based image classification/analysis. Furthermore, for the luminance processing, if the LUT can be distributed on each layer, it is possible not only to decide whether or not to modify the bright portion, the halftone, or the shadow, but also to control the above-described adjustment manners and the position areas to which the above-described adjustments are made based on the image analysis. This can be achieved by the solution proposed by the present disclosure, for example, if the decision based on the image content analysis indicates that small details in the shadow should be suppressed and larger details should be enhanced.
Furthermore, it is an object of the present disclosure to provide an Image Signal Processor (ISP) configured to perform the steps of the above method.
Furthermore, it is an object of the present disclosure to provide a terminal device comprising an ISP configured to perform the aforementioned method steps, and an image sensor connected to the ISP to acquire image data and provide the image data to the ISP for further processing.
Drawings
The disclosure is further described with reference to the accompanying drawings. The attached drawings are as follows:
FIG. 1 is an example of a pyramid-based tone mapping process;
FIG. 2 is a first embodiment according to the present disclosure;
FIG. 3 is a second embodiment according to the present disclosure;
FIG. 4 is a determination of a look-up table according to the present disclosure;
FIG. 5 is an assignment of brightness manipulation according to the present disclosure;
fig. 6A to 6C are comparisons of results according to the present disclosure;
FIG. 7 is a detailed analysis of a comparison of FIGS. 6A through 6C;
8A-8C are examples of final images according to the present disclosure;
fig. 9 is an apparatus according to an embodiment of the present disclosure.
Detailed Description
Fig. 1 shows an example of a Pyramid-based tone mapping process that may be implemented by Laplacian-Pyramid (Laplacian-Pyramid) based tone mapping with an input image as initial image data, preferably from an image sensor. The image data is deconstructed into an image pyramid having N image pyramid layers. In the example given in fig. 1, N is 4, where for a Laplacian-Pyramid based image Pyramid, a Laplacian filter is applied to the initial image data of the input image, where the input image is the image Pyramid layer with k being 1, i.e. the bottom layer. Subsequently, a laplacian filter is applied to the bottom layer, and a downsampling step is combined to generate a next image pyramid layer, namely an image pyramid layer with k being 2. This step will be further continued until the image pyramid, i.e. the top level, is reached where k ═ N. The number N may be selected depending on the resolution of the input image, or may depend on other parameters such as computational power. Then, reconstructing the Laplacian image pyramid layer by layer from the top layer to the bottom layer; and in the reconstruction process, brightness control is carried out on the intermediate layer generated in the reconstruction process. When all the layers of the laplacian image pyramid are reconstructed, the resulting image includes an improved dynamic range. Where the shadow areas are illuminated, dark objects in the relatively bright areas will remain unchanged and will not degrade the contrast in the image.
Fig. 2 is a schematic diagram of the method of the present disclosure. In a first step S01, initial image data is acquired from the image sensor.
In step S02, the initial image data is deconstructed into N image pyramid layers, where N is an integer greater than 2.
In step S03, an image pyramid layer with k equal to N and an image pyramid layer with k equal to N-1 are first reconstructed to create an intermediate layer; then, the intermediate layers are reconstructed with subsequent image pyramid layers of k-N-2 to create a new intermediate layer, and this step is repeated for k-N-2,.., 1 until the last intermediate layer is reconstructed with an image pyramid layer of k-1 to generate the final image, with the tone mapping operator applied to at least one intermediate layer.
The reconstruction of the laplacian pyramid is described as follows:
Imgk(i，j)＝Upscale(Imgk+1(i，j))+Lk(i, j) wherein k is N-1
Imgk(i，j)＝Lk(i, j), where k ═ N, the top layer.
Wherein L isk(i, j) is the image pyramid layer of the k-th layer, and UPSCALE is the resolution adaptive function between the image pyramid layer of the k + 1-th layer and the image pyramid layer of the k-th layer. IMGkIs an intermediate layer, wherein k ═ N.., 1, IMGk＝1Is the final image. In addition, i, j represents the pixel position of each image.
In the process of reconstructing the image pyramid, after a certain level is reconstructed and a corresponding intermediate layer is generated, before the next level is reconstructed, tone mapping is performed on the generated intermediate layer. This additional step is achieved as follows:
Imgk(i，j)＝ToneMapping(Imgk(i，j))。
here, the tone mapping operator may be applied to one of the plurality of intermediate layers. Optionally, the tone mapping operator is applied to more than one intermediate layer, and optionally to each intermediate layer. In this case, the tone mapping operator is applied to a previous intermediate layer before an image pyramid layer is reconstructed with the previous intermediate layer. This is schematically depicted in fig. 3, where in step S031, the image pyramid layer with k-N and the image pyramid layer with k-N-1 are reconstructed to create intermediate layers. In step S032, a first tone mapping operator is applied to the middle layer. Thereafter, in step S033, the adjusted intermediate layer is reconstructed with the image pyramid layer of k — N-2 to create a new intermediate layer. In step S034, a further tone mapping operator is applied to the new intermediate layer before it is reconstructed with the next image pyramid layer. These steps are repeated until the last intermediate layer is reconstructed with the image pyramid layer with k ═ 1 (i.e., the bottom layer) in step S035 to generate a final image. Thus, after each step of creating a new intermediate layer by reconstructing the previous intermediate layer with the pyramid image layers, a tone mapping operator is applied to this new intermediate layer.
Here, the tone mapping operator may be implemented as a luminance manipulation. The brightness manipulation may be provided as one of: a functional relationship, a Gamma function (also known as Gamma correction), or a contrast enhanced Sigmoid function.
Optionally, the brightness manipulation is provided in the form of a look-up table (LUT).
As shown in fig. 4, the brightness manipulation may be implemented as a brightness control configured to generate a LUT for each layer to be applied independently on the layer. Among them, the following may be considered for the luminance control: user preferences or manual controls of the application, tuning/configuration parameters, and frame information such as lux, gain, exposure information, histogram, etc. Together with the brightness control, the LUT applied to each layer is generated based on the above information. However, the present disclosure is not limited to a particular LUT or brightness manipulation.
The overall or total brightness manipulation applied on the image to achieve the desired display effect may be distributed in different layers. This is illustrated in fig. 5. At the top of fig. 5, the total brightness manipulation applied to the image is taken as an example. Different LUTs may be implemented as previously described. The overall brightness manipulation is distributed between the brightness manipulations of one or more intermediate layers. Wherein the overall brightness manipulation may be evenly distributed between the different layers. In addition, the overall brightness manipulation may be applied to different intermediate layers in different ways. In this case, the LUTs of different layers are different.
FIG. 6A shows an example of a method according to the present disclosure to progressively enhance the shadow region from left to right by using the luminance manipulation assigned in the intermediate layers during reconstruction of the image pyramid. In which the contrast of the image is preserved and only the shadow areas are highlighted, thereby enhancing the dynamic range and providing a display effect of a real scene that is satisfactory to the user.
By way of comparison, fig. 6B shows that applying luminance manipulation to the top layer of the laplacian pyramid to gradually enhance the shadow region from left to right will result in uneven luminance distribution and artifacts, i.e., produce under-exposed regions. Similarly, in fig. 6C, it is shown that only the final image after the pyramid reconstruction is applied with brightness control to gradually enhance the shadow region from left to right, which will increase the brightness of the image, but not increase the dynamic range of the image.
Fig. 7A shows a ratio image between the middle image (medium luminance) and the right image (brightest) in fig. 6A. The gray areas show pixels that differ significantly from the original image, and the dark pixels are less affected. Similarly, fig. 7B shows a ratio image between the middle image (medium luminance) and the right image (brightest) of fig. 6B, and fig. 7C shows a ratio image between the middle image (medium luminance) and the right image (brightest) of fig. 6C. As can be seen from this comparison, with the method of the present disclosure, the bright/dark regions can be well separated by applying the brightness manipulation to the intermediate layer when performing the reconstruction, so that the brightness manipulation only affects the dark region, while the bright region remains unchanged to preserve the contrast. As shown in fig. 7A, the present disclosure is superior to the other alternatives shown in fig. 6B, 6C and 7B, 7C, respectively. Under-exposed regions and contrast loss appear in fig. 7B and 7C. Fig. 8A, 8B and 8C show final images obtained according to the methods corresponding to fig. 6A, 6B and 6C, respectively. Fig. 8A shows the final image of the present disclosure, with high dynamic range without loss of contrast, and with uniformly distributed luminance without loss of detail or under-exposed areas in the image. Fig. 8B and 8C present vignetting or contrast degradation, while the method of the present disclosure shown in fig. 8A is capable of manipulating brightness without creating artifacts and without loss of contrast.
Referring now to fig. 9, there is shown an apparatus 10 for carrying out the above method. Alternatively, the above method is implemented in a processor 12, such as an ISP, an application-specific integrated circuit (ASIC), and a Field Programmable Gate Array (FPGA), a general purpose processor, or a graphics processor. In addition, the method may be implemented by hardware or software. Optionally, the processor 12 is connected to the image sensor 14. The image sensor 14 may be a Charge-Coupled Device (CCD) sensor, a camera or the like. Initial image data may be acquired by the image sensor 14. This initial image data from the image sensor 14 is communicated to the processor 12 for processing in accordance with the present disclosure. Optionally, the processor is also connected to a display 16 to display the final image.
The device 10 may be implemented by any kind of terminal, such as a digital camera, a smart phone, a tablet computer, a laptop or a computer, etc. Furthermore, although image sensor 14, processor 12, and display 16 are shown in FIG. 9 as being implemented in one device 10, the various modules may be implemented in more than one device. Thus, the image sensor 14 may be implemented in a smartphone to take a photograph. The initial image data acquired by the image sensor 14 may then be transmitted to a server or any other computing device over a communication connection. The server or other computing device may include a processor 12 to perform methods according to embodiments of the present disclosure. The final image may then be transferred to a smartphone, tablet, or any other device that includes the display device 16 to display the final image. Here, the final image from the server or other computing device may be transmitted to the same device or other device that acquired the initial image data. Optionally, the final image may optionally be stored in cloud storage or any other storage device and then transferred to a display device as needed to display the final image.
Claims (15)
1. An image processing method comprising:
acquiring initial image data from an image sensor;
decomposing the initial image data into N image pyramid layers, wherein N is an integer greater than 2;
reconstructing the image pyramid layer with k-N and the image pyramid layer with k-N-1 to create an intermediate layer, wherein k is N, …, 1;
reconstructing the intermediate layer and the subsequent image pyramid layer to create a new intermediate layer, and repeating the steps until the last intermediate layer is reconstructed with the image pyramid layer with k being 1 to generate a final image;
wherein a tone mapping operator is applied to at least one intermediate layer.
2. The method of claim 1, wherein the tone mapping operator is applied to each of the intermediate layers and/or each of the N image pyramid layers.
3. The method according to claim 1 or 2, characterized in that the tone mapping operator is applied locally to sub-regions of each of the intermediate layers and/or to sub-regions of each of the image pyramid layers.
4. A method according to claim 1 or 3, wherein the tone mapping operator is applied globally to the complete intermediate layers.
5. The method according to any of claims 1 to 4, wherein the tone mapping operator is applied to at least one of: the final image; the image data; n of the image pyramid layers.
6. The method according to any one of claims 1 to 5, characterized in that the reconstruction is performed by:
IMGk(i，j)＝UPSCALE(IMGk+1(i，j))+Lk(i，j)，
wherein k is N-1, and, IMGk(i，j)＝Lk(i, j) the image pyramid layer for the Nth layer, wherein Lk(i, j) is the image pyramid layer of the k-th layer, UPSCALE is the resolution adaptive function between the image pyramid layer of the k + 1-th layer and the image pyramid layer of the k-th layer, IMGk(i, j) are each of the intermediate layers, IMGk＝1Is the final image.
7. The method according to any of claims 1 to 6, wherein the tone mapping operator is applied in at least one of the following ways:
IMGk(i，j)＝ToneMapping(IMGk(i，j))；
Lk(i，j)＝ToneMapping(Lk(i，j))。
8. the method according to any one of claims 1 to 7, wherein the image data is deconstructed into a Laplacian-Pyramid, wherein k 1 layer is a Laplacian-filtered image with reduced resolution of the image data and subsequent layers are Laplacian-filtered images with reduced resolution of previous layers.
9. The method according to any of claims 1 to 8, characterized in that the tone mapping operator is implemented as a luminance manipulation.
10. The method of claim 9, wherein the brightness manipulation is implemented as one of: functional relationships, Gamma functions, contrast enhanced Sigmoid functions.
11. The method according to claim 9 or 10, wherein the brightness manipulation is determined by at least one of the following parameters: scene content, environmental attributes, overall gain of the image sensor, noise of the image sensor, luminance histogram, and user preferences.
12. Method according to any of claims 9 to 11, wherein the brightness manipulation is implemented as a look-up table.
13. A method according to any of claims 1 to 12, wherein the total tone mapping between the image data and the final image is evenly distributed between more than one intermediate layer tone mapping.
14. An image signal processor ISP configured to perform the steps of the method of any one of claims 1 to 13.
15. A terminal device comprising the image signal processor of claim 14 and an image sensor connected to the image signal processor for acquiring the image data and providing the image data to the image signal processor.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EPEP201761012 | 2020-05-22 | ||
| EP17061012 | 2020-05-22 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113724144A true CN113724144A (en) | 2021-11-30 |
Family
ID=78672606
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110379572.5A Pending CN113724144A (en) | 2020-05-22 | 2021-04-08 | Image processing method and image signal processor on terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113724144A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116452454A (en) * | 2023-04-19 | 2023-07-18 | 哈尔滨理工大学 | Multi-resolution pyramid-based tone mapping method |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07212751A (en) * | 1994-01-12 | 1995-08-11 | Fuji Photo Film Co Ltd | Method and device for forming picture data in hierarchical picture file |
| US20040101207A1 (en) * | 2002-11-27 | 2004-05-27 | General Electric Company | Method and system for improving contrast using multi-resolution contrast based dynamic range management |
| US20120082397A1 (en) * | 2009-06-29 | 2012-04-05 | Thomson Licensing | Contrast enhancement |
| CN104091309A (en) * | 2014-06-19 | 2014-10-08 | 华南理工大学 | Balanced display method and system for flat-plate X-ray image |
| CN104574284A (en) * | 2013-10-24 | 2015-04-29 | 南京普爱射线影像设备有限公司 | Digital X-ray image contrast enhancement processing method |
| CN104574361A (en) * | 2014-11-27 | 2015-04-29 | 沈阳东软医疗系统有限公司 | Image processing method and device for mammary peripheral tissue equalization |
| CN106570831A (en) * | 2016-10-09 | 2017-04-19 | 中国航空工业集团公司洛阳电光设备研究所 | Gray image contrast equalization enhancement method |
| CN107292804A (en) * | 2017-06-01 | 2017-10-24 | 西安电子科技大学 | Direct many exposure fusion parallel acceleration methods based on OpenCL |
| CN110021031A (en) * | 2019-03-29 | 2019-07-16 | 中广核贝谷科技有限公司 | A kind of radioscopic image Enhancement Method based on image pyramid |
-
2021
- 2021-04-08 CN CN202110379572.5A patent/CN113724144A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07212751A (en) * | 1994-01-12 | 1995-08-11 | Fuji Photo Film Co Ltd | Method and device for forming picture data in hierarchical picture file |
| US20040101207A1 (en) * | 2002-11-27 | 2004-05-27 | General Electric Company | Method and system for improving contrast using multi-resolution contrast based dynamic range management |
| US20120082397A1 (en) * | 2009-06-29 | 2012-04-05 | Thomson Licensing | Contrast enhancement |
| CN102473290A (en) * | 2009-06-29 | 2012-05-23 | 汤姆逊许可证公司 | Contrast enhancement |
| CN104574284A (en) * | 2013-10-24 | 2015-04-29 | 南京普爱射线影像设备有限公司 | Digital X-ray image contrast enhancement processing method |
| CN104091309A (en) * | 2014-06-19 | 2014-10-08 | 华南理工大学 | Balanced display method and system for flat-plate X-ray image |
| CN104574361A (en) * | 2014-11-27 | 2015-04-29 | 沈阳东软医疗系统有限公司 | Image processing method and device for mammary peripheral tissue equalization |
| CN106570831A (en) * | 2016-10-09 | 2017-04-19 | 中国航空工业集团公司洛阳电光设备研究所 | Gray image contrast equalization enhancement method |
| CN107292804A (en) * | 2017-06-01 | 2017-10-24 | 西安电子科技大学 | Direct many exposure fusion parallel acceleration methods based on OpenCL |
| CN110021031A (en) * | 2019-03-29 | 2019-07-16 | 中广核贝谷科技有限公司 | A kind of radioscopic image Enhancement Method based on image pyramid |
Non-Patent Citations (3)
| Title |
|---|
| YINGHUI ZHANG，KE WANG，XIAOJUAN DENG，HONGWEI LI: "Multiscale Morphological Tone Mapping Operator for High Dynamic Range Images", THIRD INTERNATIONAL SYMPOSIUM ON IMAGE COMPUTING AND DIGITAL MEDICINE (ISICDM 2019), 26 August 2019 (2019-08-26), pages 254 - 258, XP058447612, DOI: 10.1145/3364836.3364887 * |
| 刘颖;张凤;伍世虔;刘卫华;: "高动态范围图像全局与局部色调映射的融合", 传感器与微系统, vol. 35, no. 9, 30 September 2016 (2016-09-30), pages 118 - 120 * |
| 梁云，莫俊彬: "改进拉普拉斯金字塔模型的高动态图像色调映射方法", 计算机辅助设计与图形学学报, vol. 26, no. 12, 31 December 2014 (2014-12-31), pages 2182 - 2188 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116452454A (en) * | 2023-04-19 | 2023-07-18 | 哈尔滨理工大学 | Multi-resolution pyramid-based tone mapping method |
| CN116452454B (en) * | 2023-04-19 | 2023-10-03 | 哈尔滨理工大学 | Multi-resolution pyramid-based tone mapping method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5105209B2 (en) | Image processing apparatus and method, program, and recording medium | |
| EP2076013B1 (en) | Method of high dynamic range compression | |
| US9218653B2 (en) | Method and apparatus for dynamic range enhancement of an image | |
| JP5012333B2 (en) | Image processing apparatus, image processing method, and imaging apparatus | |
| JP4319620B2 (en) | Local color correction | |
| US7020332B2 (en) | Method and apparatus for enhancing a digital image by applying an inverse histogram-based pixel mapping function to pixels of the digital image | |
| US20070024721A1 (en) | Compensating for improperly exposed areas in digital images | |
| CN111986129B (en) | HDR image generation method, equipment and storage medium based on multi-shot image fusion | |
| KR20070046010A (en) | Method of digital image enhancement | |
| CN112655195A (en) | Entropy variance based automatic exposure of image sensors | |
| US11017511B2 (en) | Method and system of haze reduction for image processing | |
| JP6771977B2 (en) | Image processing equipment and image processing methods, programs | |
| US20220101503A1 (en) | Method and apparatus for combining low-dynamic range images to a single image | |
| JP2011086976A (en) | Image processor and image processing method | |
| RU2622095C1 (en) | Method of improving digital colour images | |
| CN113724144A (en) | Image processing method and image signal processor on terminal equipment | |
| JP2011100204A (en) | Image processor, image processing method, image processing program, imaging apparatus, and electronic device | |
| JP6335614B2 (en) | Image processing apparatus, control method thereof, and program | |
| KR102477391B1 (en) | Method for image processing, image signal processor and termainal device | |
| Wang et al. | Adaptive enhancement for non-uniform illumination images via pixel-wise histogram modification and color reconstruction | |
| JP2017068513A (en) | Image processing device and method thereof, program, and storage medium | |
| US20220318966A1 (en) | Method and device for image processing, and storage medium | |
| Jung et al. | Enhanced HDR Image Reproduction Using Gamma-Adaptation-Based Tone Compression and Detail-Preserved Blending | |
| CN116233332A (en) | Image processing apparatus, image processing method, generation method, and storage medium | |
| Hsin et al. | Improving image luminance appearance through recurrent local intensity adaptation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |