KR102034968B1 - Method and apparatus of image processing - Google Patents

Abstract

An image processing method according to an embodiment may include receiving an image, performing a first convolution operation to decompose the image, and performing a second convolution operation to generate a first feature map from a portion of the decomposed image. And extracting a second feature map from another portion of the decomposed image, and generating an HDR image by performing a third convolution operation on the first feature map and the second feature map.

Description

Image processing method and apparatus {METHOD AND APPARATUS OF IMAGE PROCESSING}

The embodiments below relate to an image processing method and apparatus.

The human visual system recognizes in greater detail than is normally offered in a standard dynamic range (SDR) display, and perceives the world much brighter with stronger contrast.

With the recently launched High Dynamic Range (HDR) consumer display, brightness above 1000 cd / m ² (100 cd / m ^{2 for} SDR displays), high contrast ratio, and bit depth increase above 10 increased bit depth and Wide Color Gamut (WCG). However, while HDR TVs are readily available in the market, there is a serious lack of HDR content.

Inverse Tone Mapping (ITM) or Reverse Tone Mapping is a field of research in computer graphics that aims to predict HDR images from low dynamic range (LDR) images for better graphics rendering.

Another area of research, HDR imaging, uses multiple LDR images of different exposures to create a single HDR image that contains details of a saturated region. In both areas of research, lighting calculations are performed in the HDR domain with the belief that they will more accurately represent graphical or natural scenes on SDR displays. HDR images can be viewed using a professional HDR monitor while rendering.

As a result, the HDR domain mentioned in the above-mentioned area is not necessarily the same as the currently available HDR consumer display, and the HDR image generated by the ITM method for that purpose is not suitable for viewing on HDR TV.

When the conventional ITM method is applied to an HDR TV with a maximum brightness of 1000 cd / m ² , the HDR capacity cannot be fully utilized due to weak contrast or detail, or amplification of noise.

In other words, conventional inverse tone mapping methods convert LDR images to HDR to work in the HDR domain for the purpose of more natural graphic rendering, mainly render in the HDR domain, and finally tone-map them to LDR to convert the image. Because of the purpose of viewing, reverse tone-mapped HDR video is not suitable for viewing directly on an HDR TV.

The HDR image obtained by the prior art has a problem that noise is amplified in the dark portion, or the contrast or detail of the image is not sufficiently represented.

 The prior art does not consider the color container (for example, BT. 2020) and the transfer function (for example, PQ-EOTF), there is a problem that it must be converted manually in order to apply to HDR TV.

Embodiments may provide a technique for processing an LDR image into an HDR image.

An image processing method according to an embodiment may include receiving an image, performing a first convolution operation to decompose the image, and performing a second convolution operation to generate a first feature map from a portion of the decomposed image. And extracting a second feature map from another portion of the decomposed image, and generating an HDR image by performing a third convolution operation on the first feature map and the second feature map.

The generating of the HDR image may include combining the first feature map and the second feature map, and performing the third convolution operation on the combined first feature map and the second feature map to generate an HDR image. It may include the step.

The image processing method may further include learning filter parameters of the first, second and third convolution operations.

The training may include learning filter parameters of the second and third convolution operations based on the image, and learning the filter parameters of the second and third convolution operations. Learning the filter parameters of the operation, and after learning the filter parameters of the first convolution operation, learning the filter parameters of the first to third convolution operations.

Learning the filter parameters of the second and third convolution operations may include generating a portion of the disassembled image by filtering the image, and generating another portion of the disassembled image based on the image and the portion. Generating and learning filter parameters of the second and third convolution operations based on the portion and the other portion.

Generating the portion may include generating the portion by edge preserving filtering the image.

The generating of the other part may include generating the other part by performing element-wise division on the image and the part.

Learning the filter parameters of the first to third convolutional operations comprises: end-to-end a network where the parameters of the learned first convolutional operations and the filter parameters of the learned second and third convolutional operations are combined; -to-end).

According to an embodiment, an image processing apparatus includes a receiver for receiving an image, a first convolution operation to decompose the image, and a second convolution operation to extract a first feature map from a part of the decomposed image And a processor configured to extract a second feature map from another part of the decomposed image, and generate an HDR image by performing a third convolution operation on the first feature map and the second feature map.

The processor may generate the HDR image by combining the first feature map and the second feature map, and performing the third convolution operation on the combined first feature map and the second feature map.

The processor may learn filter parameters of the first, second and third convolution operations.

The processor learns the filter parameters of the second and third convolution operations based on the image, and after learning the filter parameters of the second and third convolution operations, filters the filter parameters of the first convolution operation. After learning and learning the filter parameters of the first convolution operation, the filter parameters of the first to third convolution operations may be learned.

The processor generates a portion of the disassembled image by filtering the image, generates another portion of the disassembled image based on the image and the portion, and generates the second portion based on the portion and the other portion. And filter parameters of the third convolution operation.

The processor may generate the portion by edge preserving filtering the image.

The processor may generate the other part by performing element-wise division on the image and the part.

The processor may end-to-end learn a network in which the parameters of the learned first convolution operation and the filter parameters of the learned second and third convolution operations are combined.

1 is a schematic block diagram of an image processing apparatus according to an exemplary embodiment.
FIG. 2 illustrates a structure of a neural network used by the image processing apparatus shown in FIG. 1.
FIG. 3 illustrates an operation of training the neural network shown in FIG. 2.
4A illustrates an example of the LDR image shown in FIG. 3.
4B illustrates an example of the base layer illustrated in FIG. 3.
4C illustrates an example of the detail layer shown in FIG. 3.
5A shows an example of a network that decomposes an image.
5B shows another example of a network for decomposing an image.
5C shows another example of a network for decomposing an image.
6A shows an example of an unprocessed original image
6B illustrates an example of an image processed by the image processing apparatus.
7A shows another example of an unprocessed original image
7B shows another example of an image processed by the image processing apparatus.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, various changes may be made to the embodiments so that the scope of the patent application is not limited or limited by these embodiments. It is to be understood that all changes, equivalents, and substitutes for the embodiments are included in the scope of rights.

The terminology used herein is for the purpose of description and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The terms are intended only to distinguish one component from another, for example, without departing from the scope of the rights according to the concepts of the embodiment, the first component may be called a second component, and similarly The second component may also be referred to as the first component.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

In addition, in the following description with reference to the accompanying drawings, the same components regardless of reference numerals will be given the same reference numerals and redundant description thereof will be omitted. In the following description of the embodiment, when it is determined that the detailed description of the related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1 is a schematic block diagram of an image processing apparatus according to an exemplary embodiment.

Referring to FIG. 1, the image processing apparatus 10 may receive and process an image. The image includes an image of an object made by refraction or reflection of light, and may mean a shape of an object using lines or colors. For example, an image may consist of information in a form that can be processed by a computer.

The image processing apparatus 10 may generate a high dynamic range (HDR) image by processing a low dynamic range (LDR) image (hereinafter, referred to as a received image or an image).

The image processing apparatus 10 may process the received image by using a neural network. For example, the image processing apparatus 10 may generate the HDR image 500 by performing inverse tone mapping on the received image using a neural network.

Tone mapping is used to map the set of colors to another set in image processing and computer graphics to approximate the appearance of the HDR image 500 in a medium with limited dynamic range. It may mean a technique used.

Prints, displays, and projectors all have limited dynamic range and may not be suitable for reproducing all the light intensities present in natural scenes. Tone mapping may be to solve the problem of large contrast reduction from the brightness of the scene to the displayable range while maintaining the image detail and color appearance that are important for viewing the original scene content.

Inverse tone mapping (ITM) may mean generating the HDR image 500 from the LDR image 400 in the opposite concept of tone mapping.

The image processing apparatus 10 extracts the characteristics of the input LDR image through the deep convolution network and restores the HDR image based on the image, and thus, the HDR image for viewing on the HDR TV unlike the conventional computer graphic reverse tone mapping methods. Can produce

 The convolutional network used by the image processing apparatus 10 may decompose an LDR image using a filter (for example, a guided filter), and train the second half of the convolutional neural network first using the decomposed image. . Subsequently, a plurality of convolutional layers may be added instead of the filter, and only the added layer may be preliminarily trained, and then the entire convolutional neural network structure may be finally trained.

The image processing apparatus 10 may effectively learn an end-to-end convolutional network structure through the above-described learning method, and finally may obtain an HDR image having more improved image quality.

The image processing apparatus 10 includes a receiver 100 and a processor 200. The image processing apparatus 10 may further include a memory 300.

The receiver 100 may receive an image. For example, the image may include an LDR image 400. The receiver 100 may receive an image from the outside of the image processing apparatus 10 or may receive an image from the memory 300.

The processor 200 may process the image by performing a convolution operation. In detail, the processor 200 may generate the HDR image 500 by processing the image through a convolution operation of three parts.

The three parts may include a decomposing part of the image, a part extracting a feature from the decomposed image, and a part generating the HDR image 500 using the extracted feature.

The processor 200 may decompose an image by performing a first convolution operation. The processor 200 may extract a first feature map from a part of the decomposed image by performing a second convolution operation, and extract a second feature map from another part of the decomposed image.

The processor 200 may generate the HDR image 500 by performing a third convolution operation on the first feature map and the second feature map. The processor 200 may generate the HDR image 500 by combining the first feature map and the second feature map and performing a third convolution operation on the combined first feature map and the second feature map.

The processor 200 may train a neural network. For example, the processor 200 may learn filter parameters of the first, second, and third convolution operations. The processor 200 may learn the filter parameters of the first, second, and third convolution operations in a certain order.

For example, the processor 200 may learn filter parameters of the second and third convolution operations based on the image. After learning the filter parameters of the second and third convolution operations, the processor 200 may learn the filter parameters of the first convolution operation after the step.

The processor 200 may generate a part of the decomposed image by filtering the received image in learning the filter parameters of the second and third convolution operations. For example, the processor 200 may generate a portion of the decomposed image by edge preserving filtering the received image.

The processor 200 may generate another part of the decomposed image based on the image and the part of the decomposed image. For example, the processor 200 may generate another part of the decomposed image by performing element-wise division on the image and the part of the decomposed image.

The processor 200 may learn filter parameters of the second and third convolution operations based on the part and the other part of the disassembled image. The operation of the processor 200 to learn the second and third convolution operations using the part and the other part of the disassembled image will be described in detail with reference to FIG. 3.

After learning the filter parameters of the first convolution operation, the processor 200 may learn the filter parameters of the first to third convolution operations. The processor 200 may end-to-end learn a network in which parameters of the learned first convolution operation and filter parameters of the learned second and third convolution operations are combined.

The memory 300 may store instructions to be executed by the processor 200, filter parameters of a neural network, a feature map, and the like. In addition, the memory 300 may store the LDR image 400 and the HDR image 500.

FIG. 2 illustrates a structure of a neural network used by the image processing apparatus shown in FIG. 1.

Referring to FIG. 2, the image processing apparatus 10 may generate the HDR image 500 by processing the LDR image 400 using a convolutional neural network. The convolutional neural network may consist of three parts.

The first part may be an LDR decomposition part 210, the second part may be a feature extraction part 230, and the third part may be implemented as an HDR reconstruction part 250.

In tone mapping, a boundary preserving filter (e.g. a bilateral filter) is used for HDR input to decompose the image into a base layer and a detail layer to preserve only the detail layer while preserving the detail layer. This can be compressed. The processed base layer and detail layer can be integrated to obtain a final LDR output image.

In contrast, the purpose of the ITM algorithm may be to produce the output HDR image 500 by predicting the details lost with the extended base layer and matching the desired brightness. If the image is decomposed into two parts with different characteristics, then appropriate processing can be performed on the individual branches for more accurate prediction of the output image.

For the ITM image processing described above, the image processing apparatus 10 uses an neural network including an LDR decomposition part 210, a feature extraction part 230, and an HDR reconstruction part 250 as shown in FIG. 2. Can be processed.

The LDR decomposition part 210 may decompose the received image. The LDR decomposition part 210 may decompose the received image into a plurality of channels (or feature maps). For example, the number of output channels (eg, decomposed images) of the last layer of the LDR decomposition part 210 may be six.

The LDR decomposition part 210 may include a plurality of convolutional layers. For example, the LDR decomposition part 210 may be composed of three convolutional layers.

The LDR decomposition part 210 may decompose each image 400 received by performing a first convolution operation into a set of at least two different feature maps. The set of feature maps may include a plurality of feature maps (eg, exploded images). For example, LDR decomposition part 210 decomposes an image into the first three feature maps (eg, part of the exploded image) and the last three feature maps (eg, other parts of the exploded image). can do. That is, another part of the decomposed image may be other than a part of the decomposed image in the decomposed image.

The feature extraction part 230 may extract a feature from the decomposed image. The feature extraction part 230 may separately extract a feature for a part of the disassembled image and another part using a plurality of convolution branches.

Each convolution branch may include a plurality of convolution layers, and may perform a second convolution operation by focusing on the characteristics of each input (eg, a portion of the disassembled image and another portion). The feature extraction part 230 may generate a first feature map and a second feature map including the features of each of the decomposed images by performing a second convolution operation.

Finally, the HDR restoration part 250 may generate the HDR image 500 from the extracted features (eg, the first feature map and the second feature map) by performing a third convolution operation. The HDR reconstruction part 250 may learn how to concatenate the feature maps extracted from each convolution branch and integrate the feature maps extracted through the plurality of convolution layers to finally produce the HDR image 500. .

The image processing apparatus 10 may optimize the LDR decomposition part 210, the feature extraction part 230, and the HDR reconstruction part 250 together. In addition, the image processing apparatus 10 may effectively learn the feature extraction part 230 and the HDR reconstruction part 250 by decomposing the LDR image 400 using image filtering.

Hereinafter, an operation of training the neural network by the image processing apparatus 10 will be described with reference to FIGS. 3 to 4C.

FIG. 3 illustrates an operation of training the neural network shown in FIG. 2. 4A shows an example of the LDR image shown in FIG. 3, FIG. 4B shows an example of the base layer shown in FIG. 3, and FIG. 4C shows an example of the detail layer shown in FIG. 3.

3 to 4C, the image processing apparatus 10 may train a neural network. For example, the image processing apparatus 10 may train the convolutional neural network using a filter.

The image processing apparatus 10 may first pre-train the feature extraction part 230 and the HDR reconstruction part 250. To this end, the LDR decomposition part 210 may be replaced with a configuration in which the base layer 273 and the detail layer 275 for the LDR input are separated based on the guided-filter 271.

In the example of FIG. 3, induction filter 271 may include an edge-preserving filter that does not suffer from gradient reversal artifacts. The base layer 273 is extracted using the induction filter 271, and the detail layer 275 is obtained by performing element-wise division of the LDR image 400 by the base layer 273. Can be.

The division for each element generating the detail layer 275 may be represented by Equation 1 below.

는 원소별 나눗셈 연산을 의미할 수 있다. 도 4a는 LDR 이미지(400), 베이스 레이어(273) 및 디테일 레이어(275)의 예를 나타낸다.Here, I _LDR means the input LDR image 400 and I _base means the base layer 273,

May mean an element-wise division operation. 4A shows examples of LDR image 400, base layer 273, and detail layer 275.

After pre-learning the feature extraction part 230 and the HDR reconstruction part 250 using a pre-learning structure using induction filter 271 separation, the induction filter 271 may be plural (eg, three). The final convolutional neural network structure can be completed by replacing with a convolutional layer (eg, LDR decomposition part 210 of FIG. 2).

The three convolutional layers of the replaced LDR decomposition part 210 can be learned with the same data, but do not update the weights of the later layers by setting the learning rate of the layers to zero. Can be.

Through this, convolutional layers are trained to decompose the LDR image 400 into feature maps (eg, disassembled images), and later layers are trained to separate the induction filter 271, resulting in final loss. ) Can be lowered.

After the pre-learning, the image processing apparatus 10 has an end-to-end state in which all parts are integrated for joint optimization of the separately extracted feature extraction part 230, the HDR reconstruction part, and the LDR decomposition part 210. end-to-end) to train convolutional neural networks.

Referring to Table 1, it is possible to compare the PSNR performance according to the learning order of each part and whether the induction filter 271 is used.

Network part Learning order Image Decomposition - 2 ^nd - 2 ^nd Feature Extraction - 1 ^st 1 ^st 1 ^st HDR restore - all 1 ^st - 2 ^nd 3 ^rd PSNR of Y (dB) 46.71 46.28 47.11 47.27 PSNR of YUV (dB) 48.80 48.15 48.98 49.21

Referring to Table 1, the feature extraction part 230 and the reconstruction part 250 are first trained using the induction filter 271 using the induction filter 271, and the image decomposition part 210 is trained separately. We can see that PSNR performance is the best when we train the whole network.

5A-5C show examples of a network that decomposes an image.

5A-5C, decomposing the LDR image 400 may help the feature extraction part 230 to concentrate on each of the decomposed images. The effect of resolving the LDR image 400 can be observed by comparing the different structures shown in FIGS. 5A-5C.

The structure of FIG. 5A may represent a structure composed of convolutional layers that perform simple six residual learnings. Since the 8 bit / pixel LDR image 400 and the 10 bit / pixel ground truth HDR image are both normalized within the range [0, 1], the network can use the LDR image ( We can learn the difference between 400 and HDR image 500.

In this case, decomposition is not performed on the input LDR image 400, but residual learning is an additive separation of the output that focuses only on predicting the difference between the LDR image 400 and the HDR image 500. Can be interpreted.

The structure of FIG. 5B uses an inductive filter 271 for multiplicative input decomposition, and each of the two separate paths can predict the base layer 273 and detail layer 275 of the HDR image 500. have. The predicted base layer 273 and detail layer 275 can then be multiplied element-wise to obtain a final HDR image 500.

By providing the ground truss base layer 273 and detail layer 275 of the HDR image 500, the structure shown in FIG. 5B can fully concentrate on the decomposition for final prediction.

Although the schematic diagram of FIG. 5C uses an induction filter 271 for multiplication input decomposition, feature maps generated from individual passes can be concatenated for direct prediction of the HDR image 500.

In the structure of FIG. 5B, the network of FIG. 5C forces the last three convolutional layers while modeling the base layer 273 and detail layer 275 of the HDR image 500 for elemental multiplication integration. This allows you to learn optimal integration operations that lower the final loss. The structure of FIG. 5B may be the same as the structure used for prior learning in FIG. 3.

The results of comparing the performance of Figures 5a to 5c can be shown in Table 2.

rescue (a) * (a) (b) (c) * (c) Layer Number of filter channels (input, output) One 3,32 3,32 3,45 3,32 3,32 3,32 3,32 3,32 3,32 2 32,32 32,32 45,45 32,32 32,32 32,32 32,32 32,32 32,32 3 32,32 32,32 45,48 32,32 32,32 32,32 32,32 32,32 32,32 4 32,32 32,32 48,45 32,32 32,32 32,52 64,40 5 32,32 32,32 45,45 32,32 32,32 52,48 40,40 6 32,3 32,3 45,3 32,3 32,3 48,3 40,3 Full parameters 38,592 38,592 77,760 77,184 77,328 77,112 PSNR of Y 45.46 46.36 46.36 46.84 46.73 47.03 PSNR of YUV 47.28 48.25 48.39 48.65 48.87 48.82

Here, (a) * means no residual learning in the structure of FIG. 5A, and (c) * means multiplication by element instead of connecting feature maps after three convolution layers in the structure of FIG. 5C. It may mean.

Referring to Table 2, it can be seen that the structure using the decomposition of the image by inductive filtering exhibits high PSNR performance. When the integration scheme is learned using the convolutional layer as in (c) instead of implying the multiplicative nature of decomposition as in (b), the performance can be further improved. Residual learning can be important in simple structures such as (a).

For a fair comparison the total number of parameters in hidden layers can be adjusted so that the number of parameters in each structure is similar. PSNR performance can be compared on both Y and YUV channels.

Even with a similar number of parameters, the maximum PSNR difference measured only for Y is 0.67 dB and 0.48 dB for the YUV channel, depending on whether input decomposition is used and how the resolved input is processed.

The best structure is 5c, but (c) * can also show similar performance. Here the network can have the freedom to learn the integration of the two feature extraction steps. Comparing the structures 5a and 5b, it can be seen that the input decomposition using the induction filter 271 is very excellent.

In the case of a simple CNN structure in the 5a structure, it may be important to improve the prediction using residual learning. It may be important for the convolutional layer to focus on specific input decompositions with different characteristics and to learn by combining the information generated in different branches, to reconstruct the high quality HDR image 500.

6A to 7B illustrate examples of the original LDR image 400 and the HDR image 500 generated by the image processing apparatus 10.

6A and 7A show examples of unprocessed original images, and FIGS. 6B and 7B show examples of images processed by the image processing apparatus.

6A to 7B, 7268 frames of 3840 × 2160 UHD resolution of an LDR-HDR data pair including various scenes may be collected. The specification of the data can be shown in Table 3.

Data type Bit depth Transfer function Color container LDR 8 bits / pixel Gamma BT. 709 HDR 10 bits / pixel PQ-EOTF BT. 2020

HDR video was professionally shot, mastered, and both LDR and HDR data can be normalized to the [0, 1] range. For the synthesis of the training data, 20 sub-images of 40 × 40 per frame with a frame stride of 30 may be randomly cropped. This can generate an entire training set of size 40 × 40 × 3 × 4860.

For testing, 14 frames may be selected from six different scenes not included in the training set. All video can be converted to the YUV color space, and all three YUV channels can be used for learning. It is also possible to use only the Y channel, but it may be appropriate to use all three channels since the color container changes from BT.709 to BT.2020.

Referring to Table 4, we can see the PSNR benefits of using all three channels.

Train data Y only YUV PSNR of Y (dB) 44.36 46.36 PSNR of YUV (dB) 32.53 48.25

When measured for all three YUV channels, the large difference in PSNR is partly because the transfer function of the color container and the LDR and HDR images 500 may not match unless we learn about the U and V channels. The Y channel may gain from the complementary information of the U and V channels.

The weight decay of the convolution filter may be set to 5 × 10 ⁻⁴ , and the buyer may be set to zero. The mini batch size may be set to 32, the learning rate for the filter to 10 ⁻⁴ , and 10 ⁻⁵ for the bias.

All convolution filters are 3x3 in size, and the weights can be initialized with Xavier initialization, where the variance is weighted from a normal distribution expressed as the number of input and output neurons. The loss function of the network may be given by Equation 2.

Here, θ may mean a set of model parameters, n may mean the number of training samples, and I _LDR may mean an input LDR image 400. F is the prediction of the network F; can mean a non-linear mapping function of the convolutional neural network that provides a (I _LDR θ), I _HDR can mean the ground Truth HDR image.

The activation function is a rectified linear unit (ReLU), which can be expressed as Equation 3.

In addition, all network models can be implemented in the MatConvNet package.

While high-end HDR TVs are readily available on the market, HDR video content is lacking. Accordingly, a method of up-converting LDR legacy video for HDR TV to HDR video may be required.

While traditional ITM methods share a similar goal of upconverting, their ultimate goal is not to render inverse tonemapped HDR images 500 to consumer HDR TV displays, but to transfer LDR scenes to the HDR domain for professional graphics rendering. Was to improve. The HDR image 500 generated by the previous ITM method may have unnatural colors or no noise amplification and local contrast in dark areas when viewed on a consumer HDR TV display.

The image processing apparatus 10 may learn NateWalk to restore detail information and local contrast, thereby providing an ITM scheme using a convolutional neural network for HDR consumer display.

In order to accurately predict the HDR image 500, the image processing apparatus 10 may learn different portions of the network (LDR decomposition, feature extraction and HDR reconstruction separately before end-to-end learning).

In particular, for LDR decomposition for the pre-learning step, the image processing apparatus 10 may employ an induction filter 271 so that subsequent layers can focus on the images that have been separately resolved in separate paths.

The HDR reconstruction part 250 of the image processing apparatus 10 may learn how to integrate a feature map extracted from two paths. The resulting HDR image 500 can produce an HDR image 500 that is free of artifacts and reconstructs local contrast and details to approximate a ground truss HDR image. The image processing apparatus 10 may enable watching legacy LDR video directly as HDR video on a normal HDR TV display.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

Although the embodiments have been described with reference to the accompanying drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the following claims.

Claims (16)

Receiving an image;
Decomposing the image by performing a first convolution operation;
Performing a second convolution operation to extract a first feature map from a portion of the decomposed image, and extract a second feature map from another portion of the decomposed image; And
Generating a high dynamic range (HDR) image by performing a third convolution operation on the first feature map and the second feature map
Image processing method comprising a.
The method of claim 1,
Generating the HDR image,
Combining the first feature map and the second feature map; And
Generating an HDR image by performing the third convolution operation on the combined first feature map and the second feature map
Image processing method comprising a.
The method of claim 1,
Learning filter parameters of the first, second and third convolution operations
Image processing method further comprising.
The method of claim 3,
The learning step,
Learning filter parameters of the second and third convolution operations based on the image;
After learning the filter parameters of the second and third convolution operations, learning the filter parameters of the first convolution operations; And
After learning the filter parameters of the first convolution operation, learning the filter parameters of the first to third convolution operations
Image processing method comprising a.
The method of claim 4, wherein
Learning the filter parameters of the second and third convolution operations,
Generating a portion of the decomposed image by filtering the image;
Generating another part of the exploded image based on the image and the part; And
Learning filter parameters of the second and third convolution operations based on the portion and the other portion
Image processing method comprising a.
The method of claim 5,
Generating the portion,
Generating the portion by edge preserving filtering the image.
Image processing method comprising a.
The method of claim 5,
Generating the other part,
Generating the other part by performing element-wise division on the image and the part
Image processing method comprising a.
The method of claim 5,
Learning the filter parameters of the first to third convolution operations,
Training end-to-end a network in which the parameters of the learned first convolution operation and the filter parameters of the learned second and third convolution operations are combined;
Image processing method comprising a.
A receiver for receiving an image; And
Perform a first convolution operation to decompose the image, perform a second convolution operation to extract a first feature map from a portion of the decomposed image, and extract a second feature map from another portion of the decomposed image, A processor that generates an HDR image by performing a third convolution operation on the first feature map and the second feature map
Image processing apparatus comprising a.
The method of claim 9,
The processor,
Combining the first feature map and the second feature map, and performing the third convolution operation on the combined first feature map and the second feature map to generate an HDR image
Image processing unit.
The method of claim 9,
The processor,
Learning filter parameters of the first, second and third convolution operations
Image processing unit.
The method of claim 11,
The processor,
Training filter parameters of the second and third convolution operations based on the image, learning filter parameters of the second and third convolution operations, and then learning filter parameters of the first convolution operation, After learning the filter parameters of the first convolution operation, learning the filter parameters of the first to third convolution operations
Image processing unit.
The method of claim 12,
The processor,
Generate a portion of the disassembled image by filtering the image, generate another portion of the disassembled image based on the image and the portion, and generate the second and third convolution based on the portion and the other portion Training filter parameters
Image processing unit.
The method of claim 13,
The processor,
Generating the portion by edge preserving filtering the image.
Image processing unit.
The method of claim 13,
The processor,
Generating another part by performing element-wise division on the image and the part
Image processing unit.
The method of claim 13,
The processor,
Training end-to-end networks in which the parameters of the learned first convolution operation and the filter parameters of the learned second and third convolution operations are combined.
Image processing unit.

Images