JP2016170637A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device that suppresses deterioration in image quality due to gradation conversion.SOLUTION: An image processing unit 104 of an image processing device includes: calculation means 204 that calculates the similarity between an input image and a processed image which is obtained by subjecting the input image to predetermined processing, on the basis of the respective frequency characteristics and contrast characteristics of the input image and processed image; determination means 203 that determines a parameter for gradation conversion of the input image on the basis of the calculated similarity; and conversion means 202 that performs gradation conversion of the input image on the basis of the determined parameter.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus and an image processing method.

  In recent years, HDR (High Dynamic Range) technology has been developed that creates a plurality of images with different exposure times and adds the plurality of images to expand the dynamic range. By using this technique, it is possible to record a wider range of luminance data than conventional imaging data, and it is possible to record image data that is not blacked out or overexposed.

However, even when imaging (photographing) with a wide dynamic range is performed using the HDR technology, the luminance range of the display device may not be sufficient. In such a case, it is necessary to perform gradation conversion (gradation compression) processing for converting the gradation of the captured image in accordance with the dynamic range of the display device.
Japanese Patent Application Laid-Open No. 2004-228561 discloses tone conversion in consideration of a luminance histogram and a high frequency component histogram of an input image.

JP2010-220032

In Patent Document 1, tone conversion is performed based on a high-frequency component histogram obtained for pixels having a certain frequency or higher, but the relationship between frequency and contrast is not considered. Therefore, when the gradation conversion of the HDR image is performed by the method of Patent Document 1, depending on the image, image quality deterioration such as blackout, gradation loss, and overexposure may occur, which may be perceived by the user.
In view of the problems as described above, it is an object of the present invention to provide an image processing apparatus and an image processing method thereof that can suppress image quality deterioration due to gradation conversion.

An image processing apparatus according to an aspect of the present invention uses a similarity between an input image and a processed image obtained by performing predetermined processing on the input image, and the frequency characteristics of the input image and the processed image. And a calculation means for calculating based on the contrast characteristics;
Determining means for determining a parameter for gradation conversion of the input image based on the calculated similarity;
Conversion means for performing gradation conversion of the input image based on the determined parameter;
Is provided.

  According to the present invention, it is possible to suppress deterioration in image quality due to gradation conversion.

1 is a block diagram showing a configuration of a digital camera including an image processing apparatus according to Embodiment 1 of the present invention. FIG. 2 is a block diagram illustrating a configuration of an image processing unit that is the image processing apparatus according to the first embodiment. It is a flowchart which shows operation | movement of the image process part of FIG. It is a figure which shows an example of the gradation conversion curve used in Embodiment 1. FIG. FIG. 3 is a block diagram illustrating a configuration of a similarity calculation unit according to the first embodiment. 3 is a block diagram illustrating a configuration of a visual characteristic determination unit of Embodiment 1. FIG. It is a schematic diagram of a human frequency sensitivity characteristic. 3 is a block diagram illustrating a configuration of a similarity calculation unit according to Embodiment 1. FIG. FIG. 3 is a schematic diagram illustrating frequency division bands used in the first embodiment. It is a figure which shows the filter characteristic of the band division filter used for the frequency division of FIG. FIG. 3 is a schematic diagram illustrating a method for determining a contrast threshold in the first embodiment. It is the schematic diagram which represented the frequency band division by the band division filter in Embodiment 2 by the half value frequency.

Embodiments of an image processing apparatus and an image processing method of the present invention will be described below with reference to the drawings. The following embodiments do not limit the present invention, and all the combinations of items, elements, and steps described in the embodiments are not necessarily essential to the present invention. The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed according to the configuration of the apparatus to which the present invention is applied and various conditions.
As described above, when gradation conversion (gradation compression) of an HDR image is performed by a conventional method, there is a possibility that image quality deterioration such as blackout, gradation loss, and overexposure occurs depending on the image. This is because the visual difference before and after the gradation compression is not taken into consideration when the gradation compression is performed on a display device that can display only a narrow luminance range of imaging data obtained by imaging a wide range of luminance.

The human visual characteristic has a band-pass property in which the sensitivity decreases as the frequency increases with a peak at 4 to 5 cycles / viewing angle of 1 degree. For this reason, low-frequency edges and textures can be perceived even with relatively low contrast, whereas high-frequency edges and textures cannot be perceived unless the contrast difference is large. In order to perform gradation conversion with less edge and texture gradation collapse, it is necessary to consider both frequency and contrast.
Furthermore, human visual characteristics change according to the brightness of the subject. For example, when the subject is bright, the visual sensitivity is high, and when the subject is dark, the visual sensitivity is low. For this reason, it is necessary to perform gradation conversion processing in which the gradation loss of the edges and texture is small in consideration of the difference in luminance between the imaging environment and the display environment.

  In the following description, an example in which an image captured by a digital camera is displayed after gradation conversion (gradation compression) will be described. First, based on the imaging conditions and display conditions of the digital camera, the visual contrast characteristics in the environment at the time of imaging and the visual contrast characteristics in the environment at the time of display are obtained. Further, frequency analysis in consideration of luminance is performed on each of the high gradation image before gradation compression and the low gradation image after gradation compression. From the result of the frequency analysis and the visual contrast characteristics, the visual similarity between the image before gradation compression and the image after gradation compression is calculated, and gradation conversion processing that maximizes the similarity is performed. . If tone conversion that maximizes the degree of similarity is performed, image quality deterioration due to tone conversion can be suppressed.

Since the high-frequency component in the dark part of the image cannot be seen in the first place, in this embodiment, it is considered that even if the gradation of the dark part of the image is crushed by gradation compression, the similarity before and after gradation compression is not affected (the influence is small). That is, in this embodiment, the visual difference between the image before gradation compression and the image after gradation compression is calculated as being small in the low gradation region of the image. In the conventional gradation conversion, the high frequency components in the dark part of the image are prevented from being crushed by gradation compression.
In the present embodiment, gradation conversion is performed according to the brightness of the imaging environment and the brightness of the display environment. Therefore, different gradation conversion may be performed even for the same input image.

(Embodiment 1)
<Configuration of image processing apparatus>
FIG. 1 is a block diagram showing a configuration of a digital camera including the image processing apparatus according to the first embodiment of the present invention.
The digital camera includes an imaging unit 101, an A / D conversion unit 102, a signal processing unit 103, an image processing unit 104, an encoder unit 105, a media interface 106, a CPU 108, a ROM 109, a RAM 110, an imaging system control unit 111, an operation unit 112, and a D. / A conversion part 113 and the display part 114 are provided. The above-described components of the digital camera are connected to each other by a bus 115 directly or indirectly. An image processing unit 104 is an image processing apparatus according to this embodiment.

The imaging unit 101 includes a zoom lens, a focus lens, a shake correction lens, an aperture, a shutter, an optical low-pass filter, an iR (infrared) cut filter, a color filter, and a sensor such as a CMOS element or a CCD. The imaging unit 101 detects the amount of light of a subject (not shown). The imaging unit 101 converts the optical image of the subject into an electrical signal by detecting the amount of light of the subject. This electrical signal is an analog signal. The imaging unit 101 outputs this analog signal to the A / D conversion unit 102.
The imaging element (CMOS element or the like) of the imaging unit 101 is, for example, a color imaging element in which primary color filters of three primary colors of red (R), green (G), and blue (B) are arranged in a predetermined pattern for each pixel. is there. The optical image incident on the light receiving surface of the image sensor via the lens of the imaging unit 101 is converted into RGB signal charges corresponding to the amount of incident light, and output from the imaging unit 101 as a voltage signal (analog image signal) for each pixel. Is done.

The digital camera of this embodiment can perform high dynamic range imaging using HDR processing. For example, the imaging unit 101 detects the amount of light of the subject in a 16-bit mode.
The A / D converter 102 converts the analog signal corresponding to the amount of light of the subject into a digital value. The A / D conversion unit 102 outputs the digital value to the signal processing unit 103.
The signal processing unit 103 performs demosaicing processing, white balance processing, gamma processing, noise reduction processing, and the like on the digital value to generate a digital image. The noise reduction process is a process using a composite method, for example. The signal processing unit 103 outputs the digital image to the image processing unit 104. In the present embodiment, the digital image output from the signal processing unit 103 is a 16-bit image (high gradation image). In the present embodiment, the digital image is an RGB image.

  The image processing unit 104 performs gradation conversion processing on the digital image. The image processing unit 104 outputs the gradation-converted image to the encoder unit 105 and the D / A conversion unit 113. The image processing unit 104 has an input terminal 201 that receives a digital image from the signal processing unit 103, and an output terminal 205 that outputs an image to the encoder 105 and the D / A conversion unit 113. Details of the configuration and operation of the image processing unit 104 will be described later with reference to FIG. The image output from the image processing unit 104 is, for example, an 8-bit image (low gradation image).

The encoder unit 105 performs processing for converting the digital image after gradation conversion into a video compression format such as JPEG. The encoder unit 105 outputs the digital image to the media interface 106. The digital image after gradation conversion is also an RGB image.
The media interface 106 is an interface for connecting the digital camera to an external medium 107. The external medium 107 is, for example, a hard disk, a memory card, a CF (Compact Flash) card, an SD card, or a USB memory.

The CPU 108 controls processing / operation of each component of the digital camera via the system bus 115. The control by the CPU 108 is performed, for example, by the CPU 108 appropriately reading instructions and data stored in the ROM 109 and RAM 110 and executing the instructions. The CPU 108 also performs processing based on information and instructions input by the user of the digital camera.
The ROM 109 and the RAM 110 store programs and data necessary for processing executed by the CPU 108 and provide the CPU 108 with a work area necessary for the processing. For example, the CPU 08 reads a computer program recorded in the ROM 109 into the RAM 110, and the CPU 108 operates according to this program.

The imaging system control unit 111 controls the imaging unit 101 based on an instruction from the CPU 108. For example, the imaging system control unit 111 performs control such as adjusting the focus of the lens of the imaging unit 101, opening the shutter, and adjusting the aperture.
The operation unit 112 includes an imaging button, an operation button, a mode dial, and the like. A user of the digital camera can input (set) an instruction to the digital camera via the button or dial. The instruction input from the user is, for example, a setting for imaging (shooting), and includes an ISO sensitivity setting, a shutter speed setting, an F value (aperture value) setting, and the like. Information and instructions regarding imaging settings input from the operation unit 112 are sent to the CPU 108. As described above, the CPU 108 performs processing based on information and instructions input from the operation unit 112 and determines the imaging conditions of the digital camera. The imaging conditions are stored in the RAM 110.

The D / A conversion unit 113 performs analog conversion on the low gradation digital image output from the image processing unit 104. The D / A converter 113 outputs the analog-converted image to the front surface 114.
The display unit 114 is, for example, a liquid crystal display. The display unit 114 displays the low gradation image received from the D / A conversion unit 113 under the control of the CPU 108. The display unit 114 can display, for example, 256 gradations.
The display unit 11 may display function icons that function as function buttons (imaging buttons or operation buttons). The user can input an instruction to the digital camera by selecting the function icon (touching the function icon).

<Configuration of Image Processing Unit 104>
Hereinafter, the configuration and operation of the image processing unit 104 will be described in detail. FIG. 2 shows the configuration of the image processing unit 104.
The image processing unit 104 includes an input terminal 201, a gradation conversion process execution unit 202, a gradation conversion process determination unit 203, a similarity calculation unit 204, and an output terminal 205. The similarity calculation unit 204 is connected to the bus 115. The similarity calculation unit 204 includes a first input terminal 501, a second input terminal 502, and an output terminal 505.

An image signal (high gradation image Iin) from the signal processing unit 103 (FIG. 1) is input to the input terminal 201 of the image processing unit 104. The image signal is input from the input terminal 201 to the gradation conversion processing execution unit 202 and the similarity calculation unit 204. That is, the high gradation image Iin before gradation conversion is input to the gradation conversion process execution 202 and the similarity calculation unit 204.
The gradation conversion processing execution unit 202 performs gradation conversion processing on the high gradation image Iin. The gradation conversion processing execution unit 202 outputs the image Iconv after the gradation conversion to the gradation conversion processing determination unit 203 and the similarity calculation unit 204. The gradation conversion processing execution unit 202 performs gradation conversion on the image Iin using a plurality of gradation conversion parameters, and creates a plurality of gradation-converted images Iconv.

  The similarity calculation unit 204 receives an image Iin before gradation conversion at the first input terminal 501 and receives a plurality of images Iconv after gradation conversion at the second input terminal 502. The similarity calculation unit 204 displays the appearance of edges and textures in the setting environment of the imaging unit 101 with respect to the high gradation image Iin before gradation conversion (the amount of perceivable edges and textures), and the low gradation image Iconv after gradation conversion. The difference (difference) from the appearance of the edge and texture in the setting environment of the display unit 114 is calculated. This calculation is performed for each of the plurality of low gradation images (images after gradation conversion) Iconv. If this “difference in appearance” is small, the similarity is high. The similarity calculation unit 204 outputs information regarding the image Iconv having the highest similarity among the calculation results (similarity) from the output terminal 505 to the gradation conversion processing determination unit 203. The “perceivable edge and texture amount” is hereinafter referred to as “perceived edge / texture amount”.

The gradation conversion processing determination unit 203 outputs an image Iconv having the highest similarity calculated by the similarity calculation unit 204 among the plurality of images Iconv received from the gradation conversion processing execution unit 202 as an output image (for display). Determine (select) as an image. That is, the gradation conversion process determination unit 203 determines an optimal gradation conversion parameter for the image Iin (input image) based on the calculated similarity.
The gradation conversion method performed by the gradation conversion processing execution unit 202 in the present embodiment is selected from, for example, a technique using a tone curve, a histogram smoothing method, and a histogram specifying method. The present invention is not limited to the gradation conversion method described above. The present invention can be applied to any gradation conversion method. That is, in the present invention, for an arbitrary gradation compression process, a difference in perceived edge / texture amount before and after the process (the above-mentioned “difference in appearance”) is calculated, and an optimum gradation process is determined (selected). Then, the gradation processing is executed.

  Next, specific processing of the image processing unit 104 will be described with reference to the flowchart of FIG. The gradation conversion of this embodiment uses the simple linear tone curve (polygonal tone curve) shown in FIG. The vertical axis in FIG. 4 indicates the output gradation, and the horizontal axis indicates the input gradation. In FIG. 4, the output gradation is determined by performing linear conversion (gradation conversion) using two offset values (offs_L and offs_H) on the input gradation value. In the following example, a plurality of offset values offs_L on the low gradation side and offset values offs_H on the high gradation side are prepared, and optimum offset values offs_L and offs_H are selected from among them. The optimum offset values offs_L and offs_H are offset values that minimize the difference between the perceived edge / texture amount before and after processing. The offset values offs_L and offs_H are parameters for gradation conversion processing. A plurality of prepared offset values offs_L and offs_H may be referred to as “parameter candidates” in the following description.

In step S301 in FIG. 3, the gradation conversion processing execution unit 202 prepares a plurality of parameter candidates (offset values offs_L and offs_H) for gradation conversion processing, and sets one parameter candidate from them. Specifically, the gradation conversion processing execution unit 202 sets one set of offset values offs_L and offs_H from the plurality of offset values offs_L and offs_H. After step S301 ends, the process proceeds to step S302.
In step S302, the gradation conversion processing execution unit 202 performs gradation conversion processing on the input image Iin. Specifically, the gradation conversion processing execution unit 202 performs gradation conversion on the image Iin using the tone curve (FIG. 4) defined by the parameter candidate (offset value) determined in step S301, A gradation-converted image Iconv is generated. After step S302 ends, the process proceeds to step S303.

In step S303, the similarity calculation unit 204 calculates the similarity between the image Iconv after gradation conversion and the input image Iin. The similarity is calculated by calculating a difference between the perceived edge / texture amount of the input image Iin in the imaging environment and the perceived edge / texture amount of the image Iconv in the display environment. The smaller the difference, the higher the similarity. Details of the calculation will be described later.
In step S304, it is determined whether or not the above-described processing has been performed on all parameter candidates (all of the prepared offset values offs_L and offs_H). If the determination result is true (Yes), the process proceeds to step S305, and if not (No), the process returns to step S301.

  If the processing of steps S301 to S303 is performed on all parameter candidates, in step S305, the similarity calculation unit 204 displays information on the parameter having the highest similarity among all parameter candidates (offs_L and offs_H). This is provided to the tone compression processing determination unit 203. Then, the gradation compression processing determination unit 203 outputs the image Iconv subjected to gradation conversion with the parameter having the highest similarity to the D / A conversion unit 113 and the encoder 105.

<Configuration of Similarity Calculation Unit 204>
Next, the configuration and operation of the similarity calculation unit 204 will be described. The similarity calculation unit 204 calculates the difference between the perceived edge / texture amount of the image Iin before gradation conversion and the perceived edge / texture amount of the image Iconv after gradation conversion.
FIG. 5 shows the configuration of the similarity calculation unit 204. The similarity calculation unit 204 includes two input terminals 501 and 502, two visual characteristic determination units 503a and 503b connected to the two input terminals 501 and 502, and one visual similarity calculation unit 504, And one output terminal 505. The image Iin before gradation conversion is input from the first terminal 501 of the similarity calculation unit 204. The image Iin input to the first terminal 501 is input to the input terminal 601 of the visual characteristic determination unit 503a and the first input terminal 801 of the visual similarity calculation unit 504. The image Iconv after gradation conversion is input from the second input terminal 502. The image Iconv input to the second terminal 502 is input to the input terminal 601 of the visual characteristic determination unit 503b and the second input terminal 802 of the visual similarity calculation unit 504. The configuration of the two visual characteristic determination units 503a and 503b is the same.

The visual characteristic determination unit 503a calculates the visual contrast sensitivity characteristic of the image Iin before gradation conversion, and the visual characteristic determination unit 503b calculates the visual contrast sensitivity characteristic of the image Iconv after gradation conversion. The contrast sensitivity characteristic calculated by the visual characteristic determination unit 503a is input to the third input terminal 803 of the visual similarity calculation unit 504. The contrast sensitivity characteristic calculated by the visual characteristic determination unit 503b is input to the fourth input terminal 804 of the visual similarity calculation unit 504. The contrast sensitivity characteristic is sometimes referred to as contrast characteristic in the following description.
The visual similarity calculation unit 504 uses the two input contrast sensitivity characteristics (visual characteristics) to determine the visual difference (perceptual edge) between the image Iin before gradation conversion and the image Iconv after gradation conversion. Calculate the difference in texture amount. The calculation result is output from the output terminal 505 to the gradation conversion processing determination unit 203 (FIG. 2).

<Operation of the visual characteristic determination unit 503a>
Next, the configuration and operation of the visual characteristic determination unit 503a will be described with reference to FIGS. FIG. 6 is a functional block diagram of the visual characteristic determination unit 503a. The visual characteristic determination unit 503a includes an input terminal 601, a luminance information acquisition unit 602, a visual characteristic calculation unit 603, and an output terminal 604. The luminance information acquisition unit 602 is connected to the bus 115. The configuration and operation of the visual characteristic determination unit 503b are the same as the configuration and operation of the visual characteristic determination unit 503a.

FIG. 7 is a diagram showing the relationship between contrast sensitivity and spatial frequency. The vertical axis in FIG. 7 is the contrast sensitivity, and the horizontal axis is the spatial frequency. In this specification, the visual characteristic means a contrast sensitivity characteristic for each frequency. The visual characteristic has a bandpass characteristic as shown in FIG. That is, the visual characteristic has a bandpass characteristic that the peak sensitivity increases and the peak frequency shifts to a high frequency as the illumination environment is brighter (the luminance is higher). For example, in Figure 7, when the brightness curve and brightness 0.1 cd / ^{m 2} Comparing the curve of 1 cd / ^{m 2,} towards the curve of 1 cd / ^{m 2} is a high brightness, the peak sensitivity is high and the peak The frequency is high.

  In the following description, visual characteristics are denoted as VTFs (u). VTF is an abbreviation for Visual Transfer Function (visual transfer function, visual spatial frequency characteristic). u is the frequency. In the present embodiment, the visual characteristic VTFs (u) in the imaging (photographing) environment is determined by the visual characteristic determination unit 503a, and the visual characteristic VTFs (u) in the display environment is determined by the visual characteristic determination unit 503b. Since the visual characteristic determined by the visual characteristic determination unit 503a is different from the visual characteristic determined by 503b, there is a difference between the perceived edge / texture amount in the imaging environment and the perceived edge / texture amount in the display environment. come. In the present embodiment, this difference (difference) is quantified. The visual characteristics VTFs (u) can also be referred to as contrast sensitivity characteristics. The visual characteristic VTFs (u) of the present embodiment is a one-dimensional visual characteristic.

  Images Iin and Iconv are input from the input terminals 601 of the visual characteristic determination units 503a and 503b. The input images Iin and Iconv are sent to the luminance information acquisition unit 602. The luminance information acquisition 602 acquires imaging (imaging) conditions and display conditions via the bus 115. The luminance information acquisition 602 acquires the luminance value L of the captured image and the display image from the imaging condition and the display condition. As the luminance value L, for example, an average luminance value or a maximum luminance value in the image is used. The luminance information acquisition unit 602 outputs the acquired luminance value L to the visual characteristic calculation unit 603.

<Operation of Visual Characteristic Calculation Unit 603>
The visual characteristic calculation unit 603 calculates the visual characteristic based on the luminance value L input from the luminance information acquisition unit 602. Hereinafter, the calculation process of the visual characteristic calculation unit 603 will be described. In this embodiment, a spatial VTF derived by the Barten model (see Barten, Peter GJ. “Formula for the contrast sensitivity of the human eye.” International Society for Optics and Photonics, 2003) is used. When this space VTF is used, the contrast sensitivity characteristic VTFs (u) with respect to the frequency u is given by the following equation (1).

Here, X _o is a viewing angle [degree]. L is the luminance, which is a value acquired by the luminance information acquisition unit 602.
In the above description, the space VTF derived by the Barten model is used, but the present invention is not limited to this. For example, the contrast sensitivity characteristic acquired by changing the luminance of the subject may be measured in advance, recorded as data, and referred to.
The visual characteristic calculated by the visual characteristic calculation unit 603 is input to the visual similarity calculation unit 504 (FIG. 5).

<Operation of Visual Similarity Calculation Unit 504>
As shown in FIG. 5, the visual characteristic determination unit 503a inputs the image Iin before gradation conversion to the first input terminal 801 of the visual similarity calculation unit 504, and visually determines the visual characteristic of the image Iin. This is input to the third input terminal 803 of the similarity calculation unit 504. The visual characteristic determination unit 503b inputs the image Iconv after the gradation conversion to the second input terminal 802 of the visual similarity calculation unit 504, and the visual characteristic of the image Iconv is input to the visual similarity calculation unit 504. The fourth input terminal 804 is input. The configuration and operation of the visual similarity calculation unit 504 will be described below with reference to FIG.

  As shown in FIG. 8, the visual similarity calculation unit 504 includes four input terminals 801, 802, 803, and 804, an imaging environment luminance conversion unit 805, a display environment luminance conversion unit 806, and two frequency analysis units. 807a and 807b. The visual similarity calculation unit 504 further includes two visual characteristic application units 808a and 808b, a difference calculation unit 809, and an output terminal 810. Based on the visual characteristics input from the visual characteristic determination units 503a and 503b, the visual similarity calculation unit 504 calculates the perceived edge / texture amount between the image before gradation conversion and the image after gradation conversion. Calculate the difference. The two frequency analysis units 807a and 807b have the same configuration. The two visual characteristic application units 808a and 808b have the same configuration.

  The image Iin before gradation conversion is input from the first input terminal 801 of the visual similarity calculation unit 504 to the imaging environment luminance conversion unit 805. The visual characteristic of the image Iin is input from the third input terminal 803 to the visual characteristic application unit 808a. The image Iconv after the gradation conversion is input from the second input terminal 802 to the display environment luminance conversion unit 806. The visual characteristic of Iconv is input from the fourth input terminal 804 to the visual characteristic application unit 808b.

  The imaging environment luminance conversion unit 805 converts the image Iin into a luminance image in the imaging environment based on the imaging conditions. Then, the imaging environment luminance conversion unit 805 outputs the luminance image to the frequency analysis unit 807a. The frequency analysis unit 807a divides the luminance image into images for each frequency band. In the following description, the divided image is referred to as “divided image”. The frequency analysis unit 807a outputs the divided image to the visual characteristic application unit 808a. The visual characteristic applying unit 808a performs threshold processing on the divided image using the visual characteristic of the image Iin input from the third input terminal 803. That is, the visual characteristic application unit 808a applies the visual characteristic for the image Iin to the divided image (multiplies the contrast characteristic). The visual characteristic application unit 808a inputs an image to which the visual characteristic is applied to the difference calculation unit 809. The divided image output from the frequency analysis unit 807a is an image including the frequency characteristics of the input image (image Iin).

The display environment luminance conversion unit 806 converts the image Iconv into a luminance image in the display environment based on the display conditions. Then, the display environment luminance conversion unit 806 outputs the luminance image to the frequency analysis unit 807b. The frequency analysis unit 807b divides the luminance image into divided images. The frequency analysis unit 807b outputs the divided image to the visual characteristic application unit 808b. The visual characteristic application unit 808b performs threshold processing on the divided image using the visual characteristic of the image Iconv input from the fourth input terminal 804. That is, the visual characteristic application unit 808b applies the visual characteristic for the image Iconv to the divided image. The visual characteristic application unit 808b inputs an image to which the visual characteristic is applied to the difference calculation unit 809.
The difference calculation unit 809 calculates the difference between the image input from the visual characteristic application unit 808a and the image input from the visual characteristic application unit 808b. That is, the difference calculation unit 809 calculates a value (visual edge / texture amount difference) representing a difference in visual appearance between the image before gradation conversion and the image after gradation conversion.

<Operation of Imaging Environment Brightness Conversion Unit 805>
Processing of the imaging environment luminance conversion unit 805 will be described. The imaging environment luminance conversion unit 805 converts the image Iin, which is an RGB image, into a luminance image (luminance conversion processing). The luminance image obtained here reproduces how a person (for example, a user of a digital camera) was looking at the subject when the subject was captured. Specifically, the luminance conversion process is performed according to the following procedure.

First, degamma processing is performed on the input image Iin to obtain a luminance linear image Iin ′. The degamma process is performed using, for example, a lookup table.
Next, the luminance linear image Iin ′ is converted into a luminance image. Specifically, this conversion is performed by the processing shown by the following formula (2).

Here, α represents a gain. The gain α is determined from the relationship between ISO sensitivity, aperture, and shutter speed. For example, the gain α may be calculated in advance for all combinations of ISO sensitivity, aperture, and shutter speed. I _inR ′, I _inG ′, and I _inB ′ are the R plane, G plane, and B plane of the luminance linear image I _in ′, respectively. Further, a, b, and c are constants. For example, when the captured (captured) image is an sRGB space, a = 0.2126, b = 0.7152, and c = 0.0722.

<Operation of Display Environment Brightness Conversion Unit 806>
The processing of the display environment luminance conversion unit 806 will be described. The display environment luminance conversion unit 806 converts the image Iconv, which is an RGB image, into a luminance image (luminance conversion processing). The luminance image obtained here reproduces how a person (for example, a user of a digital camera) views the subject when the image is displayed. Specifically, the luminance conversion process is performed according to the following procedure.
First, the input image Iconv is subjected to de-gamma processing of the display of the display unit 114 to obtain a luminance linear image Iconv ′. For example, when the display gamma of the display unit 114 is 2.2, the luminance linear image Iconv ′ is obtained by the following equation (3).

  Next, the obtained luminance linear image Iconv 'is multiplied by a gain to be converted into a luminance image. Specifically, the processing is performed by the following equation (4).

Here, α represents a gain. The gain α is determined by the maximum display brightness of the display unit 114. I _convR ′, I _convG ′, and I _convB ′ are the R plane, B plane, and G plane of the luminance linear image Iconv ′, respectively. Further, a, b, and c are constants. For example, when the display color space is sRGB, a = 0.2126, b = 0.7152, and c = 0.0722.

<Operation of Frequency Analysis Unit 807a>
Next, the operation of the frequency analysis unit 807a will be described. In the present embodiment, after the Fourier transform of the luminance image, the frequency band division is performed by multiplying by the band division filter function and performing the inverse Fourier transform. FIG. 9 shows _six images I _{1 to} I ₆ obtained by frequency band division according to this embodiment. Hereinafter, the Fourier transform of the luminance image, the multiplication of the band division filter function, and the inverse Fourier transform will be described.

First, the luminance image is two-dimensionally Fourier transformed to obtain an image Ifft (u, v). Two dimensions are defined by x and y coordinates. u represents a frequency component in the x direction, and v represents a frequency component in the y direction.
Next, band division is performed on the image Ifft (u, v) using the band division filter function cortex _k (u, v). Here, k is an index of the band in the radial direction. Spatial frequency information I ₁ _fft... I ₆ _fft after band division is expressed by Expression (5), respectively.

Here, k = 1... Further, the band division filter function cortex _k (u, v) is defined as the following equations (6) and (7).

cortex_pol_k,l(ρ,θ) はcortex_k,l(u,v)を極座標で表現したものである。dom_k (ρ) の定義は式（８）の通りである。

cortex_pol _{k, l} (ρ, θ) is a representation of cortex _{k, l} (u, v) in polar coordinates. The definition of dom _k (ρ) is as shown in Equation (8).

ここでmesa (ρ)および base (ρ)は式（９）および（１０）で与えられる。

Here, mesa (ρ) and base (ρ) are given by equations (9) and (10).

であり、ρ_h、ｔｗおよびσは式（１１）に示されるとおりである。

Ρ _h, tw and σ are as shown in the equation (11).

FIG. 10 shows a schematic diagram of dom _k (ρ). The horizontal axis is normalized frequency (cycle / pixel), and the vertical axis is response.
The resulting I _k _fft (u, v) by inverse Fourier transform, to obtain six images I ₁ ~I _6.
In this embodiment, Fourier transform is used to perform frequency band division on the frequency domain, but the present invention is not limited to this. For example, frequency band division may be performed by performing direct filtering using a digital filter having the above-described characteristics. Further, frequency band division may be performed using wavelet transform (wavelet function). When the wavelet transform is used, there may be a case where time domain information lost when the frequency characteristic is obtained by the Fourier transform can be left.
Since the configuration of the frequency analysis unit 807b is the same as that of the frequency analysis unit 807a, the description of the frequency analysis unit 807a is omitted.

<Visual characteristics application unit 808a>
Hereinafter, the processing of the visual characteristic application unit 808a will be described in detail.
First, a contrast threshold is determined for each of the _six images I _{1 to} I ₆ . Specifically, the contrast threshold value is determined from the visual characteristic VTF determined by the visual characteristic determination unit 503a and the images I _{1 to} I ₆ . Figure 11 is a conceptual diagram of the calculation of the contrast threshold th ₂ contrast threshold th ₁ and the image I ₆ of the image I _1. Specifically, six contrast thresholds th _k (k = 1... 6) of the six images I _{1 to} I ₆ are obtained by the following equation (12).

Next, using the contrast threshold th _k (k = 1... 6), threshold processing is performed on the frequency bands of the images I _{1 to} I ₆ to calculate I _thk (k = 1... 6). Specifically, it is calculated by the following equation (13).

Further, the calculated I _thk (k = 1... 6) is integrated to calculate an image Ith (x, y). Specifically, the image Ith (x, y) is calculated by the following equation (14). This image Ith (x, y) indicates the perceived edge / texture amount of the image Iin.

The visual characteristic application unit 808b performs the same processing as the visual characteristic application unit 808a on the image Iconv, and calculates the perceived edge / texture amount of the image Iconv.
In the following description, the output of the visual characteristic application unit 808a for the input image Iin is expressed as Iin_th, and the output of the visual characteristic application unit 808b for the gradation converted image Iconv is expressed as Iconv_th. Iin_th is the perceived edge / texture amount of the image in the environment at the time of imaging, and Iconv_th is the perceived edge / texture amount of the image in the environment at the time of display.

<Operation of Difference Calculation Unit 809>
Next, the operation of the difference calculation unit 809 will be described. The similarity calculation unit 508 calculates a difference between the perceived edge / texture amount of the input image Iin and the perceived edge / texture amount of the gradation-converted image Iconv. Thereby, it is possible to express numerically how much the edge and texture that can be perceived in the environment at the time of imaging cannot be perceived in the display environment after gradation conversion.
Specifically, the similarity c is obtained by the following equation (15) from the output Iin_th of the visual characteristic application unit 808a and the output Iconv_th of the visual characteristic application unit 808b. In the present embodiment, the similarity is calculated based on the sum of absolute differences between Iin_th and Iconv_th.

In the present embodiment, the similarity is calculated based on the sum of absolute differences between the perceived edge / texture amounts Iin_th and Iconv_th, but the present invention is not limited to this. For example, a correlation coefficient or SSIM (Structural Similarity) may be calculated for both.
As described above, the similarity calculation unit 204 (FIG. 2) receives a plurality of images Iconv subjected to gradation conversion with a plurality of parameter candidates, so that the visual similarity calculation unit 504 of the similarity calculation unit 204 The similarity calculation described above is executed for each of the plurality of images Iconv. That is, the difference calculation unit 809 of the visual similarity calculation unit 504 calculates a plurality of similarities. Then, the difference calculation unit 809 obtains information related to Iconv having the highest similarity among the calculated plurality of similarities (information on the optimum parameter as a parameter for gradation conversion) via the output terminal 810. Output to the key conversion processing unit 203 (FIG. 2). Based on this information, the gradation conversion processing unit 203 displays the image Iconv having the highest similarity among the plurality of images Iconv input from the gradation conversion processing execution unit 202 via the output terminal 205 ( 1).

According to the present embodiment, the perceived edge / texture amount in the environment at the time of imaging (perceived edge / texture amount of the high gradation image) and the perceived edge / texture amount in the environment at the time of display (low gradation image after gradation conversion) The degree of similarity is calculated from the difference from the perceived edge / texture amount. Then, by selecting the gradation conversion having the maximum degree of similarity, the optimum gradation conversion image can be displayed on the display of the display unit 114.
In addition, verification whether appropriate gradation conversion is performed is performed as follows, for example. First, the display luminance of the display unit 114 is set to a predetermined value, and the input image is subjected to gradation conversion and displayed on the display of the display unit 114. Thereafter, the luminance of the display is changed, the input image is subjected to gradation conversion, and displayed on the display of the display unit 114. If there is no change in the image displayed on the display unit 114 before and after the change in luminance, it can be said that appropriate gradation conversion has been performed.

(Embodiment 2)
The visual similarity calculation unit 504 according to the first embodiment divides an image for each frequency band, generates a divided image, and applies a different threshold value for each frequency band (performs threshold processing), thereby perceiving edge and texture amount. Was calculated. However, if an image is divided for each frequency band and threshold processing is performed, the amount of calculation increases. Therefore, in the second embodiment, an example will be described in which the processing of the visual similarity calculation unit 504 is simplified and the calculation amount is reduced. More specifically, the perceptual edge / texture amount is calculated by performing frequency analysis such as Fourier transform on the image and multiplying by the visual characteristic function. In the following description, differences from the first embodiment will be mainly described. The difference from the first embodiment is the processing contents of the visual characteristic determination units 503a and 503b, the frequency analysis units 807a and 807b, and the visual characteristic application units 808a and 808b. Since the configurations of the visual characteristic determination units 503a and 503b and the visual similarity calculation unit 504 are the same as those in the first embodiment, the second embodiment will be described with reference to FIGS.

<Operation of Frequency Analysis Unit 807a>
Reference is first made to FIG. The image Iin is input to the first input terminal 801 of the visual similarity calculation unit 504. The contrast sensitivity characteristic calculated by the visual characteristic determination unit 503a is input to the third input terminal 803. The image Iconv is input to the second input terminal 802. The contrast sensitivity characteristic calculated by the visual characteristic determination unit 503b is input to the fourth input terminal 804. The image Iin input to the first input terminal 801 is converted into a luminance image by the imaging environment luminance conversion unit 805. The image Iconv input to the second input terminal 802 is converted into a luminance image by the display environment luminance conversion unit 806. The steps so far are the same as those in the first embodiment.

  In the second embodiment, the frequency analysis unit 807a performs frequency analysis of the luminance image by Fourier transforming the luminance image. Compared with the first embodiment, the second embodiment does not generate the band division filter, the multiplication process, and the inverse Fourier transform process. Therefore, Embodiment 2 can greatly reduce the amount of calculation. Hereinafter, the frequency analysis image obtained by the processing of the second embodiment is represented as Ifft (u, v). Here, u represents a frequency component in the horizontal direction (x direction), and v represents a frequency component in the vertical direction (y direction). The frequency analysis unit 807b performs the same process on the output from the display environment luminance conversion unit 806.

<Operation of the visual characteristic determination unit 503a>
Next, the processing of the visual characteristic determination unit 503a will be described with reference to FIG. In the second embodiment, unlike the first embodiment, the visual characteristic determination unit 503a determines a two-dimensional visual characteristic VTF (u, v) for the image Iin. In the first embodiment, one-dimensional visual characteristics (contrast characteristics) are used.
In the following, the space VTF derived by Barten's model is used. From Barten's space VTF, contrast sensitivity characteristics VTFs (u, v) for frequency components u and v are given by the following equation (16).

Here, X _o is a viewing angle [degree]. Further, L is the luminance, and in this embodiment, it is the average luminance of the calculated luminance image. The visual characteristic determination unit 503b performs the same processing as the visual characteristic determination unit 503a on the image Iconv.
In the above description, the space VTF derived by the Barten model is used, but the present invention is not limited to this. For example, the contrast sensitivity characteristic acquired by changing the luminance of the subject may be measured in advance, recorded as data, and referred to.

<Operation of Visual Characteristic Application Unit 808a>
The operation of the visual characteristic application unit 808a will be described with reference to FIG. In the present embodiment, the visual characteristic application unit 808a multiplies the output Ifft (u, v) of the frequency analysis unit 807a by vtf (u, v) obtained by the visual characteristic determination unit 503a. Further, the result Ith is subjected to inverse Fourier transform to calculate an image Ith after application of visual characteristics. Specifically, the image Ith after applying the visual characteristic is obtained by the following equation (17).

Here, ift represents an inverse Fourier transform. The image Ith generated by the visual characteristic application unit 808a is a perceptual edge / texture amount, and is input to the difference calculation unit 809. The visual characteristic application unit 808b performs the same processing as the visual characteristic application unit 808a on the output of the frequency analysis unit 807b.
Therefore, according to the second embodiment, the processing in the visual similarity calculation unit 504 can be simplified.

(Other embodiments)
In the first embodiment, band division is performed by performing multiplication in the frequency domain, but it is also possible to perform band division by a convolution operation in the spatial domain. Further, it is possible to divide into frequency bands as shown in FIG. 12 by applying a known band division filter such as a wavelet. FIG. 12 is a diagram in which the frequency band division by the band division filter is represented by a half-value frequency.
The digital camera may be a digital camera still camera or a digital video camera. A digital camera is an example of an imaging device. The present invention can be applied not only to digital cameras but also to mobile phones, tablet terminals, game machines, and the like.

  Furthermore, execution of software (program) as described below is also included in the scope of the present invention. That is, software (program) that realizes the functions of the above-described embodiment is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. This is included in the present invention.

DESCRIPTION OF SYMBOLS 101 ... Imaging part, 108 ... CPU, 114 ... Display part, 202 ... Gradation conversion process execution part, 203 ... Gradation conversion process determination part, 204 ... Similarity calculation part

Claims (14)

Calculating means for calculating the similarity between the input image and a processed image obtained by performing predetermined processing on the input image based on the frequency characteristics and contrast characteristics of the input image and the processed image;
Determining means for determining a parameter for gradation conversion of the input image based on the calculated similarity;
Conversion means for performing gradation conversion of the input image based on the determined parameter;
An image processing apparatus comprising: First characteristic determining means for determining contrast characteristics of the input image;
Second characteristic determining means for determining a contrast characteristic of the processed image;
The image processing apparatus according to claim 1, further comprising: First acquisition means for acquiring luminance information when capturing the input image;
A second acquisition means for acquiring luminance information when displaying the processed image,
The first characteristic determining means determines a contrast characteristic of the input image based on luminance information at the time of imaging,
The second characteristic determining means determines a contrast characteristic of the processed image based on luminance information at the time of display.
The image processing apparatus according to claim 1, wherein First luminance image generation means for generating a first luminance image representing luminance information at the time of imaging from the input image;
First analysis means for performing frequency analysis of the first luminance image;
First image generation means for generating a first image by reflecting the contrast characteristics of the input image in the first analysis result obtained from the first analysis means;
Second luminance image generation means for generating a second luminance image representing luminance information at the time of display from the processed image;
Second analysis means for performing frequency analysis of the second luminance image;
A second image generation means for generating a second image by reflecting the contrast characteristics of the processed image in the second analysis result obtained from the second analysis means;
4. The calculation unit according to claim 1, wherein the calculation unit calculates the similarity by calculating a difference between the first image and the second image. 5. Image processing device.   The image processing apparatus according to claim 4, wherein a frequency characteristic of the input image is included in the first analysis result, and a frequency characteristic of the processed image is included in the second analysis result.   The first image generation means performs a first threshold value processing based on a contrast characteristic of the input image on the first analysis result, and the second image generation means performs the second analysis result. The image processing apparatus according to claim 4, wherein second threshold processing based on a contrast characteristic of the processed image is performed on the image processing apparatus.   The first image generation unit multiplies the first analysis result by a contrast characteristic of the input image, and the second image generation unit performs the process on the second analysis result. 6. The image processing apparatus according to claim 4, wherein the contrast characteristic of the image is multiplied.   The image processing apparatus according to claim 1, wherein the input image is an image having a higher gradation than the processed image.   The image processing apparatus according to claim 1, wherein the contrast characteristic is a one-dimensional or two-dimensional contrast characteristic.   The image processing apparatus according to claim 1, wherein the input image is an image captured in an HDR (High Dynamic Range).   The image processing apparatus according to claim 4, wherein a difference between the first image and the second image is small in a low gradation region of the image. Calculating a similarity between the input image and a processed image obtained by performing a predetermined process on the input image based on frequency characteristics and contrast characteristics of the input image and the processed image,
Determining a parameter for gradation conversion of the input image based on the calculated similarity;
Performing gradation conversion of the input image based on the determined parameter;
An image processing method comprising: An image processing device according to any one of claims 1 to 11,
Imaging means for imaging an input image of the image processing device;
Display means for displaying an image subjected to gradation conversion by the image processing device;
Control means for controlling the imaging unit and the display unit;
An imaging apparatus comprising:   The program for functioning the said computer as each part of the image processing apparatus as described in any one of Claims 1 thru | or 11 when a computer reads and executes.

Images