JP2024052316A - Image processing device, image processing method, and computer program - Google Patents

Abstract

[Problem] To provide an image processing device, method, and program for synthesizing a plurality of images while effectively suppressing image quality problems. [Solution] In the image processing device, a synthesizing unit 202 includes DCT units 301a and 301b functioning as frequency space transforming means for transforming a first real space image and a second real space image into frequency space, respectively, and generating a first frequency image and a second frequency image, a contrast calculation unit 303 functioning as synthesis rate determination means for determining a synthesis rate of a plurality of frequency components included in the first frequency image and the second frequency image, a first synthesis rate calculation unit 302 and a first synthesis rate correction unit 304 functioning as synthesis rate determination means, a weighted addition unit 305 functioning as synthesis means for synthesizing the first frequency image and the second frequency image based on the synthesis rate of the plurality of frequency components to obtain a third frequency image, and a contrast calculation unit 303 calculating a contrast value of the first real space image. The synthesis rate determination means determines the synthesis rate of the plurality of frequency components according to the contrast value. [Selected Figure] FIG.

Description

The present invention relates to an image processing device, an image processing method, a computer program, etc. for image synthesis.

Ensuring a sufficient exposure time is an effective way to obtain images with little noise using a digital camera. However, if the exposure time is long, there is a problem that the image will become blurred due to camera movement caused by camera shake or subject movement.

An electronic blur correction method has been proposed as a method for dealing with this type of blur. With electronic blur correction, a low-noise image can be obtained by taking multiple consecutive shots with short exposure times that cause less blur, and then combining the resulting images. In addition, as shown in Patent Document 1, by lowering the composition rate in areas where there is a large difference between the images, it is possible to prevent the subject from appearing multiple times in the composite result.

This makes it possible to obtain an image with less blur. In particular, as shown in Non-Patent Document 1, a method is known in which the difference between images is calculated for each frequency component, and the synthesis rate is lowered for frequency components with large differences, thereby obtaining a low-noise image while maintaining the subject in good condition.

JP 2008-99260 A

Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, Marc Levoy, "Burst photography for high dynamic range and low-light imaging on mobile cameras", ACM Transactions on Graphics, Volume 35, Issue 6, November 2016, Article No. 192, pp. 1-12

However, when combining frequency components with different combining rates, the influence of certain frequency components in the combined result can become relatively strong or weak, resulting in image quality problems such as wavy patterns or ringing that are not present in the original subject.

The present invention aims to provide an image processing device that can effectively suppress image quality degradation and synthesize multiple images.

The image processing device of the present invention comprises:
a frequency space transforming means for transforming the first real space image and the second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image;
a synthesis rate determination means for determining a synthesis rate of a plurality of frequency components included in the first frequency image and the second frequency image;
a synthesis means for synthesizing the first frequency image and the second frequency image based on a synthesis rate of the plurality of frequency components to obtain a third frequency image;
a contrast calculation means for calculating a contrast value of the first real space image,
The synthesis rate determining means determines a synthesis rate of the plurality of frequency components in accordance with the contrast value.

The present invention makes it possible to realize an image processing device that can effectively suppress image quality degradation and synthesize multiple images.

1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment of the present invention; 2 is a functional block diagram showing an example of a configuration for performing synthesis processing of image signals of a plurality of frames in an image processing unit 107 according to the first embodiment of the present invention. FIG. 3 is a functional block diagram showing an example of the configuration of a synthesis unit 202 in FIG. 2 . 2A to 2C are diagrams showing examples of a plurality of small regions extracted by a small region extraction unit 201a; 13 is a diagram showing an example of a table indicating the correspondence relationship between a difference value D and a synthesis rate SIG_RATIO. FIG. 13 is a diagram showing an example of a table indicating the correspondence relationship between a contrast value SIG_C and a correction coefficient W; FIG. FIG. 13 is a diagram illustrating an example of a synthesis rate SIG_CORR. FIG. 11 is a functional block diagram illustrating an example of the configuration of a synthesis unit 202 according to a second embodiment. FIG. 4 is a diagram illustrating an example of a synthesis rate SIG_CORR obtained by Equation 3. FIG. 11 is a diagram showing an example of a synthesis rate SIG_CORR obtained when using Equation 4. FIG. 13 is a functional block diagram illustrating an example of the configuration of a synthesis unit 202 according to a third embodiment. FIG. 13 is a functional block diagram illustrating an example of the configuration of a synthesis unit 202 according to a fourth embodiment.

The following describes the embodiment of the present invention using examples with reference to the drawings. However, the present invention is not limited to the following examples. In each drawing, the same members or elements are given the same reference numbers, and duplicate descriptions are omitted or simplified.

Example 1
Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing an example of the configuration of an imaging device 100 as an image processing device according to a first embodiment of the present invention. The example of the configuration of the first embodiment will be described below with reference to FIG. 1.

In the first embodiment, the imaging device 100 is shown as an example of an image processing device, but the image processing device is not limited to this. For example, the image processing according to the present invention may be performed by an image processing device such as a personal computer or an application server.

The control unit 101 is, for example, a CPU, and reads out a control program for each block of the imaging device 100 from a ROM 102 (described later), and deploys it in a RAM 103 (described later) for execution. In this way, the control unit 101 controls the operation of each block of the imaging device 100.

The ROM 102 is a non-volatile memory that can be electrically erased and recorded, and stores the operating programs of each block of the imaging device 100 as well as parameters required for the operation of each block.

RAM 103 is a rewritable volatile memory, and is used for expanding programs executed by the control unit 101, etc., and for temporarily storing data generated by the operation of each block of the imaging device 100, etc.

The optical system 104 is composed of a group of lenses including a zoom lens and a focus lens, and forms an image of a subject on the imaging surface of the imaging unit 105 described below. The imaging unit 105 is an imaging element such as a CCD sensor or a CMOS sensor, and photoelectrically converts the optical image formed on the imaging surface of the imaging unit 105 by the optical system 104, and outputs the obtained analog image signal to the A/D conversion unit 106.

The A/D conversion unit 106 converts the input analog image signal into digital image data. The digital image data output from the A/D conversion unit 106 is temporarily stored in the RAM 103.

The image processing unit 107 performs various types of image processing on the image data stored in the RAM 103. Specifically, it applies various types of image processing, such as demosaicing processing, white balance correction processing, gamma processing, and tone mapping processing, to develop, display, and record digital image data. It also applies various types of image processing to improve the image quality of the digital image data, such as noise suppression processing using spatial filters and synthesis of multiple pieces of image data.

The recording unit 108 records data including image data on a built-in recording medium. The communication unit 109 wirelessly connects to an external device and communicates image data and the like. The display unit 110 includes a display device such as an LCD, and displays images stored in the RAM 103 and images recorded in the recording unit 108 on the display device. The display unit 110 also displays an operation user interface for receiving instructions from the user.

The instruction input unit 111 is an input interface that includes various physical operating components such as a touch panel and a shutter button, and accepts instructions input by the user.

Fig. 2 is a functional block diagram showing an example of the configuration for performing synthesis processing of image signals of multiple frames in the image processing unit 107 in the first embodiment of the present invention. Also, Fig. 3 is a functional block diagram showing an example of the configuration of the synthesis unit 202 in Fig. 2. Below, an example of the configuration for image signal processing in the first embodiment of the present invention will be described with reference to Figs. 2 and 3.

Some of the functional blocks shown in Figures 2 and 3 are realized by having a CPU, which serves as a computer included in the control unit of the imaging device 100, execute a computer program stored in a ROM 102, which serves as a storage medium. However, some or all of these may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP), etc., may be used.

Furthermore, the functional blocks shown in Figures 2 and 3 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths. The above explanation regarding Figures 2 and 3 also applies to Figures 8, 11, and 12.

Block 200 in Figure 2 that performs image signal synthesis processing receives image signal SIG_B, image signal SIG_R, and small area coordinate data as inputs, and outputs output signal SIG_O that synthesizes SIG_B and SIG_R. For the sake of explanation, in Example 1, it will be described as synthesizing a total of two frames of images, SIG_B and SIG_R. Also, SIG_B (first overall image) and SIG_R (second overall image) are image signals obtained by continuous shooting, and are assumed to have similar subjects.

The small region extraction unit 201a extracts a small region image signal SIG_B_P (first real space image) from the image signal SIG_B by referring to the small region coordinate data. Here, the small region coordinate data and the specific operation of the small region extraction unit 201a will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of multiple small regions extracted by the small region extraction unit 201a.

In FIG. 4, the multiple small regions are arranged in both the horizontal and vertical directions, and FIG. 4 shows only a portion of the multiple small regions. As shown in FIG. 4, the multiple small region image signals SIG_B_P are extracted by dividing the image signal SIG_B into rectangular blocks of a predetermined size. A rectangle of a predetermined size is, for example, a rectangle of 16 pixels horizontally and 16 pixels vertically.

The small area coordinate data here is data indicating the coordinates of the upper left corner of the above-mentioned small area. The small area coordinate data is calculated for each small area in sequence, for example, by a clock circuit so as to obtain the above-mentioned coordinates, and is input in sequence to the small area extraction unit 201a. The small area extraction unit 201a takes the coordinates indicated in the small area coordinate data as the upper left corner of the small area, and sequentially extracts small areas of the above-mentioned predetermined size.

This allows the small region extraction unit 201a to sequentially extract multiple small region image signals SIG_B_P (first real space image) from the image signal SIG_B (first overall image) in the positional relationship shown in FIG. 4. This concludes the explanation of the small region coordinate data and the specific operation of the small region extraction unit 201a.

Returning to the explanation of FIG. 2, the small region extraction unit 201b operates in a similar manner to the small region extraction unit 201a, and extracts a small region image signal SIG_R_P (second real space image) from the image signal SIG_R (second overall image) by referring to the small region coordinate data.

Here, the small area extraction unit 201a and the small area extraction unit 201b function as a small area extraction means for extracting a small area image from an image, and respectively extract a first real space image from a first overall image and extract a second real space image from a second overall image different from the first overall image.

The synthesis unit 202 synthesizes the small area image signal SIG_B_P (first real space image) and the small area image signal SIG_R_P (second real space image) to generate a small area image signal SIG_M_P (third real space image). Note that the synthesis unit 202 receives the small area image signal SIG_R_P of the same coordinates corresponding to the small area image signal SIG_B_P of each coordinate indicated by the small area coordinate data. For example, it is possible to input the small area image signal SIG_R_P of the same coordinates corresponding to the small area image signal SIG_B_P by control using a delay circuit.

The small area placement unit 203 places a small area image signal SIG_M_P (third real space image) based on the small area coordinate data, and outputs SIG_O (third overall image). Here, the small area placement unit 203 functions as a small area placement means that places multiple small area images to generate an overall image, and places multiple third real space images to generate the third overall image.

Specifically, the small region placement unit 203 replaces the region in the output signal SIG_O whose upper left corner is the coordinate indicated by the small region coordinate data with the small region image signal SIG_M_P, updates the output signal SIG_O, and outputs it. By repeating this process, the small region image signals SIG_M_P can be sequentially placed so as to achieve the positional relationship described in FIG. 4.

The small region placement unit 203 receives the small region image signal SIG_M_P corresponding to the small region image signal SIG_B_P at each coordinate indicated by the small region coordinate data. This allows the small region image signal SIG_M_P to be placed in the same region as when the small region extraction unit 201a extracted the small region image signal SIG_B_P from the image signal SIG_B.

The block that performs the synthesis process of the image signal in FIG. 2 performs processing on all the small area coordinate data from the small area extraction unit 201a and the small area extraction unit 201b to the small area placement unit 203, thereby generating the final output signal SIG_O. Note that all the small area coordinate data is, for example, small area coordinate data that indicates all the coordinates for extracting small areas from the entire area of the image signal SIG_B or the image signal SIG_R. This concludes the explanation of the block that performs the synthesis process of the image signal in FIG. 2.

Figure 3 shows an example of the configuration of the synthesis unit 202, and the operation of each block will be described. The DCT unit 301a performs discrete cosine transform (frequency transform) on the small-area image signal SIG_B_P (first real space image) and outputs a frequency space signal SIG_B_F (first frequency image). The discrete cosine transform is performed on both the horizontal and vertical directions of the small-area image signal SIG_B_P.

As a result, if the small region image signal SIG_B_P is a rectangle with a size of 16 pixels horizontally and 16 pixels vertically as described above, the frequency space signal SIG_B_F will have, for example, 16 frequency components horizontally and 16 frequency components vertically. Furthermore, by combining the horizontal and vertical components, the frequency space signal SIG_B_F will have a total of 256 frequency components.

The DCT unit 301b performs a discrete cosine transform (frequency transform) on the small region image signal SIG_R_P (second real space image) by performing an operation similar to that of the DCT unit 301a, and outputs a frequency space signal SIG_R_F (second frequency image).

Here, the DCT unit 301a and the DCT unit 301b function as a frequency space transformation means that executes a frequency space transformation step of transforming the first real space image and the second real space image into frequency space, respectively, to generate a first frequency image and a second frequency image.

The first combination ratio calculation unit 302 calculates and outputs a combination ratio SIG_RATIO for each frequency component based on the frequency space signal SIG_B_F and the frequency space signal SIG_R_F. Specifically, the first combination ratio calculation unit 302 first calculates a difference value D between the frequency space signal SIG_B_F and the frequency space signal SIG_R_F for each frequency component.

Next, the first synthesis rate calculation unit 302 refers to a table showing the correspondence between the difference value D and the synthesis rate, and calculates the synthesis rate SIG_RATIO for each frequency component so that the synthesis rate is large when the difference value D is small and the synthesis rate is small when the difference value D is large.

Figure 5 is a diagram showing an example of a table showing the correspondence between the difference value D and the synthesis rate SIG_RATIO. The horizontal axis of the graph in Figure 5 shows the difference value D, and the vertical axis shows the synthesis rate SIG_RATIO. Note that the synthesis rate SIG_RATIO is shown as a value range of 0 to 1.

The contrast calculation unit 303 calculates and outputs a contrast value SIG_C from the small region image signal SIG_B_P. Here, the contrast calculation unit 303 functions as a contrast calculation means that executes a contrast calculation step that determines the synthesis rate of multiple frequency components.

Specifically, the contrast calculation unit 303 first finds the maximum pixel value and the minimum pixel value from the multiple pixels contained in the small region image signal SIG_B_P. Next, the contrast calculation unit 303 outputs a value obtained by subtracting the minimum pixel value from the maximum pixel value as the contrast value SIG_C. In other words, the contrast value is calculated from the difference between the maximum pixel value and the minimum pixel value of the first real space image.

The first synthesis rate correction unit 304 corrects the synthesis rate SIG_RATIO for each frequency component based on the contrast value SIG_C, and outputs the synthesis rate SIG_CORR for each frequency component after correction. Specifically, the first synthesis rate correction unit 304 first refers to a table showing the correspondence between contrast values and correction coefficients, and calculates a correction coefficient W so that the correction is weak when the contrast value SIG_C is small and the correction is strong when the contrast value SIG_C is large.

The correction coefficient W has a value range of 0 to 1, and the closer the value is to 0, the stronger the correction. FIG. 6 is a diagram showing an example of a table showing the correspondence between the contrast value SIG_C and the correction coefficient W. The horizontal axis of the graph in FIG. 6 represents the contrast value SIG_C, and the vertical axis represents the correction coefficient W.

Next, the first synthesis rate correction unit 304 calculates the average value M of the synthesis rates SIG_RATIO of all frequency components based on the synthesis rate SIG_RATIO for each frequency component. Next, the first synthesis rate correction unit 304 corrects the synthesis rate SIG_RATIO for each frequency component based on the correction coefficient W and the average value M, and outputs the corrected synthesis rate SIG_RATIO.

Specifically, the first synthesis rate correction unit 304 performs a calculation, for example, as shown in the following Equation 1 on the synthesis rate SIG_RATIO for each frequency component, and obtains and outputs the synthesis rate SIG_CORR for each frequency component after correction.
SIG_CORR=W×SIG_RATIO+(1−W)×M (Formula 1)

Figure 7 is a diagram showing an example of the synthesis rate SIG_CORR, and shows an example of the synthesis rate SIG_CORR obtained by processing of the first synthesis rate calculation unit 302 and processing of the first synthesis rate correction unit 304. The horizontal axis of the graph in Figure 7 represents the difference value D, and the vertical axis represents the synthesis rate SIG_CORR.

The solid line 701 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is at its minimum. The dashed line 702 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is at its maximum. The dashed line 703 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is in an intermediate state.

In this way, in the synthesis rate SIG_CORR, the greater the contrast value SIG_C, the smaller the fluctuation in value due to the difference value D, resulting in a more uniform synthesis rate (synthesis rate M) between frequency components compared to when the contrast value SIG_C is small. In other words, the synthesis rate of the multiple frequency components is determined so that the synthesis rate of the multiple frequency components approaches uniformity as the contrast value increases. Note that the synthesis rate of the multiple frequency components may be determined so that the synthesis rate of at least one of the multiple frequency components decreases as the contrast value increases.

Here, the first synthesis rate calculation unit 302 and the first synthesis rate correction unit 304 function as synthesis rate determination means for executing a synthesis rate determination step for determining the synthesis rate of the multiple frequency components included in the first frequency image and the second frequency image. In addition, in this embodiment, the synthesis rate determination means determines the synthesis rate of the multiple frequency components according to the contrast value such that the synthesis rate between the multiple frequency components approaches a uniform value as the contrast value increases.

The weighted addition unit 305 synthesizes the frequency space signal SIG_B_F and the frequency space signal SIG_R_F based on the synthesis rate SIG_CORR for each frequency component to generate and output the frequency space signal SIG_M_F (third frequency image). Here, the weighted addition unit 305 functions as a synthesis means that executes a synthesis step of synthesizing the first frequency image and the second frequency image based on the synthesis rate of multiple frequency components to obtain a third frequency image.

Specifically, the weighted addition unit 305 performs the calculation shown in the following Equation 2 for each frequency component, for example, to generate a frequency space signal SIG_M_F.
SIG_M_F={(2-SIG_CORR)×SIG_B_F+SIG_CORR×SIG_R_F}÷2 (Formula 2)

The inverse DCT unit 306 performs an inverse discrete cosine transform on the frequency space signal SIG_M_F (third frequency image) and outputs a small region image signal SIG_M_P (third real space image) in real space. The inverse discrete cosine transform is performed on the horizontal and vertical components of the frequency space signal SIG_M_F. Here, the inverse DCT unit 306 functions as a real space transforming means that transforms the third frequency image into real space and generates a third real space image.

The above is an explanation of the synthesis unit 202 in FIG. 3. As a result, the image processing unit 107 can bring the synthesis rate closer to a uniform state (synthesis rate M) between frequency components in parts of the image signal SIG_B where the contrast is high.

In high-contrast areas of an image, the amplitude of the frequency components tends to be large in the frequency space signal SIG_B_F and the frequency space signal SIG_R_F, and as a result, the amplitude of ringing, which is an image quality hazard, tends to be large and noticeable. However, according to this embodiment, the synthesis rate can be made closer to a uniform state (synthesis rate M) between frequency components in high-contrast areas of an image.

This prevents the influence of certain frequency components from becoming relatively too strong or too weak in the composite result, and suppresses the occurrence of ringing. Also, in areas with low contrast, it makes it easier to composite while maintaining the subject in good condition.

In the first embodiment, the synthesis rate for each frequency component is adjusted using the method shown in Equation 1. However, this embodiment is not limited to this, and the synthesis rate may be adjusted by any method as long as it is possible to make the synthesis rate approach a uniform state between frequency components when the contrast of the image is high.

For example, instead of the method described in Equation 1, the first synthesis rate correction unit 304 may apply a low-pass filter in the spatial direction to the synthesis rate SIG_RATIO for each frequency component to obtain the corrected synthesis rate SIG_CORR for each frequency component.

In addition, in the first embodiment, an example was shown in which the contrast value was calculated based on the small-area image signal SIG_B_P, but this embodiment is not limited to this. For example, the contrast value may be calculated based on the small-area image signal SIG_R_P, or may be calculated based on both the small-area image signal SIG_B_P and the small-area image signal SIG_R_P.

In addition, in the first embodiment, an example is shown in which signals transformed into frequency space by discrete cosine transform are synthesized, but any method for synthesizing frequency components may be used. For example, signals transformed into frequency space by Fourier transform may be synthesized.

In addition, in the first embodiment, an example of synthesizing two frames of image signal SIG_B and image signal SIG_R is shown, but this embodiment can be applied regardless of the number of frames to be synthesized. For example, when synthesizing three or more frames, multiple small area image signals SIG_R_P may be extracted from multiple image signals SIG_R, and the synthesis unit 202 may synthesize the small area image signal SIG_B_P and the multiple small area image signals SIG_R_P.

In this case, the synthesis unit 202 calculates the synthesis rate SIG_CORR for each of the frequency space signals SIG_R_F of each frame. In addition, the weighted addition unit 305 performs weighted addition of the frequency space signal SIG_B_F and the frequency space signal SIG_R_F of each frame according to the synthesis rate SIG_CORR of each frame.

Example 2
In the first embodiment, an example was shown in which the synthesis rates between frequency components are made closer to a uniform state when the contrast of an image is high. In the second embodiment, the first embodiment is modified so that the synthesis rates of at least some of the frequency components are made smaller when the contrast of an image is high.

Unless otherwise specified, the same reference numerals as in the first embodiment will operate and process in the same manner as in the first embodiment. The basic configuration of the second embodiment is the same as that of the first embodiment, and so a description of the configuration will be omitted.

In the second embodiment, the configuration of the synthesis unit 202 shown in FIG. 3 in the first embodiment is modified as shown in FIG. 8. FIG. 8 is a functional block diagram showing an example of the configuration of the synthesis unit 202 in the second embodiment.

The second synthesis rate calculation unit 801 calculates and outputs the synthesis rate SIG_CORR for each frequency component based on the frequency space signal SIG_B_F, the frequency space signal SIG_R_F, and the contrast value SIG_C.

Specifically, the second combining rate calculation unit 801 first calculates a difference value D between the frequency space signal SIG_B_F and the frequency space signal SIG_R_F for each frequency component. Next, the second combining rate calculation unit 801 obtains and outputs a combining rate SIG_CORR for each frequency component, for example, by the calculation shown in the following Equation 3.
SIG_CORR=1−D÷{D+F×(G−SIG_C)+H} (Formula 3)

Here, F (F>0) in Equation 3 is a constant that adjusts the degree of influence of the contrast value SIG_C on the blending rate SIG_CORR. Also, G is the maximum value that SIG_C can take. Also, H (H>0) is a tiny value that prevents division by 0.

Figure 9 is a diagram showing an example of the combination rate SIG_CORR obtained by Equation 3. The horizontal axis of the graph in Figure 9 represents the difference value D, and the vertical axis represents the combination rate SIG_CORR. In addition, the solid line 901 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is at a minimum.

The dashed line 902 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is at its maximum. The dashed line 903 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is in an intermediate state.

Equation 3 is an example of determining the combination rate SIG_CORR based on the combined condition of the difference value D and the contrast value SIG_C. The larger the difference value D, the smaller the combination rate SIG_CORR. Also, the larger the contrast value SIG_C, the smaller the combination rate SIG_CORR.

In high-contrast areas of an image, the amplitude of the frequency components in the frequency space signal SIG_B_F and the frequency space signal SIG_R_F tends to be large, and as a result, the amplitude of ringing, which is an adverse effect on image quality, becomes large and becomes more noticeable.

However, according to this embodiment, synthesis is suppressed in parts of the image with high contrast, and the small-area image signal SIG_M_P resulting from synthesis can be made closer to the small-area image signal SIG_B_P before synthesis, which is in a state where no ringing occurs. This makes it possible to suppress the occurrence of ringing.

Furthermore, when synthesis is performed using components with a large difference value D, ringing is likely to occur. However, in this embodiment, by using, for example, Equation 3, the synthesis rate SIG_CORR is made smaller when the contrast value SIG_C is large and the difference value D is large.

Therefore, it is possible to suppress synthesis, especially for frequency components that are likely to cause ringing. Also, for frequency components with small difference values D that are unlikely to cause ringing, the synthesis rate can be increased to reduce noise.

In the second embodiment, the synthesis rate for each frequency component is adjusted by the method shown in Equation 3. However, the synthesis rate may be adjusted by any method as long as it is possible to reduce the synthesis rate when the contrast of the image is high. For example, the synthesis rate may be reduced when the contrast is high, and the synthesis rate may be made closer to a uniform state between frequency components as described in the first embodiment.

In this case, the second synthesis rate calculation unit 801 first calculates the synthesis rate SIG_RATIO by the same operation as the first synthesis rate calculation unit 302 in the first embodiment. Next, the second synthesis rate calculation unit 801 calculates the correction coefficient W by the same operation as the first synthesis rate correction unit 304 in the first embodiment.

Thereafter, the second synthesis rate calculation unit 801 may obtain the synthesis rate SIG_CORR for each corrected frequency component by, for example, the calculation shown in Equation 4 below.
SIG_CORR=W×SIG_RATIO (Formula 4)

Figure 10 is a diagram showing an example of the combination rate SIG_CORR obtained when using Equation 4. The horizontal axis of the graph in Figure 10 represents the difference value D, and the vertical axis represents the combination rate SIG_CORR. In addition, the solid line 1001 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is at a minimum.

The dashed line 1002 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is at its maximum. The dashed line 1003 shows an example of the combination rate SIG_CORR when the contrast value SIG_C is in an intermediate state.

In this way, when the contrast value SIG_C is large, the synthesis rate can be made smaller for the same difference value D at the synthesis rate SIG_CORR compared to when the contrast value SIG_C is small. Also, when the synthesis rate SIG_CORR is large, the fluctuation in value due to the difference value D becomes smaller as the contrast value SIG_C is larger, making it easier to achieve a more uniform synthesis rate between frequency components compared to when the contrast value SIG_C is small.

This allows the second synthesis rate calculation unit 801 to calculate a synthesis rate such that the higher the contrast, the smaller the synthesis rate, and the higher the contrast, the closer the synthesis rate is to a uniform state between frequency components. This method can also effectively suppress the occurrence of ringing.

Example 3
In the third embodiment, the first embodiment is modified to make the synthesis rate between frequency components closer to a uniform state when an image contains a dark area. Unless otherwise specified, the same reference numerals as in the first embodiment perform the same operations and processes as in the first embodiment. Also, the basic configuration of the third embodiment is the same as that of the first embodiment, and therefore the description thereof will be omitted.

In the third embodiment, the configuration of the synthesis unit 202 shown in FIG. 3 in the first embodiment is modified as shown in FIG. 11. FIG. 11 is a functional block diagram showing an example of the configuration of the synthesis unit 202 in the third embodiment.

The dark area evaluation value calculation unit 1101 calculates and outputs a dark area evaluation value SIG_L (brightness evaluation value) from the small area image signal SIG_B_P. Specifically, the dark area evaluation value calculation unit 1101 first determines whether each pixel included in the small area image signal SIG_B_P is equal to or less than a predetermined threshold value. The predetermined threshold value is a pixel value that is determined in advance as a criterion for determining whether a pixel is a pixel in a dark area.

Here, the dark area evaluation value calculation unit 1101 functions as a brightness evaluation value calculation means that executes a brightness evaluation value calculation step that calculates a brightness evaluation value indicating the brightness level of an image by a process of determining pixels whose pixel values of the first real space image are equal to or less than a threshold value.

Next, the dark area evaluation value calculation unit 1101 counts the number of pixels that are determined to be equal to or less than a predetermined threshold value, and outputs the count as a dark area evaluation value SIG_L. That is, the dark area evaluation value calculation unit 1101 calculates the number of pixels whose pixel values in the first real space image are equal to or less than a predetermined threshold value as a luminance evaluation value.

The second combination rate correction unit 1102 corrects the combination rate SIG_RATIO for each frequency component based on the dark area evaluation value SIG_L, and outputs the combination rate SIG_CORR for each frequency component after correction. Here, the second combination rate correction unit 1102 functions as a combination rate determination means that executes a combination rate determination step that determines the combination rates of multiple frequency components according to the dark area evaluation value SIG_L as a luminance evaluation value.

Specifically, the second synthesis rate correction unit 1102 first refers to a table showing the correspondence between the dark area evaluation value and the correction coefficient, and calculates the correction coefficient W so that the correction is weak when the dark area evaluation value SIG_L is small, and the correction is strong when the dark area evaluation value SIG_L is large.

The correction coefficient W has a value range of 0 to 1, and the closer the value is to 0, the stronger the correction. The example table is the same as that in the case where the horizontal axis of the table in FIG. 6 described in the first embodiment is replaced with the dark area evaluation value SIG_L.

Next, the second synthesis rate correction unit 1102 calculates the average value M of the synthesis rates SIG_RATIO of all frequency components based on the synthesis rate SIG_RATIO for each frequency component. The second synthesis rate correction unit 1102 then obtains and outputs the synthesis rate SIG_CORR for each frequency component after correction by a calculation similar to that of Equation 1 in the first embodiment. As a result, when the synthesis rate SIG_CORR is large, a more uniform synthesis rate can be achieved between frequency components compared to when the dark area evaluation value SIG_L is small.

The above is an explanation of the synthesis unit 202 in the third embodiment of FIG. 11. With this configuration, the image processing unit 107 can make the synthesis rate closer to a uniform state between frequency components in the dark parts of the small region image signal SIG_B_P.

In dark areas of an image, the ratio of the maximum pixel value to the minimum pixel value that has changed due to ringing tends to be large, which makes ringing more noticeable. In addition, dark areas of an image may be gain-uped by tone mapping processing in the subsequent stage, which tends to emphasize ringing even more.

However, according to this embodiment, by making the synthesis rate closer to uniform between frequency components in the dark areas of the image, it is possible to suppress the state in which the influence of a specific frequency component becomes relatively too strong or too weak in the synthesis result, and to suppress the occurrence of ringing. Also, in areas other than the dark areas, it is possible to easily perform synthesis while maintaining the subject in good condition.

In the third embodiment, the dark area evaluation value is calculated based on the number of pixels below the threshold, but this is not limited thereto, and any method may be used to calculate the dark area evaluation value as long as the value represents the degree of darkness. For example, the average value of the weights of the pixels below the threshold may be used as the dark area evaluation value. In this case, the dark area evaluation value calculation unit 1101 first determines whether each pixel included in the small area image signal SIG_B_P is below a predetermined threshold.

Next, the dark area evaluation value calculation unit 1101 determines a weighting value for each pixel determined to be equal to or less than the above-mentioned predetermined threshold value so that the weighting value is larger for pixels that indicate a darker pixel value. Furthermore, the dark area evaluation value calculation unit 1101 calculates the average value of the weighting values for all pixels determined to be equal to or less than the above-mentioned predetermined threshold value, and sets this as the dark area evaluation value SIG_L.

That is, for one or more pixels in the first real space image whose pixel value is equal to or less than a threshold value, weights are calculated according to the pixel values, and the luminance evaluation value is calculated by summing the weights of the one or more pixels.

This method also allows the dark area evaluation value calculation unit 1101 to calculate a dark area evaluation value that indicates the degree of darkness. Alternatively, for example, the minimum pixel value may be used as the dark area evaluation value. In this case, the dark area evaluation value calculation unit 1101 extracts the minimum pixel value from among all pixels contained in the small area image signal SIG_B_P, and sets this as the dark area evaluation value SIG_L. In other words, the minimum pixel value of the first real space image is calculated as the luminance evaluation value.

Thereby, the dark area evaluation value calculation unit 1101 may calculate a dark area evaluation value that indicates the degree of the dark area. In this way, the dark area evaluation value calculation unit 1101 as a brightness evaluation value calculation means may calculate a brightness evaluation value that indicates the brightness level of the image by a process of calculating the minimum pixel value of the first real space image.

In this way, in the third embodiment, the synthesis rate is made closer to a uniform state between frequency components when the image contains dark areas, but as an alternative method, ringing can also be suppressed by reducing the synthesis rate when the image contains dark areas. In this case, the synthesis unit 202 calculates the synthesis rate by a procedure similar to that shown in the second embodiment, so that the synthesis rate of at least some frequency components is reduced when the dark area evaluation value SIG_L is larger instead of the contrast value SIG_C.

In this way, in the third embodiment, the synthesis rate of the multiple frequency components may be determined so that the synthesis rate approaches uniformity between the multiple frequency components as the luminance evaluation value indicates a darker luminance level, and the method for doing so is not limited. In this case, it is desirable to determine the synthesis rate of the multiple frequency components so that the synthesis rate of at least one of the multiple frequency components approaches 0 as the luminance evaluation value indicates a darker luminance level.

In addition, in the third embodiment, an example was shown in which the synthesis rate for each frequency component was calculated based on the dark areas of the image, but the synthesis rate may be calculated based on a combined condition with the image contrast shown in the first and second embodiments. For example, when an image includes dark areas and has high contrast, the synthesis rate may be adjusted so that it approaches a uniform value.

Example 4
In the fourth embodiment, the first embodiment is modified to make the synthesis rate between frequency components closer to a uniform state when the maximum value of the difference in real space between images to be synthesized is large. Unless otherwise specified, the same reference numerals as those in the first embodiment perform the same operations and processes as those in the first embodiment. Also, the basic configuration example of the fourth embodiment is the same as that of the first embodiment, and the description thereof will be omitted.

In the fourth embodiment, the configuration of the synthesis unit 202 shown in FIG. 3 in the first embodiment is modified as shown in FIG. 12. FIG. 12 is a functional block diagram showing an example of the configuration of the synthesis unit 202 in the fourth embodiment.

The maximum difference calculation unit 1201 calculates and outputs the maximum difference SIG_D based on the small area image signal SIG_B_P and the small area image signal SIG_R_P. Here, the maximum difference calculation unit 1201 functions as a difference calculation means that executes a difference calculation step that calculates the maximum value of the difference between corresponding pixels in the first real space image and the second real space image.

Specifically, the maximum difference calculation unit 1201 first calculates the absolute value of the difference between pixels at the same coordinates for all coordinates of the small area image signal SIG_B_P and the small area image signal SIG_R_P. Next, the maximum difference calculation unit 1201 extracts the maximum value from the absolute values of the differences calculated for all the aforementioned coordinates, and outputs it as the maximum difference value SIG_D.

The third synthesis rate correction unit 1202 corrects the synthesis rate SIG_RATIO for each frequency component based on the maximum difference value SIG_D, and outputs the synthesis rate SIG_CORR for each frequency component after correction. Here, the third synthesis rate correction unit 1202 functions as a synthesis rate determination means that executes a synthesis rate determination step that determines the synthesis rate of multiple frequency components according to the maximum difference value.

Specifically, the third synthesis rate correction unit 1202 first refers to a table showing the correspondence between the maximum difference value and the correction coefficient, and calculates the correction coefficient W so that the correction is weak when the maximum difference value SIG_D is small, and the correction is strong when the maximum difference value SIG_D is large.

The correction coefficient W has a value range of 0 to 1, and the closer the value is to 0, the stronger the correction. An example of the table is the same as that in the case where the horizontal axis of the table in FIG. 6 described in the first embodiment is replaced with the maximum difference value SIG_D. Next, the third synthesis rate correction unit 1202 calculates the average value M of the synthesis rates SIG_RATIO of all frequency components based on the synthesis rate SIG_RATIO for each frequency component.

The third synthesis rate correction unit 1202 obtains and outputs the synthesis rate SIG_CORR for each frequency component after correction by performing a calculation similar to that of Equation 1 in the first embodiment. As a result, when the synthesis rate SIG_CORR is a large maximum difference value SIG_D, it is possible to obtain a more uniform synthesis rate between frequency components compared to when the maximum difference value SIG_D is small.

In other words, the larger the maximum value of the difference, the closer the synthesis rate of the multiple frequency components is to a uniform value. It is desirable to determine the synthesis rate of the multiple frequency components so that the larger the maximum value of the difference, the closer the synthesis rate of at least one of the multiple frequency components is to 0.

The above is an explanation of the synthesis unit 202 in the fourth embodiment of FIG. 12. With this configuration, the image processing unit 107 can make the synthesis rate closer to a uniform state between frequency components in the area where the maximum value of the difference between the small region image signal SIG_B_P and the small region image signal SIG_R_P is large.

In areas where the difference between images in real space is large, the calculation of the synthesis rate for each frequency component by the first synthesis rate calculation unit 302 tends to result in large variations in the synthesis rate between frequency components, which tends to result in ringing, which is an adverse effect on image quality.

However, according to this embodiment, in areas where the maximum value of the difference between images in real space is large, the synthesis rate is made closer to a uniform state between frequency components. This makes it possible to prevent the influence of a specific frequency component in the synthesis result from becoming relatively strong or weak, and to suppress the occurrence of ringing. Furthermore, in areas where the maximum value of the difference between images in real space is small, synthesis can be performed more easily while maintaining the subject in good condition.

One method for finding the difference between images is to calculate the sum of absolute differences (SAD) between pixels. With SAD, even in low-contrast detail areas such as textures, slight shifts in the subject between the images can easily result in a large difference being determined.

Therefore, when the synthesis rate SIG_RATIO for each frequency component is corrected based on the SAD in a manner similar to that of Example 4, the synthesis rate tends to approach a uniform value in low-contrast detail areas, which weakens the effect of preserving the subject.

On the other hand, in this embodiment, the blending rate SIG_RATIO is corrected based on the maximum absolute value of the difference between pixels. Therefore, in low-contrast detail areas, the difference is less likely to be determined to be large, and the blending rate is less likely to approach a uniform value, making it easier to maintain the subject in good condition. Also, in other areas where the difference is large, it is easier to suppress the occurrence of ringing.

In addition, in the fourth embodiment, an example was shown in which the synthesis rate between frequency components is made closer to a uniform state when the maximum value of the difference in real space between the images to be synthesized is large. However, as an alternative method, ringing can also be suppressed by decreasing the synthesis rate when the maximum value of the difference in real space between the images to be synthesized is large.

In this case, the synthesis unit 202 calculates the synthesis rate by a procedure similar to that shown in the second embodiment so that the synthesis rate of at least some frequency components is reduced when the maximum difference value SIG_D is larger instead of the contrast value SIG_C.

In addition, in the fourth embodiment, an example was shown in which the synthesis rate for each frequency component was calculated based on the maximum value of the difference in real space between the images to be synthesized, but the synthesis rate may be calculated based on the contrast of the images shown in the first and second embodiments, or on a combination condition with the dark areas of the images shown in the third embodiment. For example, when the maximum value of the difference in real space between the images is large or when the images contain dark areas, the synthesis rate may be adjusted to approach a uniform value.

The present invention has been described in detail above based on preferred embodiments thereof, but the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention includes the following combinations.

(Configuration 1) An image processing device comprising: a frequency space conversion means for converting a first real space image and a second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image; a synthesis rate determination means for determining a synthesis rate of a plurality of frequency components contained in the first frequency image and the second frequency image; a synthesis means for synthesizing the first frequency image and the second frequency image based on the synthesis rate of the plurality of frequency components to obtain a third frequency image; and a contrast calculation means for calculating a contrast value of the first real space image, wherein the synthesis rate determination means determines the synthesis rate of the plurality of frequency components according to the contrast value.

(Configuration 2) The image processing device described in Configuration 1, characterized in that the synthesis rate determination means determines the synthesis rate of the multiple frequency components so that the synthesis rate between the multiple frequency components approaches a uniform value as the contrast value increases.

(Configuration 3) The image processing device according to configuration 1 or 2, characterized in that the synthesis rate determination means determines the synthesis rate of the multiple frequency components such that the synthesis rate of at least one of the multiple frequency components decreases as the contrast value increases.

(Configuration 4) The image processing device according to any one of configurations 1 to 3, characterized in that the contrast calculation means calculates the contrast value from the difference between the maximum pixel value and the minimum pixel value of the first real space image.

(Configuration 5) The image processing device according to any one of configurations 1 to 4, further comprising a small area extraction means for extracting a small area image from an image, the small area extraction means extracting the first real space image from a first overall image, and extracting the second real space image from a second overall image different from the first overall image.

(Configuration 6) The image processing device according to any one of configurations 1 to 5, further comprising a real space conversion means for converting the third frequency image into real space and generating a third real space image.

(Configuration 7) The image processing device further includes a small area arrangement means for arranging a plurality of small area images to generate an entire image,
7. The image processing apparatus according to configuration 6, wherein the small region arrangement means arranges a plurality of the third real space images to generate a third whole image.

(Configuration 8) An image processing device comprising: a frequency space conversion means for converting a first real space image and a second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image; a synthesis rate determination means for determining a synthesis rate of a plurality of frequency components contained in the first frequency image and the second frequency image; a synthesis means for synthesizing the first frequency image and the second frequency image based on the synthesis rate of the plurality of frequency components to obtain a third frequency image; and a luminance evaluation value calculation means for calculating a luminance evaluation value indicating a luminance level of an image by at least one of a process for calculating the minimum pixel value of the first real space image and a process for determining pixels whose pixel values of the first real space image are equal to or less than a threshold value, wherein the synthesis rate determination means determines the synthesis rate of the plurality of frequency components according to the luminance evaluation value.

(Configuration 9) The image processing device described in Configuration 8, characterized in that the synthesis rate determination means determines the synthesis rate of the multiple frequency components so that the synthesis rate between the multiple frequency components approaches uniformity as the luminance evaluation value indicates a darker luminance level.

(Configuration 10) The image processing device according to configuration 8 or 9, characterized in that the synthesis rate determination means determines the synthesis rate of the plurality of frequency components such that the synthesis rate of at least one of the plurality of frequency components approaches 0 as the luminance evaluation value indicates a darker luminance level.

(Configuration 11) The image processing device according to any one of configurations 8 to 10, wherein the luminance evaluation value calculation means calculates the number of pixels in the first real space image whose pixel values are equal to or less than the threshold value as the luminance evaluation value.

(Configuration 12) The image processing device according to any one of configurations 8 to 11, characterized in that the luminance evaluation value calculation means calculates weights according to pixel values for one or more pixels in the first real space image whose pixel values are equal to or less than the threshold value, and calculates the luminance evaluation value by summing up the weights for the one or more pixels.

(Configuration 13) The image processing device according to any one of configurations 8 to 12, characterized in that the luminance evaluation value calculation means calculates the minimum pixel value of the first real space image as the luminance evaluation value.

(Configuration 14) The image processing method further includes a small area extraction means for extracting a small area image from the image,
The image processing device according to any one of configurations 8 to 13, wherein the small region extraction means extracts the first real-space image from a first overall image, and extracts the second real-space image from a second overall image different from the first overall image.

(Configuration 15) The image processing device according to any one of configurations 8 to 14, further comprising a real space conversion means for converting the third frequency image into real space and generating a third real space image.

(Configuration 16) The image processing apparatus further includes a small area arrangement means for arranging a plurality of small area images to generate an entire image,
16. The image processing apparatus according to configuration 15, wherein the small region arrangement means arranges a plurality of the third real space images to generate a third whole image.

(Configuration 17) An image processing device comprising: a frequency space conversion means for converting a first real space image and a second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image; a synthesis rate determination means for determining a synthesis rate of a plurality of frequency components contained in the first frequency image and the second frequency image; a synthesis means for synthesizing the first frequency image and the second frequency image based on the synthesis rate of the plurality of frequency components to obtain a third frequency image; and a difference calculation means for calculating a maximum value of the difference between corresponding pixels of the first real space image and the second real space image, wherein the synthesis rate determination means determines the synthesis rate of the plurality of frequency components according to the maximum value of the difference.

(Configuration 18) The image processing device described in Configuration 17, characterized in that the synthesis rate determination means determines the synthesis rate of the multiple frequency components so that the synthesis rate between the multiple frequency components approaches a uniform value as the maximum value of the difference increases.

(Configuration 19) The image processing device according to configuration 17 or 18, characterized in that the synthesis rate determination means determines the synthesis rate of the plurality of frequency components such that the synthesis rate of at least one of the plurality of frequency components approaches 0 as the maximum value of the difference increases.

(Configuration 20) The method further comprises: extracting a small area image from the image;
The image processing device according to any one of configurations 17 to 19, wherein the small region extraction means extracts the first real-space image from a first overall image, and extracts the second real-space image from a second overall image different from the first overall image.

(Configuration 21) The image processing device according to any one of configurations 17 to 20, further comprising a real space conversion means for converting the third frequency image into real space and generating a third real space image.

(Configuration 22) A small area arrangement means for arranging a plurality of small area images to generate an entire image is further provided,
22. The image processing apparatus according to configuration 21, wherein the small region arrangement means arranges a plurality of the third real space images to generate a third whole image.

(Method 1) An image processing method including a frequency space transformation step of transforming a first real space image and a second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image, a synthesis rate determination step of determining a synthesis rate of a plurality of frequency components contained in the first frequency image and the second frequency image, a synthesis step of synthesizing the first frequency image and the second frequency image based on the synthesis rate of the plurality of frequency components to obtain a third frequency image, and a contrast calculation step of calculating a contrast value of the first real space image, wherein the synthesis rate determination step determines the synthesis rate of the plurality of frequency components according to the contrast value.

(Method 2) An image processing method including a frequency space transformation step of transforming a first real space image and a second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image; a synthesis rate determination step of determining a synthesis rate of a plurality of frequency components contained in the first frequency image and the second frequency image; a synthesis step of synthesizing the first frequency image and the second frequency image based on the synthesis rate of the plurality of frequency components to obtain a third frequency image; and a luminance evaluation value calculation step of calculating a luminance evaluation value indicating a luminance level of an image by a calculation including at least one of a process of calculating a minimum pixel value of the first real space image or a process of determining pixels whose pixel values of the first real space image are equal to or less than a threshold value, wherein the synthesis rate determination step determines the synthesis rate of the plurality of frequency components according to the luminance evaluation value.

(Method 3) An image processing method including a frequency space transformation step of transforming a first real space image and a second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image, a synthesis rate determination step of determining a synthesis rate of a plurality of frequency components contained in the first frequency image and the second frequency image, a synthesis step of synthesizing the first frequency image and the second frequency image based on the synthesis rate of the plurality of frequency components to obtain a third frequency image, and a difference calculation step of calculating a maximum value of the difference between corresponding pixels of the first real space image and the second real space image, wherein the synthesis rate determination step determines the synthesis rate of the plurality of frequency components according to the maximum value of the difference.

(Program) A program for controlling each means of the image processing device described in any one of configurations 1 to 22 by a computer.

In order to realize part or all of the control in this embodiment, a computer program that realizes the functions of the above-mentioned embodiment may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium on which the program is stored constitute the present invention.

100: Imaging device 101: Control unit 102: ROM
103: RAM
104: Optical system 105: Imaging unit 106: A/D conversion unit 107: Image processing unit 108: Recording unit 109: Communication unit 110: Display unit 111: Instruction input unit 201a: Small region extraction unit 201b: Small region extraction unit 202: Combining unit 203: Small region arrangement unit 301a: DCT unit 301b: DCT unit 302: First combination rate calculation unit 303: Contrast calculation unit 304: First combination rate correction unit 305: Weighted addition unit 306: Inverse DCT unit 801: Second combination rate calculation unit 1101: Dark area evaluation value calculation unit 1102: Second combination rate correction unit 1201: Maximum difference calculation unit 1202: Third combination rate correction unit

Claims (26)

a frequency space transforming means for transforming the first real space image and the second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image;
a synthesis rate determination means for determining a synthesis rate of a plurality of frequency components included in the first frequency image and the second frequency image;
a synthesis means for synthesizing the first frequency image and the second frequency image based on a synthesis rate of the plurality of frequency components to obtain a third frequency image;
a contrast calculation means for calculating a contrast value of the first real space image,
The image processing apparatus according to claim 1, wherein the synthesis rate determining means determines a synthesis rate of the plurality of frequency components in accordance with the contrast value. The image processing device according to claim 1, characterized in that the synthesis rate determination means determines the synthesis rate of the multiple frequency components so that the synthesis rate between the multiple frequency components approaches a uniform value as the contrast value increases. The image processing device according to claim 1, characterized in that the synthesis rate determination means determines the synthesis rate of the multiple frequency components such that the synthesis rate of at least one of the multiple frequency components decreases as the contrast value increases. The image processing device according to claim 1, characterized in that the contrast calculation means calculates the contrast value by the difference between the maximum pixel value and the minimum pixel value of the first real space image. The method further comprises: extracting a small region image from the image;
2. The image processing device according to claim 1, wherein the small region extraction means extracts the first real-space image from a first overall image, and extracts the second real-space image from a second overall image different from the first overall image. The image processing device according to claim 1, further comprising a real space conversion means for converting the third frequency image into real space and generating a third real space image. A small area arrangement means for arranging the plurality of small area images to generate an entire image,
7. The image processing apparatus according to claim 6, wherein said small region arrangement means arranges a plurality of said third real space images to generate a third whole image. a frequency space transforming means for transforming the first real space image and the second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image;
a synthesis rate determination means for determining a synthesis rate of a plurality of frequency components included in the first frequency image and the second frequency image;
a synthesis means for synthesizing the first frequency image and the second frequency image based on a synthesis rate of the plurality of frequency components to obtain a third frequency image;
a luminance evaluation value calculation means for calculating a luminance evaluation value indicating a luminance level of an image by at least one of a process of calculating a minimum pixel value of the first real space image and a process of determining a pixel whose pixel value of the first real space image is equal to or less than a threshold value;
The image processing device according to claim 1, wherein the synthesis rate determination means determines a synthesis rate of the plurality of frequency components in accordance with the luminance evaluation value. The image processing device according to claim 8, characterized in that the synthesis rate determination means determines the synthesis rate of the plurality of frequency components so that the synthesis rate between the plurality of frequency components approaches uniformity as the luminance evaluation value indicates a darker luminance level. The image processing device according to claim 8, characterized in that the synthesis rate determination means determines the synthesis rate of the plurality of frequency components such that the synthesis rate of at least one of the plurality of frequency components approaches 0 as the luminance evaluation value indicates a darker luminance level. The image processing device according to claim 8, characterized in that the luminance evaluation value calculation means calculates the number of pixels in the first real space image whose pixel values are equal to or less than the threshold value as the luminance evaluation value. The image processing device according to claim 8, characterized in that the luminance evaluation value calculation means calculates weights according to pixel values for one or more pixels of the first real space image whose pixel values are equal to or less than the threshold value, and calculates the luminance evaluation value by summing up the weights for the one or more pixels. The image processing device according to claim 8, characterized in that the luminance evaluation value calculation means calculates the minimum pixel value of the first real space image as the luminance evaluation value. The method further comprises: extracting a small region image from the image;
9. The image processing device according to claim 8, wherein the small region extraction means extracts the first real-space image from a first overall image, and extracts the second real-space image from a second overall image different from the first overall image. The image processing device according to claim 8, further comprising a real space conversion means for converting the third frequency image into real space and generating a third real space image. A small area arrangement means for arranging the plurality of small area images to generate an entire image,
16. The image processing apparatus according to claim 15, wherein said small region arrangement means arranges a plurality of said third real space images to generate a third whole image. a frequency space transforming means for transforming the first real space image and the second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image;
a synthesis rate determination means for determining a synthesis rate of a plurality of frequency components included in the first frequency image and the second frequency image;
a synthesis means for synthesizing the first frequency image and the second frequency image based on a synthesis rate of the plurality of frequency components to obtain a third frequency image;
a difference calculation means for calculating a maximum value of differences between corresponding pixels of the first real space image and the second real space image,
The image processing device according to claim 1, wherein the synthesis rate determination means determines a synthesis rate of the plurality of frequency components in accordance with a maximum value of the differences. The image processing device according to claim 17, characterized in that the synthesis rate determination means determines the synthesis rate of the multiple frequency components so that the synthesis rate between the multiple frequency components approaches a uniform value as the maximum value of the difference increases. The image processing device according to claim 17, characterized in that the synthesis rate determination means determines the synthesis rate of the plurality of frequency components such that the synthesis rate of at least one of the plurality of frequency components approaches 0 as the maximum value of the difference increases. The method further comprises: extracting a small region image from the image;
18. The image processing device according to claim 17, wherein the small region extraction means extracts the first real-space image from a first overall image, and extracts the second real-space image from a second overall image different from the first overall image. The image processing device according to claim 17, further comprising a real space conversion means for converting the third frequency image into real space and generating a third real space image. A small area arrangement means for arranging the plurality of small area images to generate an entire image,
22. The image processing apparatus according to claim 21, wherein said small region arrangement means arranges a plurality of said third real space images to generate a third whole image. a frequency space transformation step of transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively;
a synthesis rate determination step of determining a synthesis rate of a plurality of frequency components included in the first frequency image and the second frequency image;
a synthesis step of synthesizing the first frequency image and the second frequency image based on a synthesis rate of the plurality of frequency components to obtain a third frequency image;
a contrast calculation step of calculating a contrast value of the first real space image,
The image processing method according to claim 1, wherein the synthesis rate determination step determines a synthesis rate of the plurality of frequency components in accordance with the contrast value. a frequency space transformation step of transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively;
a synthesis rate determination step of determining a synthesis rate of a plurality of frequency components included in the first frequency image and the second frequency image;
a synthesis step of synthesizing the first frequency image and the second frequency image based on a synthesis rate of the plurality of frequency components to obtain a third frequency image;
a luminance evaluation value calculation step of calculating a luminance evaluation value indicating a luminance level of an image by a calculation including at least one of a process of calculating a minimum pixel value of the first real space image and a process of determining a pixel whose pixel value of the first real space image is equal to or less than a threshold value;
The image processing method according to claim 1, wherein the synthesis rate determination step determines a synthesis rate of the plurality of frequency components in accordance with the luminance evaluation value. a frequency space transformation step of transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively;
a synthesis rate determination step of determining a synthesis rate of a plurality of frequency components included in the first frequency image and the second frequency image;
a synthesis step of synthesizing the first frequency image and the second frequency image based on a synthesis rate of the plurality of frequency components to obtain a third frequency image;
a difference calculation step of calculating a maximum value of differences between corresponding pixels of the first real space image and the second real space image,
The image processing method according to claim 1, wherein the synthesis rate determination step determines a synthesis rate of the plurality of frequency components in accordance with a maximum value of the differences. A program for controlling each means of the image processing device according to any one of claims 1 to 22 by a computer.