JP2024049798A - Image processing apparatus, image processing method, and computer program - Google Patents

Abstract

To provide an image processing apparatus for synthesizing a plurality of images while properly suppressing image quality adverse effect.SOLUTION: An image processing apparatus includes: frequency space transformation means which transforms a first real-space image and a second real-space image to frequency spaces, respectively, to generate a first frequency image and a second frequency image; synthesis means which synthesizes the first frequency image with the second frequency image based on frequency components, to generate a third frequency image; real space transformation means which transforms the third frequency image into a real space to obtain a third real-space image; saturated pixel determination means which determines a saturated pixel in the first real-space image; and correction means which corrects a pixel value of a corresponding pixel, which is a pixel of the third real-space image located in coordinates corresponding to the saturated pixel, so as to be close to a pixel value of the saturated pixel.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing device, an image processing method, a computer program, etc.

In imaging devices such as digital cameras, ensuring a sufficient exposure time is effective in obtaining images with little noise. However, if the exposure time is extended, there is a problem that the image will become blurred due to camera movement caused by camera shake or movement of the subject.

An electronic blur correction method has been proposed as a method for dealing with this type of blur. With electronic blur correction, multiple images are taken in succession with short exposure times that cause less blur, and the resulting images are then synthesized to obtain a low-noise image. In addition, as shown in Patent Document 1, by lowering the synthesis rate in areas where there is a large difference between images, it is possible to prevent the subject from appearing multiple times in the synthesis result. This makes it possible to obtain an image with less blur.

Also, as shown in Non-Patent Document 1, a method is known in which the difference between images is calculated for each frequency component, and the synthesis rate is lowered for frequency components with large differences, thereby obtaining a low-noise image while maintaining the subject in good condition.

JP 2008-99260 A

Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, Marc Levoy, "Burst photography for high dynamic range and low-light imaging on mobile cameras", ACM Transactions on Graphics, Volume 35, Issue 6, November 2016, Article No. 192, pp. 1-12

However, when combining frequency components with different combining rates, the influence of a particular frequency component in the combined result can become relatively strong or weak, resulting in wavy patterns or ringing that are not present in the original subject. In particular, in areas where the pixel values in the image are saturated, this effect can cause the saturation state to be released, disrupting the state in which the pixel values are constant and resulting in a noticeable impairment of image quality.

The present invention aims to provide an image processing device that can effectively suppress image quality degradation and synthesize multiple images.

The image processing device of the present invention comprises:
a frequency space transforming means for transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively;
a synthesis means for synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image;
a real space conversion means for converting the third frequency image into a real space to obtain a third real space image;
a saturated pixel determination means for determining saturated pixels of the first real space image;
and a correcting means for correcting a pixel value of a corresponding pixel, which is a pixel of the third real space image at a coordinate corresponding to the saturated pixel, so as to approach the pixel value of the saturated pixel.

The present invention makes it possible to realize an image processing device that can effectively suppress image quality degradation and synthesize multiple images.

1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment of the present invention; 2 is a functional block diagram showing an example of a configuration for performing synthesis processing of image signals of a plurality of frames in an image processing unit 107 according to the first embodiment of the present invention. FIG. 3 is a functional block diagram showing an example of the configuration of a synthesis unit 202 in FIG. 2 . 2A to 2C are diagrams showing examples of a plurality of small regions extracted by a small region extraction unit 201a; 13 is a diagram showing an example of a table indicating the correspondence relationship between a difference value D and a synthesis rate SIG_RATIO. FIG. FIG. 11 is a functional block diagram showing an example of a basic configuration of an image processing unit 107 according to a second embodiment. FIG. 6 is a functional block diagram showing an example of the configuration of a correction unit 602.

The following describes the embodiment of the present invention using examples with reference to the drawings. However, the present invention is not limited to the following examples. In each drawing, the same members or elements are given the same reference numbers, and duplicate descriptions are omitted or simplified.

Example 1
1 is a block diagram showing a configuration example of an image capture device 100 as an image processing device according to a first embodiment of the present invention. The configuration example of the first embodiment will be described below with reference to FIG.

In the first embodiment, the imaging device 100 is shown as an example of an image processing device, but the image processing device is not limited to this. For example, the image processing according to the present invention may be performed by an image processing device such as a personal computer or an application server.

The control unit 101 is, for example, a CPU, and reads out a control program for each block of the imaging device 100 from a ROM 102 (described later), and deploys it in a RAM 103 (described later) for execution. In this way, the control unit 101 controls the operation of each block of the imaging device 100.

The ROM 102 is a non-volatile memory that can be electrically erased and recorded, and stores the operating programs of each block of the imaging device 100 as well as parameters required for the operation of each block.

RAM 103 is a rewritable volatile memory, and is used for expanding programs executed by the control unit 101, etc., and for temporarily storing data generated by the operation of each block of the imaging device 100, etc.

The optical system 104 is composed of a group of lenses including a zoom lens and a focus lens, and forms an image of a subject on the imaging surface of the imaging unit 105 described below. The imaging unit 105 is an imaging element such as a CCD sensor or a CMOS sensor, and photoelectrically converts the optical image formed on the imaging surface of the imaging unit 105 by the optical system 104, and outputs the obtained analog image signal to the A/D conversion unit 106.

The A/D conversion unit 106 converts the input analog image signal into digital image data. The digital image data output from the A/D conversion unit 106 is temporarily stored in the RAM 103.

The image processing unit 107 performs various types of image processing on the image data stored in the RAM 103. Specifically, it applies various types of image processing, such as demosaicing processing, white balance correction processing, gamma processing, and tone mapping processing, to develop, display, and record digital image data. It also applies various types of image processing to improve the image quality of the digital image data, such as noise suppression processing using spatial filters and synthesis of multiple pieces of image data.

The recording unit 108 records data including image data on a built-in recording medium. The communication unit 109 wirelessly connects to an external device and communicates image data and the like. The display unit 110 includes a display device such as an LCD, and displays images stored in the RAM 103 and images recorded in the recording unit 108 on the display device. The display unit 110 also displays an operation user interface for receiving instructions from the user.

The instruction input unit 111 is an input interface that includes various physical operating components such as a touch panel and a shutter button, and accepts instructions input by the user.

Fig. 2 is a functional block diagram showing an example of the configuration for performing synthesis processing of image signals of multiple frames in the image processing unit 107 in the first embodiment of the present invention. Also, Fig. 3 is a functional block diagram showing an example of the configuration of the synthesis unit 202 in Fig. 2. Below, an example of the configuration for image signal processing in the first embodiment of the present invention will be described with reference to Figs. 2 and 3.

Some of the functional blocks shown in Figures 2 and 3 are realized by having a CPU, which serves as a computer included in the control unit of the imaging device 100, execute a computer program stored in a ROM 102, which serves as a storage medium. However, some or all of these may be realized by hardware. As the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP), etc., may be used.

Furthermore, the functional blocks shown in Figures 2 and 3 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths. The above explanation regarding Figures 2 and 3 also applies to Figures 6 and 7.

Block 200 in FIG. 2 that performs image signal synthesis processing receives image signal SIG_B, image signal SIG_R, and small area coordinate data as input, and outputs image signal SIG_O obtained by synthesizing SIG_B and SIG_R. For the sake of explanation, in Example 1, it is assumed that a total of two frames of images, SIG_B and SIG_R, are synthesized. Also, SIG_B (first overall image) and SIG_R (second overall image) are image signals obtained by continuous shooting, and are assumed to have similar subjects.

The small region extraction unit 201a extracts a small region image signal SIG_B_P (first real space image) from the image signal SIG_B by referring to the small region coordinate data. Here, the small region coordinate data and the specific operation of the small region extraction unit 201a will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of multiple small regions extracted by the small region extraction unit 201a.

The multiple rectangles shown in FIG. 4 are examples of multiple small regions extracted by the small region extraction unit 201a. The multiple small regions are arranged in a row in both the horizontal and vertical directions, and FIG. 4 shows only a portion of the multiple small regions. The multiple small region image signals SIG_B_P are extracted in a state where the image signal SIG_B is divided into blocks of rectangles of a predetermined size, as shown in FIG. 4. A rectangle of a predetermined size is, for example, a rectangle of 16 pixels horizontally and 16 pixels vertically.

The small area coordinate data here is data indicating the coordinates of the upper left corner of the above-mentioned small area. The small area coordinate data is calculated for each small area in sequence, for example, by a clock circuit so as to obtain the above-mentioned coordinates, and is input in sequence to the small area extraction unit 201a. The small area extraction unit 201a takes the coordinates indicated in the small area coordinate data as the upper left corner of the small area, and sequentially extracts small areas of the above-mentioned predetermined size.

This allows the small region extraction unit 201a to sequentially extract multiple small region image signals SIG_B_P (first real space image) from the image signal SIG_B (first overall image) in the positional relationship shown in FIG. 4. This concludes the explanation of the small region coordinate data and the specific operation of the small region extraction unit 201a.

From here on, we return to the explanation of FIG. 2. Small region extraction unit 201b operates in a similar manner to small region extraction unit 201a, and extracts a small region image signal SIG_R_P (second real space image) from image signal SIG_R (second overall image) by referring to the small region coordinate data. Here, small region extraction unit 201a and small region extraction unit 201b function as small region extraction means that extract a small region image from an image, and each extracts a first real space image from a first overall image and a second real space image from a second overall image that is different from the first overall image.

The synthesis unit 202 synthesizes the small area image signal SIG_B_P (first real space image) and the small area image signal SIG_R_P (second real space image) to generate the small area image signal SIG_M_P. Note that the synthesis unit 202 receives the small area image signal SIG_R_P of the same coordinates that correspond to the small area image signal SIG_B_P of each coordinate indicated by the small area coordinate data.

For example, a small area image signal SIG_R_P having the same coordinates as the small area image signal SIG_B_P can be input by controlling a delay circuit. The specific operation of the synthesis unit 202 will be described with reference to FIG. 3.

Figure 3 shows an example of the configuration of the synthesis unit 202, and the operation of each block will be described. The DCT unit 301a performs discrete cosine transform (frequency transform) on the small-area image signal SIG_B_P (first real space image) and outputs a frequency space signal SIG_B_F (first frequency image). The discrete cosine transform is performed on both the horizontal and vertical directions of the small-area image signal SIG_B_P.

As a result, if the small region image signal SIG_B_P is a rectangle with a size of 16 pixels horizontally and 16 pixels vertically as described above, the frequency space signal SIG_B_F will have, for example, 16 frequency components horizontally and 16 frequency components vertically. Furthermore, by combining the horizontal and vertical components, the frequency space signal SIG_B_F will have a total of 256 frequency components.

The DCT unit 301b performs a discrete cosine transform (frequency transform) on the small region image signal SIG_R_P (second real space image) by the same operation as the DCT unit 301a, and outputs a frequency space signal SIG_R_F (second frequency image). Here, the DCT unit 301a and the DCT unit 301b function as a frequency space transforming means that executes a frequency space transforming step that transforms the first real space image and the second real space image into frequency space, respectively, to generate a first frequency image and a second frequency image.

The synthesis rate calculation unit 302 calculates and outputs the synthesis rate SIG_RATIO for each frequency component based on the frequency space signal SIG_B_F and the frequency space signal SIG_R_F.

Specifically, the combination ratio calculation unit 302 first calculates the difference value D between the frequency space signal SIG_B_F and the frequency space signal SIG_R_F for each frequency component. Next, the combination ratio calculation unit 302 refers to a table showing the correspondence between the difference value D and the combination ratio, and calculates the combination ratio SIG_RATIO for each frequency component so that the combination ratio is large when the difference value D is small and the combination ratio is small when the difference value D is large.

Figure 5 is a diagram showing an example of a table showing the correspondence between the difference value D and the synthesis rate SIG_RATIO. The horizontal axis of the graph in Figure 5 shows the difference value D, and the vertical axis shows the synthesis rate SIG_RATIO. Note that the synthesis rate SIG_RATIO is shown as a value range of 0 to 1.

The weighted addition unit 303 synthesizes the frequency space signal SIG_B_F (first frequency image) and the frequency space signal SIG_R_F (second frequency image) based on the synthesis rate SIG_RATIO for each frequency component to generate a frequency space signal SIG_M_F (third frequency image). Here, the weighted addition unit 303 functions as a synthesis means that executes a synthesis step of synthesizing the first frequency image and the second frequency image with a synthesis rate that differs between at least two frequency components based on the frequency components to generate a third frequency image.

Specifically, the weighted addition unit 303 performs the calculation shown in the following formula 1 for each frequency component to generate the frequency space signal SIG_M_F.

SIG_M_F = {(2 - SIG_RATIO) x SIG_B_F + SIG_RATIO x SIG_R_F} ÷ 2 ... (Formula 1)

According to Equation 1, when the synthesis rate SIG_RATIO is the maximum value of 1, the frequency space signal SIG_B_F and the frequency space signal SIG_R_F are synthesized at a ratio of 50% each. When the synthesis rate SIG_RATIO is the minimum value of 0, the frequency space signal SIG_B_F is synthesized at a ratio of 100% and the frequency space signal SIG_R_F is synthesized at a ratio of 0%.

The inverse DCT unit 304 performs an inverse discrete cosine transform on the frequency space signal SIG_M_F (third frequency image) and outputs a small region image signal SIG_M_P (third real space image) in real space. The inverse discrete cosine transform is performed on the horizontal and vertical components of the frequency space signal SIG_M_F. Here, the inverse DCT unit 304 functions as a real space transforming means that executes a real space transforming step that transforms the third frequency image into real space and obtains a third real space image.

The above is a specific operation of the synthesis unit 202. As a result, the synthesis unit 202 can synthesize the small area image signal SIG_B_P and the small area image signal SIG_R_P using different synthesis rates between frequency components. Furthermore, according to Equation 1, synthesis can be performed using the subject of the small area image signal SIG_B_P as a reference. From here on, we will return to the explanation of Figure 2.

The resaturation unit 203 corrects the desaturation of the small-area image signal SIG_M_P using the small-area image signal SIG_B_P, and outputs the corrected small-area image signal SIG_S_P. Specifically, the resaturation unit 203 first determines whether the pixel value of each pixel of the small-area image signal SIG_B_P (first real-space image) is equal to or greater than a predetermined threshold value. Here, the resaturation unit 203 functions as a saturated pixel determination means that executes a saturated pixel determination step that determines saturated pixels of the first real-space image.

The predetermined threshold value is, for example, the maximum pixel value at which the pixel value is saturated, or a pixel value close to the maximum pixel value considered to be saturated. Next, the resaturation unit 203 switches for each pixel between outputting the pixel value of the small region image signal SIG_M_P or the pixel value of the small region image signal SIG_B_P depending on whether the pixel value is equal to or greater than the predetermined threshold value.

When the pixel value of the small area image signal SIG_B_P is equal to or greater than a predetermined threshold value, the resaturation unit 203 selects and outputs the pixel value of the small area image signal SIG_B_P at the coordinates of the pixel in question as the output small area image signal SIG_S_P (third real space image). In other words, the resaturation unit 203 functions as a correction means that executes a correction step to correct the pixel value of the corresponding pixel, which is the pixel of the third real space image at the coordinates corresponding to the saturated pixel, to approach the pixel value of the saturated pixel by replacing it with the pixel value of the saturated pixel.

In addition, if the pixel value of the small area image signal SIG_B_P is not equal to or greater than the predetermined threshold, the resaturation unit 203 selects and outputs the pixel value of the small area image signal SIG_M_P at the coordinates of that pixel as the output small area image signal SIG_S_P.

This allows the re-saturation unit 203 to replace pixel values of the small-area image signal SIG_M_P that may have been desaturated from the small-area image signal SIG_B_P with pixel values of the small-area image signal SIG_B_P, which is the signal before the synthesis process, and output the replaced pixel values. Furthermore, for other pixels, the pixel values of the small-area image signal SIG_M_P can be output as is.

The small area placement unit 204 places a small area image signal SIG_S_P (third real space image) based on the small area coordinate data, and outputs SIG_O (third overall image) consisting of the third real space image. Here, the small area placement unit 204 functions as a small area placement means that places multiple third real space images and generates a third overall image.

Specifically, the small region placement unit 204 replaces the region whose upper left corner is the coordinate indicated by the small region coordinate data in the image signal SIG_O with the small region image signal SIG_S_P, and updates and outputs the image signal SIG_O.

By repeating this, the small area image signals SIG_S_P can be sequentially arranged so as to obtain the positional relationship described in FIG. 4. It is assumed that the small area arrangement unit 204 receives the small area image signals SIG_S_P corresponding to the small area image signals SIG_B_P at each coordinate indicated by the small area coordinate data. This allows the small area image signals SIG_S_P to be arranged in the same area as when the small area extraction unit 201a extracted the small area image signal SIG_B_P from the image signal SIG_B.

The block in FIG. 2 that performs the synthesis process of the image signal can generate the final image signal SIG_O by performing processes on all the small area coordinate data from the small area extraction unit 201a and the small area extraction unit 201b to the small area placement unit 204. Note that all the small area coordinate data is, for example, small area coordinate data that indicates all the coordinates for extracting small areas from the entire area of the image signal SIG_B or the image signal SIG_R.

The above is an explanation of the blocks that perform the synthesis process of the image signal in FIG. 2. In this way, according to the first embodiment, the operation of the synthesis unit 202 may cause the image after synthesis to become desaturated, but the operation of the resaturation unit 203 can correct the desaturation.

In the first embodiment, an example was shown in which pixel values in the saturated region after synthesis were replaced with pixel values before synthesis for correction, but any method of correction may be used as long as it brings the pixel values in the saturated region after synthesis closer to the pixel values before synthesis.

For example, the pixel value of the saturated region after synthesis may be corrected to an interpolated value with the pixel value of the saturated region before synthesis. In this case, when the pixel value of the small region image signal SIG_B_P is equal to or greater than the above-mentioned predetermined threshold, the re-saturation unit 203 outputs the pixel value calculated by the following formula 2 at the coordinates of the pixel.
W × SIG_B_P + (1-W) × SIG_M_P ... (Formula 2)

Here, W (0<W≦1) is the ratio by which the pixel value of the small region image signal SIG_M_P approaches the pixel value of the small region image signal SIG_B_P. In this way, even with the method of Equation 2, the re-saturation unit 203 can correct the release of the saturated state that may occur in the synthesized image due to the operation of the synthesis unit 202. That is, the pixel value of the corresponding pixel may be corrected based on an interpolated value calculated from the pixel value of the saturated pixel and the corresponding pixel value of the small region image signal SIG_M_P (third frequency image).

In addition, while Example 1 shows an example of processing an image signal having one channel per pixel, the present invention can also be applied to an image signal having multiple channels per pixel. Multiple channels are, for example, channels that indicate multiple colors such as RGB. In other words, the first real space image (small area image signal SIG_B_P) may be configured such that one pixel is composed of multiple channels (color channels).

In this case, the image processing unit 107 performs the compositing process shown in Figures 2 and 3 on each of the multiple channels in sequence, thereby performing compositing process and desaturation correction on the multiple channels. However, in this case, color distortion (change in hue) may occur because the desaturation correction is applied to only some of the multiple channels per pixel.

To prevent this, as another example, control may be performed so that either all of the multiple channels per pixel are corrected, or none are corrected. In this case, the image processing unit 107 first processes SIG_B and SIG_R, which have multiple channels per pixel, up to the synthesis unit 202, and obtains small area image signals SIG_B_P and SIG_M_P, which have multiple channels per pixel.

Then, for example, the resaturation unit 203 determines for each pixel of the small area image signal SIG_B_P whether one or more of the multiple channels per pixel are equal to or greater than a predetermined threshold value. That is, it determines that a pixel in the first real space image at coordinates corresponding to a pixel in which the pixel value of one or more color channels of the first real space image (small area image signal SIG_B_P) is equal to or greater than a threshold value is a saturated pixel. Furthermore, the resaturation unit 203 outputs a small area image signal SIG_S_P having multiple channels per pixel according to the determination result.

Specifically, if one or more of the multiple channels is equal to or greater than the aforementioned predetermined threshold, the resaturation unit 203 selects and outputs the pixel values of the small area image signal SIG_B_P in all of the multiple channels at the coordinates of the pixel in question as the small area image signal SIG_S_P. That is, one pixel of the small area image signal SIG_S_P (third real space image) is composed of the multiple channels, and the resaturation unit 203 makes the pixel values of the multiple channels of the corresponding pixel closer to the pixel values of the multiple channels of the saturated pixel.

In addition, if none of the multiple channels is equal to or greater than the aforementioned predetermined threshold, the resaturation unit 203 selects and outputs the pixel values of the small area image signal SIG_M_P in all of the multiple channels at the coordinates of the pixel in question as the small area image signal SIG_S_P. This allows the resaturation unit 203 to perform correction to remove the saturation state while suppressing color distortion.

In addition, in the first embodiment, the saturated area was determined based on the pixel values of the small-area image signal SIG_B_P, but in this case, the pixel values of the small-area image signal SIG_M_P may be replaced with the pixel values of pixels that are locally saturated by noise components in the small-area image signal SIG_B_P. If replaced with noise components, the noise reduction effect of synthesis in the small-area image signal SIG_S_P will be weakened. To prevent this, the saturated area may be determined based on the pixel values of an image generated based on the small-area image signal SIG_B_P.

For example, the re-saturation unit 203 may perform a smoothing process using a low-pass filter as a smoothing means on the small region image signal SIG_B_P (first real space image) to obtain a smoothed image, and then determine the saturated region based on the pixel values of the smoothed image.

That is, a pixel in the first real space image at coordinates corresponding to a pixel in the smoothed image whose pixel value is equal to or greater than a threshold value may be determined to be a saturated pixel. In this way, a pixel in the first real space image at coordinates corresponding to a pixel in an image (such as a smoothed image) in which one pixel is composed of multiple color channels and whose pixel value in one or more color channels is equal to or greater than a threshold value may be determined to be a saturated pixel.

This prevents pixel values in the small-region image signal SIG_M_P from being replaced with pixel values in the small-region image signal SIG_B_P that are locally saturated due to noise components.

Alternatively, the saturated region may be determined based on an image obtained by a process of shrinking the bright pixels of the small region image signal SIG_B_P in the spatial direction, or by a process of shrinking the bright pixels in the spatial direction and then expanding them. These methods can also achieve the same effect as the low-pass filter.

Even when processing a signal having multiple channels per pixel as described above, it is possible to prevent the pixel values of the small-area image signal SIG_M_P from being replaced with pixel values of pixels that are locally saturated due to noise components. In this case, the saturated area is determined based on the pixel values of an image having multiple channels generated based on each channel of the small-area image signal SIG_B_P having multiple channels.

In addition, in the first embodiment, an example was shown in which signals transformed into frequency space by discrete cosine transform were synthesized, but the present invention can be applied to any method for synthesizing frequency components. For example, signals transformed into frequency space by Fourier transform may be synthesized.

In addition, in the first embodiment, an example of synthesizing two frames, image signal SIG_B and image signal SIG_R, is shown, but the present invention can be applied regardless of the number of frames to be synthesized. For example, when synthesizing three or more frames, multiple small area image signals SIG_R_P may be extracted from multiple image signals SIG_R, and the synthesis unit 202 may synthesize the small area image signal SIG_B_P and the multiple small area image signals SIG_R_P.

In this case, the synthesis unit 202 calculates a synthesis ratio SIG_RATIO for each of the frequency space signals SIG_R_F of each frame. In addition, the weighted addition unit 303 performs weighted addition of the frequency space signal SIG_B_F and the frequency space signal SIG_R_F of each frame according to the synthesis ratio SIG_RATIO of each frame.

Example 2
In the first embodiment, an example is shown in which a process for correcting the release of a saturated state during image synthesis processing is performed. In the second embodiment, a process for correcting the release of a saturated state is performed during post-processing of a synthesized image. Unless otherwise specified, the same reference numerals as in the first embodiment perform the same operations and processes as in the first embodiment.

Fig. 6 is a functional block diagram showing an example of the basic configuration of the image processing unit 107 of the second embodiment. In Fig. 6, the image processing unit 107 includes a frequency space synthesis unit 601 and a correction unit 602. The frequency space synthesis unit 601 and the correction unit 602 each communicate image data with the RAM 103 via a bus.

The frequency space synthesis unit 601 synthesizes the image signals SIG_B and SIG_R with different synthesis rates between frequency components by the same process as that described in the first embodiment using FIG. 2, and outputs the image signal SIG_O. The correction unit 602 performs various corrections on the image signals. Note that in the second embodiment, the correction unit 602 performs tone mapping processing as the correction. The correction unit 602 will be described in detail later.

First, the operation of the control unit 101 in the second embodiment will be described. The control unit 101 can receive a user instruction to perform post-processing on the synthesized image signal SIG_O generated by the frequency space synthesis unit 601 via the instruction input unit 111. In the second embodiment, the post-processing includes a tone mapping process.

When the control unit 101 receives a user instruction to perform tone mapping processing on the synthesized image signal SIG_O, it reads the image signal SIG_O and the pre-synthesis image signal SIG_B corresponding to the image signal SIG_O into the RAM 103 via the recording unit 108. Note that the control unit 101 records the image signals SIG_O and SIG_B on the recording medium via the recording unit 108 after the synthesis process of the frequency space synthesis unit 601 is completed.

In addition, when the control unit 101 records the synthesized image signal SIG_O on the recording medium after the synthesis process of the frequency space synthesis unit 601 is completed, the control unit 101 adds additional information indicating the image signal SIG_B before synthesis to the image signal SIG_O and records it.

For example, the control unit 101 adds additional information to SIG_O in a format such as Exif. This allows the control unit 101 to refer to the additional information of the combined image signal SIG_O and read the pre-combination image signal SIG_B that corresponds to the combined image signal SIG_O.

Next, the control unit 101 controls the correction unit 602 to perform tone mapping of the image signal SIG_O using the image signal SIG_B as reference information. After that, the control unit 101 records the image signal SIG_S_T after tone mapping on a recording medium via the recording unit 108. This completes the operation of the control unit 101. Hereinafter, the operation of the correction unit 602 will be described with reference to FIG. 7.

Figure 7 is a functional block diagram showing an example of the configuration of the correction unit 602. Below, an example of the configuration of the tone mapping process of the image signal in the second embodiment will be described with reference to Figure 7.

The correction unit 602 in FIG. 7, which performs tone mapping processing of the image signal, receives the synthesized image signal SIG_O and the unsynthesized image signal SIG_B as input, and outputs an image signal SIG_S_T in which tone mapping processing has been performed on SIG_O with reference to SIG_B.

The tone mapping unit 701a performs tone mapping processing on the image signal SIG_O (third overall image) consisting of the third real space image with predetermined parameters, and outputs the image signal SIG_O_T (fifth overall image) after the tone mapping processing. The predetermined parameters are, for example, parameters specified by the user or parameters calculated by the image processing unit 107 according to the image signal SIG_O. Here, the tone mapping unit 701a functions as a second post-processing means that executes a second post-processing step in which the third overall image is subjected to tone mapping processing as post-processing to generate a fifth overall image.

As a result, new saturated regions may appear in the image signal SIG_O_T after tone mapping processing. In this case, since the image signal SIG_O is an image obtained by synthesizing frequency components at different synthesis rates, the pixel values of the new saturated regions of the image signal SIG_O_T may not be constant due to the effects of wavy patterns or ringing, which may be noticeable as a drawback to image quality.

The tone mapping unit 701b performs tone mapping processing on the image signal SIG_B (first overall image) consisting of the first real space image with the same parameters as the tone mapping unit 701a, and outputs the image signal SIG_B_T after the tone mapping processing. This makes it possible to obtain the image signal SIG_B_T (fourth overall image) in which a new saturated area has been generated in the same area as the image signal SIG_O_T.

Here, the tone mapping unit 701b functions as a first post-processing means that executes a first post-processing step of performing tone mapping processing as post-processing on the image signal SIG_B (first overall image) to generate a fourth overall image. In addition, the first post-processing means and the second post-processing means perform post-processing on the first overall image and the third overall image, respectively, using the same parameters.

The re-saturation unit 702 determines the saturated area of the image signal SIG_B_T (fourth overall image), corrects the saturated area of the image signal SIG_O_T (fifth overall image), and outputs the corrected image signal SIG_S_T.

Here, the resaturation unit 702 functions as saturated pixel determination means that executes a saturated pixel determination step that determines saturated pixels in the image signal SIG_B_T (fourth overall image). The resaturation unit 702 also functions as correction means, and executes a correction step that corrects the pixel value of a corresponding pixel, which is a pixel of the fifth overall image at a coordinate corresponding to a saturated pixel, so that the pixel value approaches the pixel value of the saturated pixel.

The operation of the resaturation unit 702 is the same as that of the resaturation unit 203 in the first embodiment. Specifically, the operation is the same as that in the resaturation unit 203, except that the small area image signal SIG_B_P is replaced with the image signal SIG_B_T, the small area image signal SIG_M_P is replaced with the image signal SIG_O_T, and the small area image signal SIG_S_P is replaced with the image signal SIG_S_T. This enables the resaturation unit 702 to suppress image quality problems in the newly saturated areas that occur in the image signal SIG_O_T.

The above is an explanation of the blocks that perform tone mapping processing of the image signal in Figure 7. Note that in the second embodiment, a configuration for suppressing image quality problems in new saturated areas that are generated by tone mapping processing is explained. However, by applying this embodiment to any post-processing that generates new saturated areas, it is possible to suppress image quality problems in new saturated areas.

The present invention has been described in detail above based on preferred embodiments thereof, but the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention includes the following combinations.

(Configuration 1) An image processing device comprising: a frequency space conversion means for converting a first real space image and a second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image; a synthesis means for synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image; a real space conversion means for converting the third frequency image into real space to obtain a third real space image; a saturated pixel determination means for determining saturated pixels in the first real space image; and a correction means for correcting a pixel value of a corresponding pixel, which is a pixel of the third real space image at a coordinate corresponding to the saturated pixel, to approach the pixel value of the saturated pixel.

(Configuration 2) The image processing device described in Configuration 1, characterized in that the synthesis means synthesizes the first frequency image and the second frequency image with a synthesis rate that differs between at least two frequency components.

(Configuration 3) The image processing device according to configuration 1 or 2, characterized in that the correction means replaces the pixel value of the corresponding pixel with the pixel value of the saturated pixel.

(Configuration 4) The image processing device according to any one of configurations 1 to 3, characterized in that the correction means corrects the pixel value of the corresponding pixel based on an interpolated value calculated from the pixel value of the saturated pixel and the pixel value of the third frequency image.

(Configuration 5) The image processing device according to any one of configurations 1 to 4, further comprising a smoothing means for performing a smoothing process on the first real-space image to generate a smoothed image, and the saturated pixel determination means determines, as the saturated pixel, a pixel of the first real-space image at a coordinate corresponding to a pixel of the smoothed image whose pixel value is equal to or greater than a threshold value.

(Configuration 6) The image processing device according to any one of configurations 1 to 5, characterized in that one pixel of the first real space image is composed of multiple color channels.

(Configuration 7) The saturated pixel determination means
The image processing device according to configuration 6, characterized in that a pixel of the first real space image at a coordinate corresponding to a pixel whose pixel value of one or more color channels of the first real space image is equal to or greater than a threshold value is determined to be the saturated pixel.

(Configuration 8) The saturated pixel determination means
The image processing device according to configuration 6 or 7, characterized in that a pixel of the first real space image at a coordinate corresponding to a pixel whose pixel value of one or more color channels is equal to or greater than a threshold value in an image in which one pixel is composed of a plurality of color channels and which is generated based on the first real space image, is determined to be the saturated pixel.

(Configuration 9) The third real space image has one pixel constituted by the plurality of channels,
9. The image processing device according to any one of configurations 6 to 8, wherein the correction means brings pixel values of the plurality of channels of the corresponding pixel closer to pixel values of the plurality of channels of the saturated pixel.

(Configuration 10) The image processing method further includes a small area extraction means for extracting a small area image from the image,
The image processing device according to any one of configurations 1 to 9, wherein the small region extraction means extracts the first real-space image from a first overall image, and extracts the second real-space image from a second overall image different from the first overall image.

(Configuration 11) The image processing device according to any one of configurations 1 to 10, further comprising a small area arrangement means for arranging a plurality of the third real space images to generate a third overall image.

(Configuration 12) An image processing device comprising: a frequency space conversion means for converting a first real space image and a second real space image into a frequency space, respectively, to generate a first frequency image and a second frequency image; a synthesis means for synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image; a real space conversion means for converting the third frequency image into real space to obtain a third real space image; a first post-processing means for performing post-processing on a first overall image consisting of the first real space image to generate a fourth overall image; a second post-processing means for performing post-processing on a third overall image consisting of the third real space image to generate a fifth overall image; a saturated pixel determination means for determining saturated pixels in the fourth overall image; and a correction means for correcting a pixel value of a corresponding pixel, which is a pixel of the fifth overall image at a coordinate corresponding to the saturated pixel, to approach the pixel value of the saturated pixel.

(Configuration 13) The image processing device described in Configuration 12, characterized in that the first post-processing means and the second post-processing means perform post-processing on the first overall image and the third overall image, respectively, using the same parameters.

(Configuration 14) The image processing device according to configuration 12 or 13, characterized in that the post-processing includes tone mapping processing.

(Method 1) An image processing method comprising: a frequency space transformation step of transforming a first real space image and a second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively; a synthesis step of synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image; a real space transformation step of transforming the third frequency image into real space to obtain a third real space image; a saturated pixel determination step of determining saturated pixels in the first real space image; and a correction step of correcting the pixel value of a corresponding pixel, which is a pixel of the third real space image at a coordinate corresponding to the saturated pixel, to approach the pixel value of the saturated pixel.

(Method 2) An image processing method comprising: a frequency space transformation step of transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively; a synthesis step of synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image; a real space transformation step of transforming the third frequency image into real space to obtain a third real space image; a first post-processing step of performing post-processing on a first overall image consisting of the first real space image to generate a fourth overall image; a second post-processing step of performing post-processing on a third overall image consisting of the third real space image to generate a fifth overall image; a saturated pixel determination step of determining saturated pixels in the fourth overall image; and a correction step of correcting the pixel value of a corresponding pixel, which is a pixel of the fifth overall image at a coordinate corresponding to the saturated pixel, to approach the pixel value of the saturated pixel.

(Program) A computer program for controlling each means of an image processing device described in any one of configurations 1 to 14 by a computer.

In order to realize some or all of the control in this embodiment, a computer program that realizes the functions of the above-mentioned embodiment may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium on which the program is stored constitute the present invention.

100: Imaging device 101: Control unit 102: ROM
103: RAM
104: Optical system 105: Imaging unit 106: A/D conversion unit 107: Image processing unit 108: Recording unit 109: Communication unit 110: Display unit 111: Instruction input unit 201a: Small region extraction unit 201b: Small region extraction unit 202: Synthesis unit 203: Resaturation unit 204: Small region arrangement unit 301a: DCT unit 301b: DCT unit 302: Synthesis rate calculation unit 303: Weighted addition unit 304: Inverse DCT unit 601: Frequency space synthesis unit 602: Correction unit 701a: Tone mapping unit 701b: Tone mapping unit 702: Resaturation unit

Claims (17)

a frequency space transforming means for transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively;
a synthesis means for synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image;
a real space conversion means for converting the third frequency image into a real space to obtain a third real space image;
a saturated pixel determination means for determining saturated pixels of the first real space image;
a correction unit that corrects a pixel value of a corresponding pixel, which is a pixel of the third real space image at a coordinate corresponding to the saturated pixel, so as to approach the pixel value of the saturated pixel. The image processing device according to claim 1, characterized in that the synthesis means synthesizes the first frequency image and the second frequency image with a synthesis rate that differs between at least two frequency components. The image processing device according to claim 1, characterized in that the correction means replaces the pixel value of the corresponding pixel with the pixel value of the saturated pixel. The image processing device according to claim 1, characterized in that the correction means corrects the pixel value of the corresponding pixel based on an interpolated value calculated from the pixel value of the saturated pixel and the pixel value of the third frequency image. a smoothing unit that performs a smoothing process on the first real space image to generate a smoothed image;
2. The image processing apparatus according to claim 1, wherein the saturated pixel determining means determines, as the saturated pixel, a pixel of the first real space image at a coordinate corresponding to a pixel of the smoothed image whose pixel value is equal to or greater than a threshold value. The image processing device according to claim 1, characterized in that one pixel of the first real space image is composed of multiple color channels. The saturated pixel determination means
The image processing apparatus according to claim 6 , further comprising: determining, as the saturated pixel, a pixel of the first real space image at a coordinate corresponding to a pixel whose pixel value in one or more color channels of the first real space image is equal to or greater than a threshold value. The saturated pixel determination means
7. The image processing device according to claim 6, wherein a pixel of the first real space image at a coordinate corresponding to a pixel whose pixel value of one or more color channels is equal to or greater than a threshold value in an image generated based on the first real space image and in which one pixel is composed of a plurality of color channels is determined to be the saturated pixel. the third real space image has one pixel constituted by the plurality of channels;
7. The image processing apparatus according to claim 6, wherein said correction means brings the pixel values of said plurality of channels of said corresponding pixel closer to the pixel values of said plurality of channels of said saturated pixel. The method further comprises: extracting a small region image from the image;
2. The image processing device according to claim 1, wherein the small region extraction means extracts the first real-space image from a first overall image, and extracts the second real-space image from a second overall image different from the first overall image. The image processing device according to claim 1, further comprising a small area arrangement means for arranging a plurality of the third real space images to generate a third overall image. a frequency space transforming means for transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively;
a synthesis means for synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image;
a real space conversion means for converting the third frequency image into a real space to obtain a third real space image;
a first post-processing means for performing post-processing on a first overall image formed from the first real space image to generate a fourth overall image;
a second post-processing means for performing the post-processing on a third overall image formed from the third real space image to generate a fifth overall image;
a saturated pixel determining means for determining saturated pixels of the fourth entire image;
a correction unit that corrects a pixel value of a corresponding pixel, which is a pixel of the fifth entire image at a coordinate corresponding to the saturated pixel, so as to approach the pixel value of the saturated pixel. The image processing device according to claim 12, characterized in that the first post-processing means and the second post-processing means perform post-processing on the first overall image and the third overall image, respectively, using the same parameters. The image processing device according to claim 12, characterized in that the post-processing includes a tone mapping process. a frequency space transformation step of transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively;
a synthesis step of synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image;
a real space transformation step of transforming the third frequency image into a real space to obtain a third real space image;
a saturated pixel determination step of determining saturated pixels of the first real space image;
a correction step of correcting a pixel value of a corresponding pixel, which is a pixel of the third real space image at a coordinate corresponding to the saturated pixel, so as to approach the pixel value of the saturated pixel. a frequency space transformation step of transforming the first real space image and the second real space image into a frequency space to generate a first frequency image and a second frequency image, respectively;
a synthesis step of synthesizing the first frequency image and the second frequency image based on frequency components to generate a third frequency image;
a real space transformation step of transforming the third frequency image into a real space to obtain a third real space image;
a first post-processing step of performing post-processing on a first overall image formed from the first real space image to generate a fourth overall image;
a second post-processing step of performing the post-processing on a third overall image formed from the third real space image to generate a fifth overall image;
a saturated pixel determination step of determining saturated pixels of the fourth entire image;
a correcting step of correcting a pixel value of a corresponding pixel, which is a pixel of the fifth entire image at a coordinate corresponding to the saturated pixel, so as to approach the pixel value of the saturated pixel. A computer program for controlling each unit of the image processing apparatus according to any one of claims 1 to 14 by a computer.

Images