JP2014039169A - Image processor, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce noise and to generate a highly dynamic range image and the like by synthesizing images where a motion compensation image by highly precise motion estimation is applied.SOLUTION: A moving amount between images is estimated, motion compensation processing where one image is position-adjusted to another image is performed, a motion compensation image is generated, and an image where noise is reduced by synthesis processing of the motion compensation image and the positioned image is generated. A motion compensation image generation section generates the motion compensation image using a selected image selected in accordance with a previously specified image selection standard. An image processor includes a motion estimation appropriateness determination section for determining whether the input image is an appropriate image suitable for motion estimation or not, for example. Only when the input image is the appropriate image, the input image is stored in a frame memory as a reference image applied for motion estimation.

Description

  The present disclosure relates to an image processing device, an image processing method, and a program. In particular, the present invention relates to an image processing apparatus, an image processing method, and a program that perform, for example, image noise reduction processing, dynamic range expansion processing, and the like by combining multiple images.

  For example, when image noise reduction processing (NR: Noise Reduction) is executed, composition processing using a plurality of images taken continuously is performed. That is, a process is performed in which corresponding pixels that are the shooting pixel regions of the same subject are detected from the continuously shot images, and the pixel values of the output image are calculated using the pixel values of the plurality of corresponding pixels.

Specifically, for example,
A preceding photographed image at photographing time t−1,
Subsequent shot image at shooting time t,
When performing noise reduction of the subsequent captured image using these two captured images, the following processing is executed.

  First, a process of combining the preceding photographed image at the photographing time t−1 with the position of the subsequent photographed image at the time t is performed. In many cases, the image position of two consecutively photographed images has moved due to camera shake or the like, and this is alignment for correcting this shift. This process is called motion compensation (or position compensation), and an image that has been moved for alignment is called a motion compensation (MC) image.

  In order to generate a motion-compensated image (MC image), first, motion estimation that estimates the amount and direction of motion between two images, a preceding captured image at the capturing time t−1 and a subsequent captured image at the capturing time t. I do. Based on this motion estimation information, a motion compensation process (MC) is performed in which the preceding photographed image at the photographing time t−1 is combined with the image position of the subsequent photographed image at the photographing time t.

  The motion compensated image (MC image) generated by motion compensation for the preceding photographed image at the photographing time t−1 is an image obtained by photographing the same subject at the corresponding pixel position of the subsequent photographed image at the photographing time t. A correction pixel value is calculated using two pixel values of corresponding pixels of these two images. For example, when noise is included in the pixels of the subsequent captured image at the capturing time t, the noise can be reduced by using the pixel value of the preceding captured image at the capturing time t−1.

  In addition, as a prior art disclosed about image processing techniques, such as noise reduction using a several image, there exist patent document 1 (Unexamined-Japanese-Patent No. 2009-194700) or Patent Document 2 (Unexamined-Japanese-Patent No. 2009-290827) etc. .

  Such an output image generation process based on a plurality of consecutively photographed images is called not only the noise reduction process described above but also a super-resolution process (SR: Super Resolution) that generates a high-resolution image from a low-resolution image. It is also used in high resolution image processing. Alternatively, it is also used in a dynamic range expanded image generation process using continuously photographed images with different exposure times.

  The sequence of image correction processing using continuously shot images will be described with reference to the drawings. FIG. 1 shows a processing sequence in the case of sequentially executing motion estimation processing, generation of a motion compensated image as image alignment, and signal processing such as noise reduction for a continuous input image such as a moving image. FIG.

Images taken along the time axis are sequentially input to an image processing device, for example, a signal processing unit of the imaging device. When the input image t-1 that is a captured image at time t-1 is input to the signal processing unit, the signal processing unit stores the stored image t-2 that is the previous captured image in the frame memory (step S01a). FM), and processing using the stored image t-2 and the input image t-1 is executed.
Specifically, signal processing such as motion estimation processing using these two images, motion compensation image generation processing based on the motion estimation result, and noise reduction processing using the motion compensation image is executed.
Note that in the motion compensated image generation process, for example, the image position of the stored image t-2 is moved to be aligned with the image position of the input image t-1.

Further, in step S01b, the input image t-1 is stored in the frame memory (FM).
In FIG. 1, step S01a is shown to the left of time t-1 on the time axis, but the processing in step S01a and the processing in step S01b are both executed after time t.

Next, when a new input image t is input to the signal processing unit, in step S02a, the signal processing unit acquires the stored image t-1 that is the previous captured image from the frame memory (FM). Then, processing using the stored image t-1 and the input image t is executed.
Specifically, signal processing such as motion estimation processing using these two images, motion compensation image generation processing based on the motion estimation result, and noise reduction processing using the motion compensation image is executed.
In step S02b, the input image t is stored in the frame memory (FM).

  The signal processing unit sequentially executes this series of processing, performs processing such as noise reduction on each image, and performs processing such as outputting the processing result or storing it in the storage unit.

However, when processing is performed by combining two consecutively shot images in this way, if there is an image that is not suitable for motion estimation, not only the processing accuracy for that image will deteriorate, but also the processing accuracy of the signal processing for the adjacent image This causes a problem of lowering.
For example, it is difficult to search for a corresponding area between the previous and next captured images for an image with many areas where the subject is out of focus, and correct motion estimation is not performed.

A problem in the case of inputting an inappropriate image that makes accurate motion estimation difficult will be described with reference to FIG.
In FIG. 2, for example, it is assumed that the input image t-1 is an inappropriate image that is not suitable for motion estimation.

In this case, the process of step S01a shown in FIG.
The accuracy of signal processing such as motion estimation using the frame memory (FM) stored image t-2 and the input image t-1, generation of a motion compensated image, and subsequent noise reduction is reduced.
Further, the process of step S02a shown in FIG.
The accuracy of signal processing such as motion estimation using the frame memory (FM) stored image t-1 and the input image t, generation of a motion compensated image, and subsequent noise reduction is reduced.
In this way, the processing for two consecutive images becomes inaccurate due to the presence of one unqualified image. For example, when noise reduction processing is performed as signal processing, two consecutive images in which noise is not reduced are generated by erroneous processing. Will occur.

As described above, as image signal processing to which a motion compensated image generated based on motion estimation between two images is applied, in addition to noise reduction processing (NR), for example, super-resolution processing (SR), high resolution processing, etc. There is a dynamic range image (HDR image) generation process.
Also in these various signal processes, the problem described with reference to FIG. 2 exists in common, and there is a problem that the quality of a plurality of output images is reduced by one inappropriate image.

  The high dynamic range image (HDR image) generation process is described in, for example, Japanese Patent Application Laid-Open No. 2008-160881.

JP 2009-194700 A JP 2009-290827 A JP 2008-160881 A

  The present disclosure has been made in view of, for example, the above-described problems. Even when an image for which accurate motion estimation is difficult is input, the effect is minimized and a higher-quality processed image is generated. It is an object of the present invention to provide an image processing apparatus, an image processing method, and a program that are made possible.

The first aspect of the present disclosure is:
A motion estimator that estimates the amount of motion between two images of a first image and a second image that are captured images at different timings;
A motion compensated image generating unit that generates a motion compensated image by executing a motion compensation process for aligning the second image with the position of the first image based on the motion amount estimated by the motion estimator;
A signal processing unit that generates a synthesized image by synthesizing the motion compensation image and the first image;
The motion compensated image generation unit is in an image processing apparatus that generates a motion compensated image using a selected image selected according to a predetermined image selection criterion.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the image processing device includes a motion estimation suitability determination unit that determines whether or not the input image is a qualified image suitable for motion estimation, and the motion The estimation suitability determination unit performs control to store the input image in a frame memory as a reference image to be applied to the motion estimation only when the input image is a qualified image, and the motion estimation unit The motion estimation using the reference image stored in the memory is executed, and the motion compensation image generation unit generates a motion compensation image using the reference image.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the motion estimation unit inputs determination information indicating whether or not the input image is a qualified image as determination information of the motion estimation suitability determination unit. When it is determined that the input image is a qualified image based on the determination information, motion estimation using the input image and the reference image is executed.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the motion estimation unit inputs determination information indicating whether or not the input image is a qualified image as determination information of the motion estimation suitability determination unit. If the input image is determined to be an ineligible image not suitable for motion estimation based on the determination information, the motion estimation unit does not perform motion estimation using the input image and the reference image. When the motion estimation is not executed in step (b), a motion amount correction unit is provided that outputs an exceptional motion amount determined according to a predetermined algorithm to the motion compensated image generation unit.

  Furthermore, in one embodiment of the image processing device of the present disclosure, the signal processing unit inputs determination information indicating whether or not the input image is a qualified image as determination information of the motion estimation appropriateness determination unit. Based on the determination information, signal processing is executed on the input image.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the motion estimation appropriateness determination unit is configured to capture at least one of a blurred area, a saturated pixel area, and a noisy area in the input image, or when an input image is captured. A process for determining whether or not the input image is an image suitable for motion estimation is performed by applying an index of either the camera movement or the luminance range of the input image.

  Furthermore, in one embodiment of the image processing device according to the present disclosure, when the continuous estimation image set at a different exposure time is input as the input image, the motion estimation appropriateness determination unit determines an effective pixel value of each exposure image. By comparing the areas, an image having a larger effective pixel area is controlled as a reference image stored in the frame memory.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the image processing device further uses the image stored in the frame memory as a reference image to match the image position of the input image with the image position of the reference image. A motion estimation direction switching unit that executes a motion estimation direction switching process.

Furthermore, in an embodiment of the image processing apparatus according to the present disclosure, the image processing apparatus further inputs, as the input image, continuous shot images set at different exposure times, and executes exposure ratio correction. The motion estimation unit performs motion estimation between continuously shot images that have been subjected to exposure ratio correction in the exposure ratio correction unit, calculates a motion amount between the continuously shot images, and The compensation image generation unit uses the amount of motion to generate a motion compensation image in which one image position of the continuously shot image is matched with the other image position,
The signal processing unit executes a synthesis process of the motion compensated image and a reference image stored in a frame memory.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the reference image stored in the frame memory is an image having a predetermined exposure time, and the exposure ratio correction unit includes the input image, the frame, The exposure ratio correction between the reference images stored in the memory is executed, and the motion estimation unit executes the motion estimation between the input image subjected to the exposure ratio correction and the reference image in the exposure ratio correction unit.

  Furthermore, in one embodiment of the image processing device of the present disclosure, the motion estimation unit calculates a motion vector between images set at the same exposure time, inputs the motion vector calculated by the motion estimation unit, A motion vector calculation unit that calculates a motion vector to be applied to generation of a motion compensated image in the motion compensated image generation unit by linear calculation on the motion vector.

Furthermore, the second aspect of the present disclosure is:
An image processing method executed in an image processing apparatus,
A motion estimation step in which a motion estimation unit estimates a motion amount between two images of a first image and a second image that are captured images at different timings;
A motion compensated image generating step for generating a motion compensated image by executing a motion compensation process for aligning the second image with the position of the first image based on the motion amount estimated in the motion estimating step. When,
A signal processing unit that performs a signal processing step of generating a synthesized image by synthesizing the motion compensated image and the first image;
The motion compensation image generation step is an image processing method for generating a motion compensation image using a selected image selected according to a predetermined image selection criterion.

Furthermore, the third aspect of the present disclosure is:
A program for executing image processing in an image processing apparatus;
A motion estimation step for causing the motion estimation unit to estimate a motion amount between the first image and the second image that are captured images at different timings;
A motion compensation image generation step for causing the motion compensation image generation unit to generate a motion compensation image by executing a motion compensation process for aligning the second image with the position of the first image based on the motion amount estimated in the motion estimation step. When,
Performing a signal processing step of causing a signal processing unit to generate a composite image by combining the motion compensated image and the first image;
In the motion compensation image generation step, a program for generating a motion compensation image using a selected image selected according to a predetermined image selection criterion is provided.

  Note that the program of the present disclosure is a program that can be provided by, for example, a storage medium or a communication medium provided in a computer-readable format to an information processing apparatus or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the information processing apparatus or the computer system.

  Other objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on embodiments of the present disclosure described below and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

According to the configuration of an embodiment of the present disclosure, a configuration for generating high-quality noise reduction, a high dynamic range image, and the like is realized by image synthesis processing using a motion compensated image based on highly accurate motion estimation.
Specifically, the amount of motion between the images is estimated, a motion compensation image is generated by matching one image with the position of the other image, and a motion compensated image is generated. An image subjected to noise reduction or the like is generated by the synthesis process. The motion compensated image generation unit generates a motion compensated image using a selected image selected according to a predetermined image selection criterion. For example, a motion estimation suitability determination unit that determines whether or not an input image is a qualified image suitable for motion estimation, and applies the input image to motion estimation only when the input image is a qualified image It is stored in the frame memory as a reference image.
With these processes, high-quality noise reduction and a high dynamic range image can be generated by image synthesis processing using a motion compensated image based on highly accurate motion estimation.

It is a figure explaining the processing sequence in the case of performing sequentially signal processing, such as a motion estimation process, the production | generation of a motion compensation image, and noise reduction, to continuous input images, such as a moving image. It is a figure explaining the processing sequence in the case of performing sequentially signal processing, such as a motion estimation process, the production | generation of a motion compensation image, and noise reduction, to continuous input images, such as a moving image. It is a figure explaining the processing sequence in the case of performing sequentially signal processing, such as a motion estimation process, the production | generation of a motion compensation image, noise reduction, etc. with respect to continuous input images, such as a moving image, according to the process of this indication. It is a figure explaining the noise reduction process structure as signal processing. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure explaining the process of the image processing apparatus according to one Example of this indication. It is a figure which shows the flowchart explaining the process sequence of the image processing apparatus according to one Example of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure which shows the flowchart explaining the process sequence of the image processing apparatus according to one Example of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure which shows the flowchart explaining the process sequence of the image processing apparatus according to one Example of this indication. It is a figure explaining the process of the image processing apparatus according to one Example of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure which shows the flowchart explaining the process sequence of the image processing apparatus according to one Example of this indication. It is a figure which shows the flowchart explaining the process sequence of the image processing apparatus according to one Example of this indication. It is a figure explaining the process of the image processing apparatus according to one Example of this indication. It is a figure explaining the production | generation process of a high dynamic range image. It is a figure explaining the structural example which performs a high dynamic range image generation process and a noise reduction process together. It is a figure explaining the structural example which performs a high dynamic range image generation process and a noise reduction process together. FIG. 4 is a diagram illustrating an example of an input image sequence in an embodiment of the image processing apparatus of the present disclosure. It is a figure explaining the image composition process performed in one Example of the image processing apparatus of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure explaining the process of the image processing apparatus according to one Example of this indication. It is a figure explaining the process of the image processing apparatus according to one Example of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure which shows the flowchart explaining the process sequence of the image processing apparatus according to one Example of this indication. It is a figure which shows the flowchart explaining the process sequence of the image processing apparatus according to one Example of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure explaining the process of the image processing apparatus according to one Example of this indication. It is a figure explaining composition and processing of an image processing device according to an example of this indication. It is a figure which shows the flowchart explaining the process sequence of the image processing apparatus according to one Example of this indication. It is a figure explaining the structural example of the image processing apparatus of this indication.

The details of the image processing apparatus, the image processing method, and the program of the present disclosure will be described below with reference to the drawings. The description will be made according to the following items.
1. 1. Example of processing for determining appropriateness of motion estimation for image 1-1. Outline of processing 1-2. An image processing apparatus that performs processing based on determination of suitability for motion estimation (Example 1)
1-3. An image processing apparatus that performs processing based on determination of propriety of motion estimation (second embodiment)
1-4. Image processing apparatus for performing processing based on determination of propriety of motion estimation (Example 3)
1-5. Details of motion estimation appropriateness determination processing 1-6. Example (Example 4) of performing noise reduction by superimposing continuous shot images
1-7. Example (Example 5) in which high dynamic range image generation processing is performed
2. 2. Example in which noise reduction by combining multiple images and high dynamic range processing are executed together 2-1. General configuration example and problem of executing both noise reduction processing and high dynamic range processing 2-2. Overview of processing to execute noise reduction and high dynamic range image generation according to the present disclosure 2-3. Example (Example 6) which performs noise reduction and high dynamic range image generation according to the present disclosure together
2-4. Example (Example 7) which performs noise reduction and high dynamic range image generation according to the present disclosure together
2-5. Example in which motion estimation is performed between images having the same exposure time (Example 8)
3. 3. Example of overall configuration of image processing apparatus Summary of composition of this disclosure

[1. Example of Executing Signal Processing that Performs Appropriate Motion Estimation for Each Image and Selects and Applies Images Suitable for Motion Estimation]
First, as a first embodiment of the image processing apparatus of the present disclosure, determination of suitability of motion estimation for an image is performed, motion estimation is performed by selecting and applying an image suitable for motion estimation, and the motion estimation result is used. An embodiment for executing signal processing such as noise reduction will be described.

[1-1. About processing overview]
FIG. 3 is a diagram showing a processing sequence in the present embodiment. Similar to FIGS. 1 and 2 described above, for example, motion estimation processing, generation of motion compensated images as image alignment, and signal processing such as noise reduction are performed on continuous input images such as moving images. It is a figure explaining the processing sequence in the case of performing sequentially.

Images taken along the time axis are sequentially input to an image processing device, for example, a signal processing unit of the imaging device. When the input image t-1 that is the captured image at time t-1 is input to the signal processing unit, the signal processing unit stores the stored image t-2 that is the previous captured image in the frame memory (step S11a). FM), and processing using the stored image t-2 and the input image t-1 is executed.
Specifically, signal processing such as motion estimation processing using these two images, motion compensation image generation processing based on the motion estimation result, and noise reduction processing using the motion compensation image is executed.

  Here, it is assumed that the input image t-1 is an image for which accurate motion estimation is difficult, for example, a “blurred image”. Hereinafter, such an image for which motion estimation is difficult is referred to as an unqualified image.

  The process in step S11a is executed using the input image t-1 that is the unqualified image. As a result, the processed image obtained as the processing result of step S11a is highly likely to be an image in which the intended processing effect cannot be obtained. For example, in the configuration in which the noise reduction process is performed, there is a high possibility that the image generated in step S11a will be an image for which sufficient noise reduction is not performed.

  In the processing example described above with reference to FIG. 2, the input image t-1 which is an unqualified image is stored in the frame memory and used in the next signal processing (step S02a in FIG. 2). . Therefore, the processed image in step S02a shown in FIG. 2 also becomes an image in which a target processing effect, for example, a noise reduction effect cannot be obtained.

However, in the embodiment shown in FIG. 3, when it is determined that the input image is an unsuitable image that is not suitable for motion estimation, as shown in step S11b of FIG. 3, the storage process for the frame memory (FM) is stopped. .
As a result, the previous image, that is, the stored image t-2 is continuously stored in the frame memory (FM). This stored image t-2 is not an ineligible image, but an image capable of good motion estimation.

In step S12a shown in FIG. 3, a process using a new input image t and a stored image t-2 stored in the frame memory (FM) is executed.
Specifically, signal processing such as motion estimation processing using these two images, motion compensation image generation processing based on the motion estimation result, and noise reduction processing using the motion compensation image is executed. These two images are not ineligible images, but are images capable of highly accurate motion estimation. Therefore, it is possible to execute signal processing such as highly accurate motion estimation processing, motion compensation image generation processing based on motion estimation results, and noise reduction processing using motion compensation images.

As described above, in the process shown in FIG. 2 described above, the processes of the two processing blocks of Step S01a and Step S02a are both affected by the influence of one ineligible image (input image t-1). For example, the two output images are error images that are not sufficiently reduced in noise.
However, in the processing shown in FIG. 3, only one processing block in step S11a is affected by the influence of the ineligible image (input image t-1), and the processing in step S12a is not affected.

  For example, when the signal processing executed in each step is noise reduction processing, an image for which noise reduction is insufficient is only an image generated in one processing block in step S11a. In the processing in step S12a, good noise reduction is performed. It is possible to generate and output an image that has been processed.

FIG. 4 shows a configuration example of the signal processing unit when the signal processing executed in the image processing apparatus is noise reduction processing.
As shown in FIG. 4, the signal processing unit of the image processing apparatus inputs a new input image 101 and a motion compensated image 102 obtained by matching the past input image with the image position of the input image.
For example, the still region detection unit 111 compares these two images, extracts a still region where a still subject is photographed, and generates a still region map 103 having extraction information of the still subject pixel region. This still area determination process can be executed based on a similarity determination process between blocks, for example, by comparing the input image 101 and the motion compensated image 102 between pixel blocks formed of a predetermined pixel area.

The pixel value correction unit 112 corrects the pixel value of the pixel of the input image 101 determined as a still region based on the still region 103 using the corresponding pixel of the input image 101 and the motion compensated image 102. Specifically, for example, an average value of pixel values of corresponding pixels of the input image 101 and the motion compensated image 102 is set as a corrected pixel value. For the pixel determined to be a motion region, the pixel value of the input image 101 is used as the output pixel as it is without executing correction.
The processed image 104 generated in this way is output as an image with reduced noise.

  The example shown in FIG. 4 is a signal processing configuration example when noise reduction is performed as signal processing. The processing of the present disclosure includes not only such noise reduction (NR) processing, but also, for example, super-resolution (SR) processing for increasing resolution, or high dynamic range (HDR) generated by combining a plurality of images with different exposure times. The present invention can be applied to a configuration that executes various signal processing such as image generation processing.

[1-2. Image processing apparatus for performing processing based on determination of propriety of motion estimation (Example 1)]
With reference to FIG. 5 and the following, an embodiment of an image processing apparatus that performs processing based on determination of propriety of motion estimation will be described as a first embodiment of the present disclosure.
FIG. 5 is a diagram for explaining the main configuration and processing of an image processing apparatus that executes image processing according to the first embodiment.

The image processing apparatus illustrated in FIG. 5 includes a frame memory 201, a motion estimation appropriateness determination unit 202, a motion estimation unit 203, a motion compensated image generation unit 205, a signal processing unit 206, and a frame memory stored image update control switch 207.
The configuration illustrated in FIG. 5 corresponds to a partial configuration of an imaging device that captures moving images or still images, or a device such as a PC that can perform image processing.
Although not shown in FIG. 5, the image processing apparatus includes, for example, a control unit including a CPU having a program execution function, a memory storing a program executed in the control unit, and the like. The process to which the configuration of FIG. 5 is applied is executed as a process according to a program executed in the control unit, for example.

The image processing apparatus shown in FIG. 5 executes processing for continuously shot images. For example, signal processing such as noise reduction processing, super-resolution processing, and high dynamic range image generation processing.
These signal processes are executed in the signal processing unit 206 shown in FIG.
A processed image 215 that is an output image of the signal processing unit 206 is an image that has been subjected to noise reduction processing, super-resolution processing, high dynamic range, or at least one of these processing.

As the input image 211 shown in FIG. 5, continuously shot images are sequentially input.
The input image 211 is input to the motion estimation appropriateness determination unit 202.
The motion estimation appropriateness determination unit 202 performs analysis of the input image 211 to determine whether or not the input image 211 is an image that can execute an accurate motion estimation process.

This motion estimation appropriateness determination process is executed based on, for example, the ratio of the blur region included in the image. For example, it is difficult to search for a corresponding area between the preceding and following captured images for an image with many areas that are out of focus and blurred, and correct motion estimation is not performed.
Note that there are various processing modes for the determination process of the motion estimation appropriateness for the image, and a specific example will be described later.

When it is determined that the input image 211 is an image capable of executing an accurate motion estimation process, the motion estimation appropriateness determination unit 202 outputs control information for updating the reference image stored in the frame memory 201. That is, the reference image update information 218 in which the update flag [0] is set is output.
On the other hand, if it is determined that the input image 211 is not an image that can be subjected to accurate motion estimation processing, the control information that maintains the current reference image is output without updating the reference image stored in the frame memory 201. . That is, the reference image update information 218 in which the update flag [1] is set is output.
The frame memory stored image update control switch 207 is switched according to the update flag values [0] and [1].

  Note that the reference image stored in the frame memory 201 is an image that is a generation source of the motion compensated image, and the motion compensated image generation unit 205 uses the “reference image” to convert the reference image into the image of the input image. A motion compensated image that matches the position is generated.

As described above, in the image processing apparatus according to the first embodiment, when the input image 211 is an image capable of executing accurate motion estimation processing and the value of the update flag is [0], the input image 211 is newly set. A process of storing in the frame memory 201 as a simple reference image is performed.
On the other hand, when the input image 211 is not an image capable of performing accurate motion estimation processing and the value of the update flag is [1], the input image 211 is not stored in the frame memory 201 but is stored in the frame memory 201. A process of maintaining the reference image as it is is performed.

The reference image stored in the frame memory 201 is compared with the input image 211 in the motion estimation unit 203 and applied to a motion estimation process for calculating a motion amount 212 between the images based on the positional deviation between the two images.
For example, it is assumed that image frames F1, F2, F3, F4,...
When all of these input image frames F1, F2, F3, and F4 are images suitable for motion estimation processing, the reference image that is a stored image in the frame memory 201 is sequentially updated with the image frames F1, F2, F3, and F4. Is done.

  However, for example, when the image frame F3 is an ineligible image that is not suitable for motion estimation, the reference image that is a stored image in the frame memory 201 is updated in the order of the image frames F1, F2, F2, and F4. That is, the image frame F <b> 3 that is an inappropriate image is not stored in the frame memory 102.

The motion estimation unit 203 compares the input image 211 and the reference image stored in the frame memory 201, and calculates a motion amount 212 between the images based on the positional deviation between the two images.
The motion amount 212 is output to the motion compensated image generation unit 205. The motion amount 212 includes motion vector information, frame information of two image frames for which motion vectors are calculated, and the like.

  The motion compensation image generation unit 205 applies the motion amount 212 input from the motion estimation unit 203 to generate a motion compensation image 214 in which the image position of the reference image stored in the frame memory 201 is matched with the image position of the input image. To do. The generated motion compensated image 214 is output to the signal processing unit 206 together with the input image 211.

The signal processing unit 206 executes signal processing using the input image 211 and a reference image that is aligned with the image position of the input image 211, that is, the motion compensation image 214.
For example, the noise reduction processing (NR), super-resolution processing (SR), or high dynamic range (HDR) image generation processing described with reference to FIG. 4 is executed.
The signal processing unit 206 outputs a processed image 215 generated by signal processing.
The processed image 215 is further fed back to the motion compensated image generation unit 205 to generate a motion compensated image based on the processed image, and the signal processing unit 205 further reduces image noise by applying the generated image. It is good also as a structure to perform repeatedly.

A processing example using the configuration shown in FIG. 5 will be described with reference to FIG.
Assume that the input image is input in the order of A, B, C, D, E... As shown in FIG. These image frames A to E are images taken continuously.
In these image frames A to E, an image suitable for motion estimation and an unsuitable image are mixed.
The image frames A, D, and E are images suitable for motion estimation, and the image frames B and C are images that are not suitable for motion estimation, that is, ineligible images.

The motion estimation appropriateness determination unit 202 determines whether each image is suitable for motion estimation or is unsuitable, and outputs reference image update information corresponding to the determination result.
This is (3) reference image update information shown in FIG.
The frame memory stored image update control switch 207 shown in FIG. 5 is switched according to the value of this reference image update information.
The reference image stored in the frame memory 201 is updated according to this switch change.

The reference image stored in the frame memory 201 is updated when the latest preferred image is input, as shown in (2) Reference image stored in frame memory in FIG.
At the timing when the input image is the image frame A, the reference image is not stored in the frame memory 201.
At the timing when the input image is an image frame B, the frame memory 201 stores an image frame A as a reference image.

At the timing when the input image is the image frame C, the frame memory 201 still stores the image frame A as a reference image. This is because the image frame B is an unqualified image and is not stored in the frame memory 201.
Even when the input image is at the timing of the image frame D, the frame memory 201 still stores the image frame A as a reference image. This is because the image frame C is also an unqualified image and is not stored in the frame memory 201.

  At the timing when the input image is an image frame E, the frame memory 201 is in a state where an image frame D is stored as a reference image. This is because the image frame D is an image suitable for motion estimation, and the stored image in the frame memory 201 is updated.

The motion estimation unit 203 receives the input image 211 and the reference image stored in the frame memory 201, and estimates the motion amount between these images.
However, in this embodiment, the reference image stored in the frame memory 201 is not necessarily an image one frame before the input image 211. There are various settings such as one frame before, two frames before, and three frames before.

As shown in FIG. 6 (4), the motion estimation unit 203 uses the reference image stored in the frame memory 201 shown in FIG. 6 (2) to match the image position of the input image shown in FIG. 6 (1). Estimate the amount of motion.
For example, as shown in FIG. 6 (4), when the input image is an image B, a motion amount for adjusting the image A, which is a reference image stored in the frame memory 201, to the image position of the input image B is calculated.
When the input image is the image C, a motion amount for adjusting the image A, which is a reference image stored in the frame memory 201, to the image position of the input image C is calculated.
The same applies hereinafter.

The motion compensated image generation unit 205 applies the motion amount 212 input from the motion estimation unit 203 to generate a motion compensated image 214 that combines the image position of the reference image stored in the frame memory 201 with the image position of the input image, for example. Generate.
For example, as shown in FIG. 6 (5), when the input image is an image B, a motion compensated image is generated by matching the image A, which is a reference image stored in the frame memory 201, with the image position of the input image B.
When the input image is the image C, a motion compensated image is generated by matching the image A, which is a reference image stored in the frame memory 201, with the image position of the input image C.
The same applies hereinafter.
As shown in FIG. 5, the motion compensation image 214 generated by the motion compensation image generation unit 205 is output to the signal processing unit 206 together with the input image 211.

Next, a processing sequence of processing executed by the image processing apparatus shown in FIG. 5 will be described with reference to a flowchart shown in FIG.
For example, in the image processing apparatus having the configuration shown in FIG. 5, the processing according to the flow shown in FIG. 7 is performed under the control of the control unit of the image processing apparatus, although not shown in FIG. The control unit executes a program stored in the memory of the image processing apparatus to control processing according to the flow of FIG.
The flow shown in FIG. 7 is a process executed corresponding to one input image in the continuously shot images input by the image processing apparatus, and the process according to the flow shown in FIG. 7 is repeated for each input image. Executed.

First, in step S101, an image is input. Here, the kth image frame is input.
Next, the process after step S102 and the process after step S111 are executed in parallel.

First, the processing after step S102 will be described.
In step S102, a motion estimation process is executed. This process is a process executed by the motion estimation unit 203 shown in FIG. 5 and is a process for estimating the amount of motion between the reference image stored in the frame memory 201 and the input image.

  Next, in step S103, motion compensation between the reference image and the input image is performed by applying the amount of motion between the reference image and the input image stored in the frame memory 201 calculated in step S102. Generate an image. This process is a process executed by the motion compensated image generation unit 205 shown in FIG.

Next, in step S104, signal processing using the input image and the motion compensated image is executed. This process is a process executed by the signal processing unit 206 shown in FIG. Examples of signal processing include noise reduction processing, super-resolution processing, and high dynamic range image generation processing.
In step S105, the image generated as the signal processing result in step S104 is output.

Next, the process after step S111 is demonstrated.
In step S111, it is determined whether or not the input image is an image suitable for motion estimation. This process is a process executed by the motion estimation appropriateness determination unit 202 shown in FIG.

  If it is determined in step S112 that the input image is an image suitable for motion estimation, the process proceeds to step S113. In step S113, the frame memory stored image update control switch 207 shown in FIG. 5 is controlled to execute a reference image update process for replacing the reference image stored in the frame memory 201 with the input image.

  On the other hand, if it is determined in step S112 that the input image is an ineligible image that is not suitable for motion estimation, the process proceeds to step S114. In step S114, the frame memory stored image update control switch 207 shown in FIG. 5 is controlled to maintain the reference image stored in the frame memory 201 as it is. In other words, the reference image stored in the frame memory is applied to the motion estimation for the next input image without being replaced with the input image.

  With this processing, the reference image applied to motion estimation is always an image suitable for motion estimation, and highly accurate motion estimation is always executed. As a result, a highly accurate motion compensated image based on the highly accurate motion estimation result is generated, and it is possible to execute highly accurate signal processing using the highly accurate motion compensated image.

[1-3. Image processing apparatus that performs processing based on determination of appropriateness of motion estimation (second embodiment)]
Next, an image processing apparatus according to the second embodiment of the present disclosure will be described with reference to FIG.
Similarly to the first embodiment, the second embodiment is an embodiment of an image processing apparatus that performs processing based on determination of suitability for motion estimation.
FIG. 8 is a diagram for explaining the main configuration and processing of an image processing apparatus that executes image processing according to the second embodiment.

The image processing apparatus shown in FIG. 8 includes a frame memory 201, a motion estimation appropriateness determination unit 202, a motion estimation unit 203, a motion amount correction unit 204, a motion compensation image generation unit 205, a signal processing unit 206, and a frame memory stored image update control. A switch 207 is included.
The configuration illustrated in FIG. 8 corresponds to a partial configuration of an imaging device that captures a moving image or a still image, or a device such as a PC that can perform image processing.
Although not shown in FIG. 8, the image processing apparatus includes, for example, a control unit including a CPU having a program execution function, a memory storing a program executed in the control unit, and the like. The process to which the configuration of FIG. 8 is applied is executed as a process according to a program executed in the control unit, for example.

The image processing apparatus shown in FIG. 8 is different in that a motion amount correction unit 204 is added to the configuration described above with reference to FIG.
In other configurations, basically, the same processing as the processing of the components described above with reference to FIG.

Similar to the image processing apparatus described with reference to FIG. 5, the image processing apparatus illustrated in FIG. For example, signal processing such as noise reduction processing, super-resolution processing, and high dynamic range image generation processing.
These signal processes are executed in the signal processing unit 206 shown in FIG.
A processed image 215 that is an output image of the signal processing unit 206 is an image that has been subjected to noise reduction processing, super-resolution processing, high dynamic range, or at least one of these processing.

As the input image 211 shown in FIG. 8, continuously shot images are sequentially input.
The input image 211 is input to the motion estimation appropriateness determination unit 202.
The motion estimation appropriateness determination unit 202 performs analysis of the input image 211 to determine whether or not the input image 211 is an image that can execute an accurate motion estimation process.

This motion estimation appropriateness determination process is executed based on, for example, the ratio of the blur region included in the image. For example, it is difficult to search for a corresponding area between the preceding and following captured images for an image with many areas that are out of focus and blurred, and correct motion estimation is not performed.
Note that there are various processing modes for the determination process of the motion estimation appropriateness for the image, and a specific example will be described later.

When it is determined that the input image 211 is an image that can be subjected to accurate motion estimation processing, the motion estimation appropriateness determination unit 202 uses an update flag [] as control information for updating the reference image stored in the frame memory 201. 0] is output as reference image update information 218.
On the other hand, when it is determined that the input image 211 is not an image capable of executing accurate motion estimation processing, the reference image update in which the update flag [1] is set as control information that does not update the reference image stored in the frame memory 201 Information 218 is output.

The frame memory stored image update control switch 207 is switched according to the update flag values [0] and [1].
When the input image 211 is an image that can execute accurate motion estimation processing and the value of the update flag is [0], the input image 211 is set to be stored in the frame memory 201.
On the other hand, if the input image 211 is not an image capable of performing accurate motion estimation processing and the value of the update flag is [1], the input image 211 is set not to be stored in the frame memory 201 and stored in the frame memory 201. The reference image is maintained as it is.

The reference image stored in the frame memory 201 is an image to be applied to the motion estimation process in which the motion estimation unit 203 compares the input image 211 with the input image 211 and calculates the motion amount 212 between the images based on the positional deviation between the two images. .
For example, it is assumed that image frames F1, F2, F3, F4,...
When all of these input image frames F1, F2, F3, and F4 are images suitable for motion estimation processing, the reference image that is a stored image in the frame memory 201 is sequentially updated with the image frames F1, F2, F3, and F4. Is done.

  However, for example, when the image frame F3 is an ineligible image, the reference image that is a stored image in the frame memory 201 is updated in the order of the image frames F1, F2, F2, and F4. That is, the image frame F <b> 3 that is an inappropriate image is not stored in the frame memory 102.

The motion estimation unit 203 compares the input image 211 and the reference image stored in the frame memory 201, and calculates a motion amount 212 between the images based on the positional deviation between the two images.
The motion amount 212 is output to the motion amount correction unit 204. The motion amount 212 includes motion vector information and frame information of two image frames for which motion vectors are calculated.

  However, in the present embodiment, the motion estimation unit 203 receives information from the motion estimation suitability determination unit 202 as to whether the input image is an image suitable for motion estimation or an inappropriate image, that is, reference image update information 218. It has a configuration to input. The motion estimation unit 203 performs the motion estimation process only when the input image is an image suitable for motion estimation. When the input image is not an image suitable for motion estimation but an unqualified image, the motion estimation process is not executed.

The motion amount correction unit 204 receives the motion amount 212 from the motion estimation unit 203 and further receives reference image update information 218 from the motion estimation appropriateness determination unit 202.
Based on the reference image update information 218, the motion amount correction unit 204 determines whether the input image is an image suitable for motion estimation or an inappropriate image, and the input image is an image suitable for motion estimation. In some cases, the motion amount 212 input from the motion estimation unit 203, specifically, for example, a motion vector indicating the motion between the input image and the reference image is output to the motion compensated image generation unit 205.

On the other hand, if the input image is not an image suitable for motion estimation but an unqualified image, the motion amount 212 is not input from the motion estimation unit 203, so that a preset exceptional motion amount is output to the motion compensated image generation unit 205. .
As the exceptional motion amount, for example, motion vector information without motion, that is, zero vector is applied.
Alternatively, the motion vector calculated in the previous frame may be directly applied and output as the exceptional motion amount. Or it is good also as a setting which outputs the motion vector calculated in the following flame | frame. Or it is good also as a structure which calculates the vector estimated based on the motion vector set in the frame before and behind, and outputs the calculated vector.
In any case, an image applied to motion vector calculation is a vector calculated using an image suitable for motion estimation that does not include an ineligible image.

The motion amount correction unit 204 responds to the determination result based on the determination information based on the reference image update information 218, that is, the determination information on whether the input image is an image suitable for motion estimation or an inappropriate image. The obtained motion amount information is output to the motion compensated image generation unit 205.
That is, when the input image is an image suitable for motion estimation, the motion amount 212 input from the motion estimation unit 203 is output to the motion compensation image generation unit 205, and the input image is not an image suitable for motion estimation but is ineligible. If it is an image, the exceptional motion amount is output to the motion compensated image generation unit 205.

  The motion compensated image generation unit 205 applies the motion amount or the exceptional motion amount 213 input from the motion amount correction unit 204 to match the image position of the reference image stored in the frame memory 201 with the image position of the input image, for example. A motion compensated image 214 is generated. The generated motion compensated image 214 is output to the signal processing unit 206 together with the input image 211.

The signal processing unit 206 executes signal processing using the input image 211 and a reference image that is aligned with the image position of the input image 211, that is, the motion compensation image 214.
For example, the noise reduction processing (NR), super-resolution processing (SR), or high dynamic range (HDR) image generation processing described with reference to FIG. 4 is executed.
The signal processing unit 206 outputs a processed image 215 generated by signal processing.
The processed image 215 is further fed back to the motion compensated image generation unit 205 to generate a motion compensated image based on the processed image, and the signal processing unit 205 further reduces image noise by applying the generated image. It is good also as a structure to perform repeatedly.

A processing sequence of processing executed by the image processing apparatus shown in FIG. 8 will be described with reference to a flowchart shown in FIG.
9 is performed under the control of the control unit of the image processing apparatus in the image processing apparatus having the configuration shown in FIG. 8, for example. The control unit executes a program stored in the memory of the image processing apparatus to control processing according to the flow of FIG.
The flow shown in FIG. 9 is a process executed corresponding to one input image in the continuously shot images input by the image processing apparatus, and the process according to the flow shown in FIG. 9 is repeated for each input image. Executed.

First, in step S121, an image is input. Here, the kth image frame is input.
Next, in step S122, it is determined whether the input image is an image suitable for motion estimation or an unsuitable image.
This process is a process executed by the motion estimation appropriateness determination unit 202 shown in FIG.

If it is determined in step S123 that the input image is an image suitable for motion estimation, the processing after step S124 and the processing in step S131 are executed.
On the other hand, if it is determined in step S123 that the input image is an ineligible image that is not suitable for motion estimation, the processing in step S125 and the processing in step S132 are executed.

First, the process of step S131 and the process of step S132 will be described.
When it is determined in step S123 that the input image is an image suitable for motion estimation, the reference image stored in the frame memory 201 is controlled by controlling the frame memory stored image update control switch 207 shown in FIG. A reference image update process for replacing with an input image is executed.

  On the other hand, if it is determined in step S123 that the input image is an ineligible image that is not suitable for motion estimation, the frame memory stored image update control switch 207 shown in FIG. Control is performed to maintain the stored reference image as it is. In other words, the reference image stored in the frame memory is applied to the motion estimation for the next input image without being replaced with the input image.

  With this processing, the reference image applied to motion estimation is always an image suitable for motion estimation, and highly accurate motion estimation is always executed. As a result, a highly accurate motion compensated image based on the highly accurate motion estimation result is generated, and it is possible to execute highly accurate signal processing using the highly accurate motion compensated image.

Next, processing in steps S124 to S126 will be described.
If it is determined in step S123 that the input image is an image suitable for motion estimation, motion estimation processing is executed in step S124. This process is a process executed by the motion estimation unit 203 shown in FIG. 8, and is a process for estimating the amount of motion between the reference image stored in the frame memory 201 and the input image.

  On the other hand, when it is determined that the input image is not an image suitable for motion estimation but an unqualified image, in step S125, an exceptional motion amount is calculated according to a preset algorithm. Specifically, for example, a motion vector corresponding to an input image calculated based on a zero vector or a motion vector calculated in the preceding and subsequent image frames is set as the exceptional motion amount.

Step S126 is processing of the motion amount correction unit 204 shown in FIG. The following processing is executed according to the determination result information of the motion estimation appropriateness determination unit 202.
If the input image has a motion estimation suitable degree, the motion amount calculated in step S124 is output to the motion compensated image generation unit 205.
On the other hand, if the input image does not have the motion estimation suitability, the exceptional motion amount calculated in step S125 is output to the motion compensated image generation unit 205.

Next, in step S127, a motion compensation image is generated by applying a motion amount or an exceptional motion amount. This process is a process executed by the motion compensated image generation unit 205 shown in FIG. This motion compensated image generation processing is executed by applying the amount of motion calculated in step S124 when the input image has a motion estimation suitable degree.
On the other hand, when the input image does not have the motion estimation suitability, the exceptional motion amount calculated in step S125 is applied and executed.

Next, in step S128, signal processing using the input image and the motion compensation image is executed. This process is a process executed by the signal processing unit 206 shown in FIG. Examples of signal processing include noise reduction processing, super-resolution processing, and high dynamic range image generation processing.
In step S129, the image generated as the signal processing result in step S128 is output.

[1-4. Image processing apparatus that performs processing based on determination of propriety of motion estimation (third embodiment)]
Next, a third embodiment of the present disclosure will be described with reference to FIG.
The third embodiment is also an embodiment of an image processing apparatus that performs processing based on determination of propriety of motion estimation as in the first and second embodiments described above.
FIG. 10 is a diagram for explaining the main configuration and processing of an image processing apparatus that executes image processing according to the third embodiment.

The image processing apparatus illustrated in FIG. 10 includes a frame memory 201, a motion estimation appropriateness determination unit 202, a motion estimation unit 203, a motion compensation image generation unit 205, a signal processing unit 206, and a frame memory stored image update control switch 207.
The configuration illustrated in FIG. 10 corresponds to, for example, a partial configuration of an imaging device that captures a moving image or a still image, or a device such as a PC that can perform image processing.
Although not shown in FIG. 10, the image processing apparatus includes, for example, a control unit including a CPU having a program execution function, a memory storing a program executed in the control unit, and the like. The process to which the configuration of FIG. 10 is applied is executed as a process according to a program executed in the control unit, for example.

The image processing apparatus shown in FIG. 10 has the same configuration as that of the first embodiment described above with reference to FIG.
The difference from the first embodiment shown in FIG. 5 is that the determination result of the motion estimation appropriateness determination unit 202, that is, the reference image indicating whether the input image 211 is an image suitable for motion estimation or an unqualified image. The update information 218 is input to the motion estimation unit 203 and the signal processing unit 206.

The motion estimation unit 203 executes or cancels the motion estimation process according to the reference image update information 218 that is the determination result of the motion estimation appropriateness determination unit 202.
Specifically, the motion estimation unit 203 performs the following processing.
When the reference image update information 218 that is the determination result of the motion estimation appropriateness determination unit 202 is updated [0], that is, when the determination result that the input image 211 is an image suitable for motion estimation is input, the motion estimation process And the motion amount 212 is output to the motion compensated image generation unit 205.
On the other hand, the reference image update information 218 that is the determination result of the motion estimation appropriateness determination unit 202 is maintained [1], that is, the determination result that the input image 211 is not an image suitable for motion estimation but an inappropriate image is input. If so, the motion estimation process is stopped. In this case, the motion amount 212 is not output to the motion compensated image generation unit 205.

The motion compensation image generation unit 205 generates the motion compensation image 214 only when the motion amount 212 is input from the motion estimation unit 203 and outputs the motion compensation image 214 to the signal processing unit 206.
That is, only when the input image 211 is an image suitable for motion estimation, the motion compensated image 214 is generated and output to the signal processing unit 206. When the input image 211 is an ineligible image, the motion compensated image 214 is output. And only the input image 211 is output to the signal processing unit 206.

The signal processing unit 206 performs signal processing execution or cancellation processing according to the reference image update information 218 that is the determination result of the motion estimation appropriateness determination unit 202.
Specifically, the signal processing unit 206 executes the following processing.
When the reference image update information 218, which is the determination result of the motion estimation appropriateness determination unit 202, is updated [0], that is, when the determination result that the input image 211 is an image suitable for motion estimation is input, signal processing is performed. Run.
On the other hand, the reference image update information 218 that is the determination result of the motion estimation appropriateness determination unit 202 is maintained [1], that is, the determination result that the input image 211 is not an image suitable for motion estimation but an inappropriate image is input. If so, the signal processing is stopped. In this case, the input image 211 is output as the processed image 215 as it is.

The signal processing executed by the signal processing unit 206 is signal processing such as noise reduction processing, super-resolution processing, and high dynamic range image generation processing.
In the configuration of the present embodiment, signal processing is performed only when the input image 211 is an image suitable for motion estimation, and signal processing is not performed when the input image 211 is an image that is unsuitable for motion estimation. It is a configuration.
The processing of other processing units is the same as that of the first embodiment described above with reference to FIG.

The processing sequence of the third embodiment will be described with reference to the flowchart shown in FIG.
For example, in the image processing apparatus having the configuration shown in FIG. 10, the processing according to the flow shown in FIG. 11 is performed under the control of the control unit of the image processing apparatus, although not shown in FIG. 10. The control unit executes a program stored in the memory of the image processing apparatus to control processing according to the flow of FIG.
The flow shown in FIG. 11 is a process executed corresponding to one input image in the continuously shot images input by the image processing apparatus, and the process according to the flow shown in FIG. 11 is repeated for each input image. Executed.

First, an image is input in step S141. Here, it is assumed that the k-th image frame is input.
Next, in step S142, it is determined whether the input image is an image suitable for motion estimation or an unsuitable image that is not suitable.
This process is a process executed by the motion estimation appropriateness determination unit 202 shown in FIG.

If it is determined in step S143 that the input image is an image suitable for motion estimation, the processing after step S144 and the processing in step S151 are executed.
On the other hand, if it is determined in step S143 that the input image is an ineligible image that is not suitable for motion estimation, the processing in step S161 and the processing in step S152 are executed.

First, the process of step S151 and the process of step S152 will be described.
If it is determined in step S143 that the input image is suitable for motion estimation, the reference image stored in the frame memory 201 is controlled by controlling the frame memory stored image update control switch 207 shown in FIG. 10 in step S151. A reference image update process for replacing with an input image is executed.

  On the other hand, if it is determined in step S143 that the input image is an ineligible image that is not suitable for motion estimation, the frame memory stored image update control switch 207 shown in FIG. Control is performed to maintain the stored reference image as it is. In other words, the reference image stored in the frame memory is applied to the motion estimation for the next input image without being replaced with the input image.

  By this processing, an image applied to motion estimation is always an image suitable for motion estimation, and highly accurate motion estimation is always executed. As a result, a highly accurate motion compensated image based on the highly accurate motion estimation result is generated, and it is possible to execute highly accurate signal processing using the highly accurate motion compensated image.

Next, the process of steps S144 to S147 will be described.
If it is determined in step S143 that the input image is an image suitable for motion estimation, motion estimation processing is executed in step S144. This process is a process executed by the motion estimation unit 203 shown in FIG. 10, and is a process for estimating the amount of motion between the reference image stored in the frame memory 201 and the input image.

  Step S145 is processing of the motion compensated image generation unit 205 shown in FIG. The motion compensated image generation unit 205 executes the process of step S145 only when the determination result of the motion estimation appropriateness determination unit 202 is input as the determination result that the input image has the motion estimation appropriateness. That is, a motion compensation image is generated by applying the motion amount estimated in step S144.

Next, in step S146, signal processing is executed. This process is a process executed by the signal processing unit 206 shown in FIG. The signal processing unit 206 executes the process of step S146 only when the input image has received a determination result indicating that there is a motion estimation appropriateness as the determination result of the motion estimation appropriateness determining unit 202. That is, signal processing using the input image and the motion compensated image is executed. Examples of signal processing include noise reduction processing, super-resolution processing, and high dynamic range image generation processing.
In step S147, the image generated as the signal processing result in step S146 is output.

On the other hand, when it is determined in step S143 that the input image is not an image suitable for motion estimation but an unqualified image, the input image is set as a processed image as it is in step S161, and the input image is processed as a processed image in step S147. Output as.
That is, if it is determined that the input image is not an image suitable for motion estimation but an unqualified image, all the processes in steps S144 to S146 are omitted, and the input image is output as a processed image as it is.

[1-5. Details of motion estimation appropriateness determination processing]
Next, a process executed by the motion estimation suitability determination unit 202 shown in each of the above-described embodiments, that is, a process for determining whether the input image 211 is an image suitable for the motion estimation process or an unqualified image. A specific example will be described.

Various indexes can be applied to the process of determining whether the input image 211 is an image suitable for the motion estimation process or an inappropriate image. FIG. 12 shows an example of an index applicable to this determination process. As shown in FIG. 12, for example, the following index can be used to determine whether the input image 211 is an image suitable for motion estimation processing or an inappropriate image.
(1) Area of blurred area in image (2) Camera motion information at the time of image capture (3) Area of pixel value saturated area in image (4) Area of noisy area in image (5) Image area Luminance range (range from maximum luminance to minimum luminance)
(6) Effective pixel area of maximum exposure image and minimum exposure image

  For example, when the motion estimation appropriateness determination unit 202 performs the determination process by applying “(1) the area of the blurred region in the image”, the area occupation ratio of the “blurred region” in the entire image of the input image is predetermined. If it is less than the threshold, the input image is determined to be an image suitable for motion estimation, and if the area occupancy of the “blurred area” is greater than or equal to a predetermined threshold, the input image is an ineligible image that is not suitable for motion estimation. Judge that there is.

  In addition, when the motion estimation appropriateness determination unit 202 performs the determination process by applying “(2) camera motion information at the time of image capture”, the motion estimation appropriateness determination unit 202 includes a gyro mounted on the camera or the like. The movement of the camera at the time of image capture acquired from the sensor is input as image attribute information together with the image. The motion estimation appropriateness determination unit 202 performs determination based on this information. If the “camera movement” at the time of shooting the input image is less than the predetermined threshold, the input image is determined to be an image suitable for motion estimation. If the “camera movement” is equal to or greater than the predetermined threshold, the input image Is determined to be an ineligible image not suitable for motion estimation.

  When performing the determination process by applying “(3) the area of the pixel value saturation region in the image”, if the area occupation ratio of the “pixel value saturation region” in the entire image of the input image is less than the predetermined threshold value, The input image is determined to be an image suitable for motion estimation. If the area occupancy rate of the “pixel value saturation region” is equal to or greater than a predetermined threshold value, the input image is determined to be an ineligible image not suitable for motion estimation.

  When performing the determination process by applying “(4) the area of a noisy area in an image”, if the area occupation ratio of the “noisy area” in the entire image of the input image is less than a predetermined threshold value, The input image is determined to be an image suitable for motion estimation. If the area occupancy ratio of the “noisy region” is equal to or greater than a predetermined threshold value, the input image is determined to be an ineligible image not suitable for motion estimation.

When performing determination processing by applying “(5) image luminance range (range from maximum luminance to minimum luminance)”, the range of pixel values of the entire image of the input image is analyzed, and the maximum pixel value to the minimum pixel value If the luminance range is greater than or equal to the predetermined threshold, the input image is determined to be an image suitable for motion estimation, and if the luminance range is less than the predetermined threshold, the input image is an ineligible image that is not suitable for motion estimation. judge.
Note that, when this process is performed, the determination may be made by applying only pixel values excluding pixels of several percent of the maximum luminance and the minimum luminance in the image.
Further, the variance of the pixel value may be calculated and the determination based on the variance value may be performed.

The determination process to which “(6) the area of the effective pixel area of the maximum exposure image and the minimum exposure image” is applied is a process in the case of generating a high dynamic range image. When generating a high dynamic range image, continuous captured images with different exposure times are input, and a plurality of images with different exposure times are combined to generate one output image.
In this case, for example, a long exposure image includes many saturated pixels, while a short exposure image includes many low luminance pixels. An effective pixel is a pixel that is not saturated in a long-time exposure image, and in a short-time exposure image, a pixel having an effective pixel value other than an extremely low luminance pixel is set. A high dynamic range image is generated by performing image synthesis in which these effective pixels are combined.

In such a configuration for generating a high dynamic range image, processing is performed by continuously inputting a plurality of images having different exposure times from a long-time exposure image to a short-time exposure image continuously shot by the imaging apparatus.
In this case, one image is stored in the frame memory as a reference image from these images having a plurality of different exposure times.

  Note that the reference image is one selected image among continuous shot images selected when generating a noise-reduced image or a high dynamic range (HDR) image. Based on the other images, a motion compensated image that matches the image position of this “reference image” is generated, and the “reference image” and the “motion compensated image generated based on the other images” are combined to generate noise. Signal processing such as reduction and high dynamic range (HDR) image generation is performed.

In such a reference image selection process, a determination process using “the area of the effective pixel area of the maximum exposure image and the minimum exposure image” is an effective technique.
An image having a larger effective pixel area is used as an image suitable for motion estimation, and is stored in the frame memory as a reference image.

As described above, the motion estimation appropriateness determination unit 202 determines that each input image has been subjected to motion estimation based on, for example, any one of (1) to (6) illustrated in FIG. It is determined whether the image is an image or an ineligible image.
Furthermore, only the input image determined to be an image suitable for motion estimation by the motion estimation appropriateness determination unit 202 is newly stored in the frame memory 201 and used as a reference image or a standard image to be applied to subsequent motion estimation. Is done.

FIG. 12 shows the following indices as indices applicable to the motion estimation appropriateness determination process.
(1) Area of blurred area in image (2) Camera motion information at the time of image capture (3) Area of pixel value saturated area in image (4) Area of noisy area in image (5) Image area Luminance range (range from maximum luminance to minimum luminance)
(6) Effective pixel area of maximum exposure image and minimum exposure image

Of each of these indicators, the following indicators:
(1) Area of blurred region in image (3) Area of pixel value saturated region in image (4) Area of noisy region in image (5) Luminance range of image (range from maximum luminance to minimum luminance)
The determination process using each of these indices can be executed by, for example, the motion estimation appropriateness determination unit 202 analyzing the input image.

  In addition, (2) when performing determination processing using camera motion information at the time of image capture, as described above, as a setting for acquiring sensor measurement information of camera motion at the time of image capture as attribute information corresponding to an image. Then, the determination process is performed according to the attribute information.

  When the motion estimation suitability determination unit 202 needs to perform image comparison between the input image and the reference image stored in the frame memory, for example, as shown in FIG. In addition, not only the input image 211 but also the image reading from the frame memory 201 can be executed.

Further, (6) when performing the determination process using the area of the effective pixel region of the maximum exposure image and the minimum exposure image, for example, the determination process using the configuration shown in FIG. 14 is performed.
The configuration shown in FIG. 14 is a configuration in which an effective area determination unit 232 is added to the configuration of FIG. 5 described as the first embodiment.

The configuration shown in FIG. 14 is applied to a configuration in which a continuous exposure long exposure image and a short exposure image are sequentially input as input images.
The effective area determination unit 232 extracts a pixel area having an effective pixel value from each of the long exposure image and the short exposure image that are sequentially input as input images. Furthermore, the following two maps are created as maps reflecting the extraction results.
Maximum exposure effective area map consisting of selection information of effective pixel area of long exposure image,
Minimum exposure effective area map consisting of selection information of effective pixel area of short exposure image,
The effective area determination unit 232 outputs these maps to the motion estimation appropriateness determination unit 202.

  The motion estimation appropriateness determination unit 202 inputs the above two maps from the effective area determination unit 232 and determines which of the long exposure image and the short exposure image has a larger effective pixel area. Depending on the determination result, the larger effective pixel area is selected as a reference image and stored in the frame memory 201.

[1-6. Example of performing noise reduction by superimposing continuously shot images (Example 4)]
Next, an embodiment that performs noise reduction by superimposing continuously shot images will be described as a fourth embodiment.
In the fourth embodiment, similarly to the first to third embodiments, processing based on determination of suitability for motion estimation is performed. A configuration example of the image processing apparatus of the fourth embodiment is shown in FIG.

  In a configuration in which noise is reduced by superimposing continuous shot images, the first shot image among a plurality of shot images continuously shot by the user pressing the shutter of the imaging device is used as a reference image, and the subsequent shot image is aligned with the reference image. It is common to do.

However, the first captured image may be an image suitable for motion estimation. In such a case, alignment between the images may not be performed well, and good noise reduction may not be performed.
In the embodiment described below, in such a case, the reference image is appropriately changed so that good alignment and noise reduction can be performed.

The configuration illustrated in FIG. 15 corresponds to a configuration in which a motion estimation direction switching unit 251 is added to the configuration illustrated in FIG.
The input image 211 and the reference image of the frame memory 201 are input to the motion estimation direction switching unit 251. Further, reference image update information 228 indicating whether or not the input image is suitable for motion estimation is input from the motion determination suitability determination unit 202.

In the first to third embodiments described above, an image stored in the frame memory 201 is a “reference image”, that is, an image that is a generation source of a motion compensated image. The image stored in the memory 201 is a “reference image”.
The “reference image” is an image having the same image position as the processed image to be output. For other input images in the continuous shot image other than the reference image, a synthesis process is executed after generating a motion compensated image set to be the same image position as the reference image, and as one noise reduced image The processed image 215 is output.
Therefore, in this embodiment, the motion determination suitability determination unit 202 outputs “reference image” update information 228 indicating whether or not the input image is suitable for motion estimation.

  In this configuration, the top image of the continuous shot images is stored in the frame memory 201 as a temporary reference image. However, after that, when it is determined that the newly input subsequent captured image is an image suitable for motion estimation, processing for changing the reference image to the image is performed. This reference image update process is the same process as the reference image update process in the first to third embodiments described above.

  Furthermore, the motion estimation direction switching unit 251 determines whether or not the input image 211 is an image suitable for motion estimation according to the reference image update information 228 input from the motion determination appropriateness determination unit 202, and includes the determination information. Accordingly, a process of changing the motion estimation direction is performed.

  That is, when the newly input image is an image suitable for motion estimation, the image is set as a reference image, and the position of each continuous shot image is set in the direction of the reference image. If the newly input image is ineligible for motion estimation, the image is not set as the reference image, and each continuous shooting is performed in the direction of the set reference image or the reference image set thereafter. Set to execute image alignment.

One of the motion estimation target image 261 and the motion estimation reference image 262 shown as outputs of the motion estimation direction switching unit 251 shown in FIG. 15 is an image that is a registration target, and the other is an image that is a registration processing target. It is.
Other configurations and processes are the same as those described with reference to FIG.

A processing sequence according to the present embodiment will be described with reference to flowcharts shown in FIGS.
For example, when the image processing apparatus generates and records or outputs a noise-reduced image of a still image, a synthesis process using a plurality of continuously shot images is executed. That is, the noise reduction image is generated by executing the signal processing described above with reference to FIG.
The flow shown in FIGS. 16 to 17 is a flowchart for explaining a processing sequence in the case where such a noise-reduced image generation process is performed.

  The processing according to the flow illustrated in FIGS. 16 to 17 is performed under the control of the control unit of the image processing apparatus, which is not illustrated in FIG. 15, for example, in the image processing apparatus having the configuration illustrated in FIG. The control unit executes a program stored in the memory of the image processing apparatus to control processing according to the flow shown in FIGS.

First, in step S161, a reference image provisional determination process is performed. The first image that is the first captured image of the continuously captured images is set as a temporary reference image.
The processes in steps S162 to S185 below are executed for each captured image of the continuously captured images.

In step S163, a captured image is input.
Next, in step S164, it is determined whether the input image is an image suitable for motion estimation or an unsuitable image that is not suitable. This process is a process executed by the motion estimation appropriateness determination unit 202 shown in FIG.

In step S165, it is determined whether the reference image has been determined.
If the reference image has been determined, the process proceeds to step S166, and if not, the process proceeds to step S167.

  When the reference image has been determined and the process proceeds to step S166, it is determined in step S166 whether or not the input image is an image suitable for motion estimation. If it is determined that it is suitable, a motion estimation process using the input image and the reference image is executed in step S169. This process is a process executed by the motion estimation unit 203 shown in FIG. 15, and is a process for estimating the amount of motion between the reference image stored in the frame memory 201 and the input image.

  On the other hand, if it is determined in step S166 that the input image is an ineligible image that is not suitable for motion estimation, an exceptional motion amount is calculated in step S170. This is the process of the motion amount correction unit 204 shown in FIG. As described above, the exceptional motion amount is, for example, a zero vector.

On the other hand, if it is determined in step S165 that the reference image has not been determined, the process proceeds to step S167. In step S167, it is determined whether the input image is an image suitable for motion estimation. If it is determined that it is suitable, in step S168, the input image is determined as the next reference image, and the process proceeds to the motion estimation process in step S169.
On the other hand, if it is determined in step S167 that the input image is not an image suitable for motion estimation but an inappropriate image, the process proceeds to step S171. In step S171, a temporary setting process is performed in which the input image is not set as a reference image, and the next image frame is set as a reference image. In step S172, the amount of exceptional motion is calculated. This is the process of the motion amount correction unit 204 shown in FIG. As described above, for example, a zero vector is used as the exceptional motion amount.

When any one of step S170, step S169, and step S172 is completed, the process proceeds to step S181 shown in FIG.
In step S181, a motion amount correction process is executed. This process is a process executed by the motion amount correction unit 204 shown in FIG.
If the input image is an image suitable for motion estimation, the amount of motion between the reference image and the input image is calculated and output to the motion compensated image generation unit 205. On the other hand, when the input image is not an image suitable for motion estimation but an ineligible image, an exceptional motion amount, such as a zero vector, is output to the motion compensated image generation unit 205.

Next, in step S182, a motion compensated image is generated. That is, the motion compensation image is generated by applying the motion amount or the exceptional motion amount. This process is a process executed by the motion compensated image generation unit 205 shown in FIG. This motion compensated image generation processing is executed by applying the amount of motion calculated in step S181 when the input image has a motion estimation suitability.
On the other hand, when the input image does not have the motion estimation suitability, the exceptional motion amount calculated in step S170 or step S172 is applied and executed.

Next, in step S183, signal processing using the input image and the motion compensated image is executed. This process is a process executed by the signal processing unit 206 shown in FIG. Examples of signal processing include noise reduction processing, super-resolution processing, and high dynamic range image generation processing.
In step S184, the image generated as the signal processing result in step S183 is output.
In step S185, it is determined whether or not there are remaining continuously shot images. If there are, the processes in step S163 and subsequent steps are repeated.
If there are no remaining unprocessed images, the process ends.
In this way, a noise reduced image to which a plurality of continuously shot images is applied is generated and output.

Reference image setting and motion compensation image generation processing applied to the motion estimation processing of the present embodiment will be described with reference to FIG.
For example, four images of images k to k + 3 are continuously shot, and the continuous shot images are combined to generate one noise appearance image.
In general, as shown in FIG. 18 (a), after setting the image k, which is the first captured image, as a reference image, and generating a motion compensated image so as to align other image positions with the reference image, A synthesis process for each image is executed.
On the other hand, in the processing according to the present embodiment, as shown in FIG. 18B, for example, the first image k is an image that is not suitable for motion estimation, and the next image k + 1 is an image suitable for motion estimation. If this happens, this image is set as the reference image. A motion compensated image obtained by aligning another image with this image k + 1 is generated, and a noise reduced image based on image synthesis is generated.

[1-7. Embodiment in which High Dynamic Range Image Generation Processing is Performed (Embodiment 5)]
Next, an example in which a high dynamic range image generation process is performed will be described as Example 5.
In the fifth embodiment, similarly to the first to fourth embodiments described above, it is determined whether the input image is an image suitable for motion estimation or an unsuitable image, and processing according to the determination result is executed.
In the configuration for generating the high dynamic range image, a plurality of photographed images with different exposure times are continuously photographed, and the photographed images with the plurality of different exposure times are combined to form one high dynamic range image. Generate an image.

Also in this process, as in the above-described fourth embodiment, it is possible to select which image of a plurality of different exposure time images that have been continuously photographed, that is, whether the image of each degree is used as a reference image.
In this embodiment, an image that is not suitable for motion estimation is not set as a reference image, and an image suitable for motion estimation is selected as a reference image.
This process is the same as that of the fourth embodiment.
The configuration of the image processing apparatus applied to the present embodiment is the configuration of FIG. 15 as in the fourth embodiment.

[2. Example of executing both noise reduction by combining multiple images and high dynamic range processing]
Next, a description will be given of an embodiment in which continuously taken images with different exposure times are combined to perform noise reduction and high dynamic range processing together.
Whether a noise reduced image or a high dynamic range image is generated, a process of combining a plurality of continuously shot images after alignment is performed.

[2-1. General configuration example and problems of executing both noise reduction processing and high dynamic range processing]
First, a general configuration example in which noise reduction processing and high dynamic range processing are executed together and problems will be described.
In the case of generating a high dynamic range image, a process of continuously photographing images set at a plurality of different exposure times and synthesizing these images is performed.
A basic sequence of high dynamic range image generation processing will be described with reference to FIG.

The example shown in FIG. 19 is a processing example using a short-exposure image 301 and a long-exposure image 302 and these two types of setting images having different exposure times.
The short-exposure image 301 and the long-exposure image 302 are continuously photographed by one shutter operation by the user.
The captured image data processing unit of the imaging apparatus first performs exposure correction by multiplying the short exposure image 301 and the long exposure image 302 by a constant corresponding to the exposure ratio in steps S201 and S202.
For example, when the exposure ratio between the short exposure time and the long exposure time (= exposure time ratio) is 1:16, the short exposure image is multiplied by 16 and the long exposure image is multiplied by 1 at the stage of exposure correction. Thus, an exposure correction short exposure image 303 and an exposure correction long exposure image 304 are generated.
Thereafter, in step S203, the pixel values of the corresponding pixels are synthesized (blended) to determine the pixel value of a high dynamic range (HDR) image 305 as an output image.

A pixel value determination processing sequence of a high dynamic range (HDR) image 305 that is an output image will be described.
For example,
D _S : Pixel value of exposure corrected short exposure image D _L : Pixel value of exposure corrected long exposure image D _H : Pixel value of high dynamic range image to be output
The pixel value of the high dynamic range (HDR: High Dynamic Range) image 305 is calculated according to the following formula in the blend process in step S203.
D _H = (1.0−α) × D _S + α × D _L
The pixel value blending according to the above formula is performed for each corresponding pixel position of the exposure correction short-time exposure image and the exposure correction long-time exposure image. That is, blending processing is performed for each shooting pixel position of the same subject to determine each pixel value of the output image (HDR image).

By such processing, for example, even if a pixel has a saturated pixel value in the long exposure image 302, it is possible to calculate an effective pixel value by using the pixel value of the corresponding pixel in the short exposure image 301. In addition, even in a pixel in which the brightness is low in the short-time exposure image 301 and the so-called blackening occurs, an effective pixel value can be set by using the pixel value of the long-time exposure image 302.
By such processing, it is possible to set an effective pixel value for each pixel of low luminance to high luminance, and it is possible to generate a high dynamic range image with an expanded dynamic range.

In the configuration shown in FIG. 19, a combination of two types of exposure time is used. However, various settings can be made for the exposure time, and an arbitrary number N of two or more can be set. For example, the exposure time can be set to four types of settings.
However, when a high dynamic range process and a noise reduction process are executed using a large number of captured images having different exposure times in this way, a large number of frames for temporarily storing each image having a different exposure time. A memory is required.

FIG. 20 shows a configuration example in which a continuous dynamic image having four types of exposure times is used to first generate a high dynamic range image by combining the images, and then perform noise reduction processing.
First, the following four types of images with different exposure times are input.
(1) The shortest time exposure image XS, 321;
(2) Short exposure image MS, 322,
(3) Long exposure images ML, 323,
(4) Longest time exposure image XL, 324,
Each of these images is an image that is continuously photographed, for example, when the user presses the shutter once for the imaging device.
Exposure time is
XS <MS <ML <XL
It is in the above relationship.

Each of these images is stored in a frame memory a331 to a frame memory d334 shown in FIG. The high dynamic range (HDR) image generation unit 335 shown in FIG. 20 combines these four different exposure images to generate one high dynamic range (HDR) image.
For example, low luminance pixels preferentially adopt the pixel value of the longest time exposure image XL324, and high luminance pixels execute pixel value blending preferentially of the pixel value of the shortest time exposure image XS321 to perform a high dynamic range image. Is generated.

  Next, the high dynamic range (HDR) image generated by the high dynamic range (HDR) image generation unit 335 is input to the noise removal unit 336. Here, for example, a generated high dynamic range (HDR) image that has been generated in advance is extracted from the frame memory e337, and for example, noise reduction processing according to the processing described above with reference to FIG. 4 is executed. . As a processing result, a noise-removed HDR image 338 shown in the figure is generated and output.

  As shown in FIG. 20, when a high dynamic range image and noise removal are to be executed sequentially, a large number of frame memories for temporarily storing each image or a large-capacity frame memory are required.

FIG. 21 is a configuration example in which noise removal is executed in advance and a high dynamic range image is generated after noise removal, unlike FIG.
In the configuration shown in FIG. 21, the noise removal unit 341 inputs the following four images.
(1) The shortest time exposure image XS, 321;
(2) Short exposure image MS, 322,
(3) Long exposure images ML, 323,
(4) Longest time exposure image XL, 324,
Further, noise reduction is performed by applying a noise reduction image to the preceding captured image stored in the frame memory a351 to frame memory d354, and the generated noise reduction image corresponding to each exposure time is stored in the frame memory a351 to frame memory d354.

  Thereafter, the high dynamic range (HDR) image generation unit 355 acquires images with different exposure times subjected to the noise reduction processing stored in the frame memory a351 to the frame memory d354, executes the synthesis processing, and performs noise removal HDR. An image 356 is generated and output.

Even in this configuration, a large number of frame memories for temporarily storing each image or a large-capacity frame memory are also required.
As described above, the configurations shown in FIGS. 20 and 21 also have a problem in that the circuit scale and power consumption increase because an increase in frame memory capacity, calculation of individual motion information, alignment processing, and the like are required. In addition, when motion compensated images are generated by individual circuits, image blurring may occur due to errors between circuits, etc., and the possibility that false colors will occur due to subsequent synthesis processing is also expected. .
In the following, an embodiment of an image processing apparatus that realizes processing that prevents such an increase in circuit scale and a decrease in quality of a generated image will be described.

[2-2. Overview of processing to combine noise reduction and high dynamic range image generation according to the present disclosure]
In the following, an overview of processing that executes noise reduction and high dynamic range image generation according to the present disclosure will be described.
For example, as shown in FIG. 22, the image processing apparatus periodically inputs image signals (from a short exposure image signal to a long exposure image signal) captured at a plurality of different exposure times to execute processing. That is,
(1) The shortest time exposure image XS,
(2) Short exposure image MS,
(3) Long exposure image ML,
(4) Longest time exposure image XL,
These four consecutive exposure images with different exposure times are periodically input to execute processing.
The exposure time has the following relationship.
XS <MS <ML <XL
The exposure time has the above relationship.

The image processing apparatus combines these four images with different exposure times to generate and output a noise-reduced high dynamic range (HDR) image.
In the following, an example in which the type of exposure time is set to 4 will be described. However, the processing of the present disclosure is performed in an arbitrary exposure time type = N, that is, an image with an arbitrary exposure time type of N = 2 or more = N. It is possible to execute as a process for.
Also, the input order of the image signals is described in the order of the short exposure image signal to the long exposure image signal, but this is not restrictive.

First, an outline of the composition processing of images with different exposure times executed in the image processing apparatus will be described with reference to FIG.
The example shown in FIG. 23 is a processing example for an image composed of a sky and a house.
Short exposure image,
Long exposure images,
A sequence is described in which both the high dynamic processing for expanding the dynamic range by applying these two different exposure images and the noise removal processing are executed together.

In FIG.
(1) Input image (2) Composite image of previous frame (3) Composite image (4) Update region These figures are shown.
The time elapses from left to right. (1) The input image is a short exposure image (fn1), a long exposure image (fn2), a short exposure image (fn3), a long exposure image (fn4),. The short exposure image (fn (i)) and the short exposure image (fn (i + 1)) are input in this order.

(1) In the input image to (3) composite image, the white area indicates a whiteout area, that is, an area where the pixel value is a saturated pixel value, and the black area is a blackened area, that is, the pixel value is almost zero. The close area is shown. It can be said that the pixel values of these areas are invalid pixel value setting areas. On the other hand, the gray area is an effective area in which an effective pixel value is set.
(1) In the input image, the house is crushed in the short-time exposure image, and the sky is white in the long-time exposure image.

  Further, (4) update area in the figure shows an update area and a non-update area of pixel values executed in (3) composite image generation processing. A white area is a pixel area to be updated, and a black area is a non-update area.

  An update area indicated by a white area corresponds to an effective pixel area included in the input image. Only the effective pixel area included in the input image is selected, and the synthesis process with the synthesized image generated in the previous frame is executed. Specifically, pixel value blending using only the effective pixel value setting area of these two images is executed, and high dynamic range processing and noise reduction processing are executed. (4) A black area shown in the update area is a non-update area, and the pixel value of the composite image generated in the previous frame is output as it is.

By such processing, a composite image obtained by performing both noise reduction and high dynamic range processing, that is, (3) composite image shown in the figure is sequentially generated and output.
As shown in FIG. 23, only the high dynamic range process for expanding the dynamic range is performed in the first cycle of exposure (first two frames), and the area corresponding to the input image from the second cycle (after the third frame). By applying noise removal processing to the dynamic range, integration of dynamic range expansion processing and noise removal processing is realized.

  In the first cycle of exposure, exposure correction corresponding to the exposure ratio is performed on each input image, and dynamic expansion processing is performed by combining exposure corrected images. Similarly, in the second and subsequent exposure cycles, the gains obtained by increasing the input images by the exposure ratio are used for synthesis. In the second and subsequent cycles, noise removal processing is simultaneously performed in addition to dynamic range expansion processing.

  The composite image that can be acquired after the second exposure cycle contains the components of each exposure image. Using this feature, noise removal processing is performed only in the effective area of each input image, and the pixel value of the synthesized image of the previous frame is applied as it is to other areas to generate a synthesized image.

  For example, when the third input image fn3 (short-time exposure image) from the left is input, the effective pixel region of the input image fn3, that is, the pixel value of the “empty region” other than the “home region” that is blacked out. Is used to perform the synthesizing process with the pixel value of the “empty area” of the synthesized image fn2 of the previous frame, and the noise removal process for only the “empty area” is performed.

When the fourth input image fn4 (long-time exposure image) from the left is input, the effective pixel area of this input image, that is, the pixels of the “house area” excluding the “empty area” that is overexposed. Using the value, a synthesis process with the pixel value of the “house region” of the synthesized image fn3 of the previous frame is executed, and a noise removal process for only the “house region” is performed.
By repeating these processes, integrated processing of dynamic range expansion processing and noise removal processing is realized.

Since the combined image of each frame in this integration process has a different area for noise removal depending on the exposure component of the input image, the dynamic range in the entire area of the image every time one set of exposure cycles is completed. The expansion process and the noise removal process are performed.
In the embodiment described below, the “combining process” is a combining process that realizes the above-described process, that is, a dynamic range expansion process and a noise removal process that are dynamic range expansion processes.

[2-3. Embodiment in which Noise Reduction and High Dynamic Range Image Generation According to the Present Disclosure are Performed Together (Embodiment 6)]
Hereinafter, an embodiment of an image processing apparatus that executes both noise reduction and high dynamic range image generation will be described as an embodiment 6 of the image processing apparatus according to the present disclosure.
FIG. 24 is a configuration diagram illustrating the main configuration of the image processing apparatus according to the sixth embodiment.

  As shown in FIG. 24, the image processing apparatus includes a frame memory 401, an exposure ratio correction unit 402, a motion estimation unit 403, a motion compensated image generation unit 404, a signal processing unit 405, and a frame memory 406.

The input image 411 is
(1) The shortest time exposure image XS,
(2) Short exposure image MS,
(3) Long exposure image ML,
(4) Longest time exposure image XL,
It becomes a periodic input of continuously shot images of these four different exposure times.
The exposure time has the following relationship.
XS <MS <ML <XL
It is in the above relationship.

These images are sequentially stored in the frame memory 401.
The exposure ratio correction unit 402 corrects the exposure ratio based on the exposure time information of the new input image and the input image of the previous frame held in the frame memory 401.

As shown in FIG. 24, the two images whose exposure ratio is corrected are input to the motion estimator 403 as the exposure ratio-corrected continuous captured two images 412.
The motion estimation unit 403 inputs the exposure ratio-corrected continuously captured two images 412 and estimates the amount of motion between the two images by block matching such as similarity determination in pixel block units of these two continuously captured images. A motion vector 413 indicating motion information between two images is calculated and output to the motion compensated image generation unit 404.

The motion compensated image generation unit 404 applies the motion vector 413 input from the motion estimation unit 403, moves the synthesis result image up to the previous frame held in the frame memory 406, and matches it to the image position of the input image. A motion compensated image is generated.
Thereby, the position of the synthesis result image up to the previous frame can be matched with the position of the input image.

The signal processing unit 405 acquires an input image from the frame memory 401, and inputs a motion compensation image based on the previous frame synthesis result image stored in the frame memory 406 from the motion compensation image generation unit 404.
The signal processing unit 405 executes synthesis processing based on these two images, that is, synthesis processing including both the high dynamic range and the noise reduction processing described above with reference to FIG. The synthesis result is output as a noise-removed HDR image 415 shown in the figure.
Note that this processing result image is stored in the frame memory 406 for application to the next processing.

In the processing described with reference to FIG. 24, an image alignment sequence in the case where the different exposure time is set to 4 will be described with reference to FIG.
FIG. 25 shows motion compensated image generation according to the sequence of input images along the time axis (t) indicating the passage of time from left to right and the motion vector executed in the motion compensated image generation unit 404 in FIG. That is, a combination of images in the alignment processing of two images is shown.

After performing exposure ratio correction based on the exposure time information of the input image and the input image one frame before, motion estimation is performed between the two images to calculate a motion vector. Using the motion vector obtained here, the composite result image up to the previous frame is aligned with the position of the input image.
By synthesizing the two images after the alignment, the dynamic range expansion process and the noise removal process are executed simultaneously. By this processing, an image with a high S / N ratio with reduced noise can be obtained by extending the dynamic range without including artifacts due to image misalignment.

[2-4. Embodiment in which Noise Reduction and High Dynamic Range Image Generation According to the Present Disclosure are Performed Together (Embodiment 7)]
Next, an image processing apparatus according to a seventh embodiment of the present disclosure will be described.
In the image processing apparatus according to the sixth embodiment described above, as understood from the sequence diagram shown in FIG. 25, image alignment is executed by applying images having different exposure times. When performing alignment over the period of the exposure image, motion estimation is performed between the longest exposure image and the shortest exposure image, and if the exposure ratio is large, there is a high possibility that the motion estimation will fail.
If the motion estimation fails, for example, the effect of noise removal by the filter processing using an IIR filter or the like may be interrupted, leading to image quality degradation such as image corruption.

A configuration for solving such a problem will be described below as a seventh embodiment.
An image alignment sequence according to the seventh embodiment will be described with reference to FIG.
26, as in FIG. 25, the sequence of input images along the time axis (t) indicating the passage of time from left to right, and the motion compensated image generation unit of the image processing apparatus of the present embodiment shown in FIG. A motion compensated image generation according to the motion vector executed in 404, that is, a combination of two images for alignment processing is shown.

In this embodiment, a specific exposure time image is selected as a reference image. For example, when there are N types of exposure time images, an image with an intermediate exposure time among these N types of exposure times is selected as the reference image.
In the example shown in FIG. 26, the long-exposure image ML is selected as a reference image, and this reference image is stored in the frame memory 401 shown in FIG. 27. When another exposure image is input, the input image and the reference image Motion estimation is performed between images and a motion vector is calculated. By using the motion vector obtained here, an alignment for matching the image position of the input image with the image position of the reference image, that is, a motion compensated image is generated.
The two images after the alignment are synthesized.

If an image having the same exposure time as the reference image stored in the frame memory 401 is input as a new input image, the stored image in the frame memory 401 is updated using the new input image as a new reference image. Then, the synthesis result image so far is synthesized according to the image position of the latest reference image.
Accordingly, it is possible to perform processing without estimating the motion between the longest time exposure image and the shortest time exposure image, and it is possible to prevent a motion estimation failure due to the magnitude of the exposure ratio. As a result, it is possible to repeatedly perform an image that is continuously exposed in the time direction without degrading the image quality when the dynamic range expansion process and the noise removal process are simultaneously combined.

  The alignment sequence shown in FIG. 26 is different from the sequence described above with reference to FIG. 25, that is, the alignment sequence in which the composite result image is always aligned with the position of the latest input image. A process of aligning the synthesized result image with the position of the input reference image is performed.

The configuration and processing of the image processing apparatus according to the seventh embodiment will be described with reference to FIG.
The image processing apparatus shown in FIG. 27 is similar to the image processing apparatus described with reference to FIG. 24, and includes a frame memory 401, an exposure ratio correction unit 402, a motion estimation unit 403, a motion compensated image generation unit 404, a signal processing unit 405, A frame memory 406 is included.

The input image 411 is
(1) The shortest time exposure image XS,
(2) Short exposure image MS,
(3) Long exposure image ML,
(4) Longest time exposure image XL,
It becomes a periodic input of continuously shot images of these four different exposure times.
The exposure time is set as follows. That is,
XS <MS <ML <XL
It is in the above relationship.

From these images, an image having a specific exposure time is stored in the frame memory 401 as a reference image.
In this example,
(1) The shortest time exposure image XS,
(2) Short exposure image MS,
(3) Long exposure image ML,
(4) Longest time exposure image XL,
Among these four types of exposure time images, “(3) long exposure image ML” is selected and stored in the frame memory 401 as a reference image.

The exposure ratio correction unit 402 corrects the exposure ratio based on the exposure time information of the new input image and the reference image held in the frame memory 401.
As shown in FIG. 27, the two images whose exposure ratio is corrected are input to the motion estimator 403 as the exposure ratio-corrected continuous captured two images 412.
The motion estimation unit 403 inputs the exposure ratio-corrected continuously captured two images 412 and estimates the amount of motion between the two images by block matching such as similarity determination in pixel block units of these two continuously captured images. A motion vector 413 indicating motion information between two images is calculated and output to the motion compensated image generation unit 404.

The motion compensation image generation unit 404 applies the motion vector 413 input from the motion estimation unit 403 to move the input image and generate a motion compensation image that matches the image position of the reference image stored in the frame memory 401. To do.
Thereby, the image position of the input image can be matched with the position of the reference image.

The signal processing unit 405 inputs a motion compensation image obtained by matching the image position of the input image with the position of the reference image from the motion compensation image generation unit 404, and further displays the synthesis result image of the previous frame stored in the frame memory 406. input.
The signal processing unit 405 executes synthesis processing based on these two images, that is, synthesis processing including both the high dynamic range and the noise reduction processing described above with reference to FIG. The synthesis result is output as a noise-removed HDR image 415 shown in the figure.
Note that this processing result image is stored in the frame memory 406 for application to the next processing.

  The difference between the image processing apparatus shown in FIG. 27 and the image processing apparatus described above with reference to FIG. 24 is the processing of the motion compensation image generation unit 404 and the signal processing unit 405. That is, the following points are different.

In the configuration of the sixth embodiment illustrated in FIG. 24, the motion compensation image generation unit 404 applies the motion vector 413 input from the motion estimation unit 403, and displays the synthesis result image up to the previous frame held in the frame memory 406. A motion compensated image that is moved to match the image position of the input image is generated.
On the other hand, in the configuration of the seventh embodiment illustrated in FIG. 27, the motion vector 413 input from the motion estimation unit 403 is applied to move the input image to match the image position of the reference image stored in the frame memory 401. A motion compensated image is generated.

In the configuration of the sixth embodiment illustrated in FIG. 24, the signal processing unit 405 acquires an input image from the frame memory 401, and the previous frame synthesis result image stored in the frame memory 406 from the motion compensation image generation unit 404. A motion compensated image based on the image is input, an image synthesis process based on these two images is executed, and output as a noise-removed HDR image 415.
On the other hand, in the configuration of the seventh embodiment illustrated in FIG. 27, the signal processing unit 405 inputs a motion compensation image obtained by matching the image position of the input image with the position of the reference image from the motion compensation image generation unit 404, and further the frame memory 406. The composite image of the previous frame stored in is input.
The signal processing unit 405 performs synthesis processing based on these two images and outputs the resultant as a noise-removed HDR image 415.

Next, the processing sequence of the image processing according to the seventh embodiment will be described with reference to the flowcharts shown in FIGS.
The processes shown in FIGS. 28 and 29 are performed under the control of the control unit of the image processing apparatus in the image processing apparatus having the configuration shown in FIG. 27, for example. The control unit executes a program stored in the memory of the image processing apparatus, and controls processing according to the flow of FIGS.

First, an image is input in step S301. For example, an image having a specific exposure time among consecutively photographed images having N different exposure times is input.
Next, in step S302, the reference image stored in the frame memory 401 is acquired, and in step S303, the previous frame composite image stored in the frame memory 406 is acquired.

Next, in step S304, the exposure time of the input image is compared with the exposure time of the reference image, and it is determined whether or not the exposure time of the input image is the same as the exposure time of the reference image.
If the exposure time of the input image is different from the exposure time of the reference image, the process proceeds to step S305, and it is further determined whether or not the exposure time of the input image is longer than the exposure time of the reference image.
If the exposure time of the input image is longer, exposure ratio correction is performed on the reference image in step S307.
On the other hand, if the exposure time of the reference image is longer, exposure ratio correction is performed on the input image in step S306.
These exposure ratio correction processes are processes executed in the exposure ratio correction unit 402 shown in FIG.
By this exposure ratio correction, the brightness of the input image and the reference image are made uniform.

Next, in step S321, motion estimation is performed between the reference image with uniform brightness and the input image, and a motion vector corresponding to the amount of movement between these two images is calculated.
This process is a process executed in the motion estimation unit 403 shown in FIG.

Further, in step S322, motion compensation is performed on the input image using the obtained motion vector to generate a motion compensated image.
This process is a process executed in the motion compensated image generation unit 404 shown in FIG.
By this motion compensation, the input image is adjusted to the position of the reference image.
In step S323, a synthesized image is generated by synthesizing the input image subjected to motion compensation and the previous frame synthesized image.
This process is a process executed by the signal processing unit 405 shown in FIG.
In step S324, the composite image generated by the above processing is stored in the frame memory and stepped.

On the other hand, if it is determined in step S304 that the exposure time of the input image and the exposure time of the reference image are the same, in step S331, motion estimation is performed between the two images to indicate the motion between the two images. A motion vector is calculated.
This process is a process executed in the motion estimation unit 403 shown in FIG.

Next, in step S332, using the obtained motion vector, motion compensation is performed on the previous frame composite image to generate a motion compensated image.
This process is a process executed in the motion compensated image generation unit 404 shown in FIG.

Further, in step S333, a synthesized image is generated by synthesizing the motion compensated image generated based on the previous frame synthesized image and the input image.
This process is a process executed by the signal processing unit 405 shown in FIG.
In step S334, the input image is stored in the frame memory.

  After the process of step S323 or step S334, the process proceeds to step S324. In step S324, the composite image generated by the above process is stored in the frame memory, and the composite image generated in step S325 is output.

[2-5. Embodiment in which motion estimation is performed between images having the same exposure time (Embodiment 8)]
Next, as an eighth embodiment of the image processing apparatus according to the present disclosure, an embodiment in which motion estimation is performed between images having the same exposure time will be described.
FIG. 30 is a diagram illustrating the configuration of the image processing apparatus according to the eighth embodiment.
The input image 511 is a periodic input of continuously shot images with N different exposure times.
These N kinds of different exposure images are stored for each image in each frame memory 1 to N of the frame memory group 501 composed of N frame memories.
That is, all the images for one set of the period of the exposure image are held in the frame memory.

When an image of the next period is input, the motion estimation unit 502 performs motion estimation between the input image included in the N frame memories and the image having the same exposure time, and moves between the same exposure images. The vector 512 is calculated and output to the motion vector calculation unit 503.
The motion vector calculation unit 503 inputs the motion vector 512 between the same exposure images from the motion estimation unit 502, and indicates the motion amount between two consecutively shot images with different exposure times based on the motion vector 512 between the same exposure images. A motion vector 513 between continuously shot images is calculated and output to the motion compensated image refiner 504. The motion vector calculation unit 512 calculates a motion vector to be applied to generation of a motion compensated image in the motion compensated image generation unit 504, for example, by linear calculation on the motion vector calculated by the motion estimation unit 502.

A motion vector 512 between the same exposure images calculated by the motion estimation unit 502;
The relationship with the continuous captured image motion vector 513 calculated by the motion vector calculation unit 503 will be described with reference to FIG.
As shown in FIG. 31, the motion vector 512 between the same exposure images calculated by the motion estimation unit 502 can be considered as a set of motion vectors 513 between continuously shot images calculated by the motion vector calculation unit 503.

  That is, by performing a linear operation on motion vectors between images having the same exposure time obtained by motion estimation, motion vectors between images having different exposure times included in the vector can be obtained.

  For vector calculation executed by the motion vector calculation unit 503, a method of dividing using the number of exposures or an exposure ratio can be applied. In addition, if the obtained motion vector is retained, a method of calculating using past vector information can be applied, and the motion between images with different exposure times can be applied from the motion vector obtained with the same exposure time. The method for calculating the vector is not particularly limited.

The processing sequence will be described with reference to the configuration of the image processing apparatus shown in FIG.
A process when the (N + 1) th image (shortest time exposure image) for which alignment is first performed after the start of imaging is input will be described.

  First, the motion estimator 502 performs motion estimation between the shortest time exposure image that is an input image and the stored image of the frame memory 1 that holds the first input image (shortest time exposure image). An inter-exposure image motion vector 512 is calculated and output to the motion vector calculation unit 503.

  The motion vector calculation unit 503 receives the motion vector 512 between the same exposure images, and obtains a motion vector between continuously shot images stored in the frame memory 1 and the frame memory 2, that is, between images having different exposures. The motion vector obtained here is output to the motion compensated image generation unit 504 as a motion vector 513 between continuously shot images.

The motion compensated image generation unit 504 generates a motion compensated image 514 in which the image position of the stored image in the frame memory 1 is matched with the image position of the stored image in the frame memory 2.
The motion compensation image generation unit 504 outputs the generated motion compensation image 514 to the signal processing unit 505.

The signal processing unit 505 performs synthesis processing of the image stored in the frame memory 2 and the motion compensation image 514 generated by the motion compensation image generation unit 504 to generate and output a noise-removed HDR image 515. The output image is stored in the frame memory 505.
Hereinafter, the same processing is repeatedly executed between each image having N types of exposure times, that is, an input image having the same exposure time and an image stored in the frame memory.

FIG. 32 is a diagram for describing processing for images that are continuously input when the N + 2nd and subsequent images are input, that is, after the processing described with reference to FIG. 30.
A process when the (N + 2) th image (second short-exposure image) is input after the image capturing is started will be described.

  First, the motion estimation unit 502 estimates motion between the second short-exposure image that is an input image and the stored image in the frame memory 2 that holds the same exposure time image (second short-exposure image). The same exposure image motion vector 512 is calculated and output to the motion vector calculation unit 503.

  The motion vector calculation unit 503 receives the motion vector 512 between the same exposure images, and obtains a motion vector between the continuously shot images stored in the frame memory 2 and the frame memory 3, that is, between images with different exposures. The motion vector obtained here is output to the motion compensated image generation unit 504 as a motion vector 513 between continuously shot images.

The motion compensated image generation unit 504 applies the motion vector 513 to generate a motion compensated image 514 that matches the image position of the stored image in the frame memory 506 with the image position of the stored image in the frame memory 3.
The motion compensation image generation unit 504 outputs the generated motion compensation image 514 to the signal processing unit 505.

The signal processing unit 505 performs synthesis processing of the image stored in the frame memory 3 and the motion compensation image 514 generated by the motion compensation image generation unit 504 to generate and output a noise-removed HDR image 515. The output image is stored in the frame memory 505.
Hereinafter, the same processing is repeatedly executed between each image having N types of exposure times, that is, between an input image having the same exposure time and a frame memory stored image.
With these processes, it is possible to obtain a high S / N image with an expanded dynamic range without including artifacts due to image misregistration regardless of the exposure ratio between images for which motion estimation is performed.

A processing sequence of the image processing apparatus according to the present embodiment will be described with reference to a flowchart shown in FIG.
The process shown in FIG. 33 is performed under the control of the control unit of the image processing apparatus in the image processing apparatus having the configuration shown in FIGS. 30 and 32, for example. The control unit executes a program stored in the memory of the image processing apparatus to control processing according to the flow of FIG.

First, an image is input in step S401. For example, an image having a specific exposure time among consecutively photographed images having N different exposure times is input.
Next, in step S402, it is determined whether or not the input image is the first cycle of an exposure image set of N different exposure times that are continuously captured.
If it is the first cycle, the process proceeds to step S421, the acquired input image is stored in the frame memory, and the process returns to the process of acquiring the input image of the next frame.
That is, until the input image enters the second period of the exposure set, the process of holding the input images captured at different exposure times in the frame memory is repeated.

When the input image is in the second cycle or later, the past input image captured with the same exposure time as the input image is already stored in the frame memory.
When a new image is input in such a state, the processing from step S403 is started.

In step S403, a past input image (hereinafter referred to as the same exposure past input image) captured with the same exposure time as the input image is acquired from the frame memory.
Next, in step S404, motion estimation is performed between the acquired past exposure past input image and the input image.
Next, in step S405, a motion vector for one exposure set period, that is, a motion vector between the same exposure images is calculated.
This process is a process executed by the motion estimation unit 502 shown in FIGS.
In step S406, the input image is stored in the frame memory.

In step S407, based on the same exposure image motion vector, a continuous captured image motion vector is calculated. With this motion vector calculation, a motion vector between input images having different exposure times (hereinafter referred to as different exposure past input images) is obtained.
This process is a process executed by the motion vector calculation unit 503 shown in FIGS.

Next, in step S408, the previous frame composite image is acquired, and in step S409, the different exposure past input image is acquired from the frame memory.
Next, in step S410, motion compensation is performed on the previous frame composite image acquired from the frame memory using the motion vector between consecutive captured images calculated in step S407, thereby generating a motion compensated image.
This process is a process executed by the motion compensated image generation unit 504 shown in FIGS. 30 and 32.
Since the previous frame composite image is at the same position as the same exposure past input image, the previous frame composite image is aligned with the position of the different exposure past input image by this motion compensation.

Next, in step S411, the motion-compensated previous frame composite image and the different exposure past input image are combined to generate a composite image.
This process is a process executed by the signal processing unit 505 shown in FIGS.
In step S412, the composite image generated by the above processing is stored in the frame memory, and the composite image generated in step S413 is output.

For example, the following effects are brought about by the processing of the image processing apparatus according to the sixth and seventh embodiments, that is, the noise reduction by combining a plurality of images and the high dynamic range processing.
Regardless of the exposure ratio between the shortest time exposure image and the longest time exposure image, it is possible to sequentially perform dynamic range expansion processing and noise removal processing without including artifacts due to motion shift. As a result, failure in motion estimation can be avoided, and dynamic range expansion processing and noise removal processing can be integrated without degrading image quality.

  In addition, by performing motion estimation only between images having the same exposure time, failure in motion estimation can be avoided regardless of the exposure ratio between consecutive images. By calculating the obtained motion vector information, the motion vector information necessary for each of the dynamic range expansion processing and noise removal processing can be realized with one motion estimation block / motion compensation block, that is, with a small number of circuits. It is possible to integrate dynamic range expansion processing and noise removal processing with scale and power consumption.

[3. Example of overall configuration of image processing apparatus]
Finally, an overall configuration example of an image processing apparatus that executes processing according to the above-described embodiments will be described.
FIG. 34 is a diagram illustrating a configuration example of an imaging apparatus 600 as an embodiment of the image processing apparatus of the present disclosure. Light incident through the optical lens 601 enters an imaging device 602 including an imaging unit, for example, a CMOS image sensor, and outputs image data by photoelectric conversion. The output image data is input to the image processing unit 603.

  The image processing unit 603 executes a process according to each of the above-described embodiments, that is, an output image generation process involving a combination process of a plurality of images. Furthermore, the image processing unit 603 also performs general camera signal processing on the captured data, for example, signal processing such as white balance (WB) adjustment and gamma correction, and generates an output image 620. The output image 620 is stored in a storage unit (not shown). Or it outputs to a display part.

  The control unit 605 outputs a control signal to each unit according to a program stored in the memory 606, for example, and controls various processes.

[4. Summary of composition of the present disclosure]
As described above, the embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present disclosure, the claims should be taken into consideration.

The technology disclosed in this specification can take the following configurations.
(1) a motion estimation unit that estimates a motion amount between two images of a first image and a second image that are captured images at different timings;
A motion compensated image generating unit that generates a motion compensated image by executing a motion compensation process for aligning the second image with the position of the first image based on the motion amount estimated by the motion estimator;
A signal processing unit that generates a synthesized image by synthesizing the motion compensation image and the first image;
The motion compensation image generation unit is an image processing device that generates a motion compensation image using a selected image selected according to a predetermined image selection criterion.

  (2) The image processing apparatus includes a motion estimation suitability determination unit that determines whether or not the input image is a qualified image suitable for motion estimation, and the motion estimation suitability determination unit includes: Only when it is a qualified image, control is performed to store the input image in a frame memory as a reference image to be applied to the motion estimation, and the motion estimation unit applies a motion to which the reference image stored in the frame memory is applied. The image processing apparatus according to (1), wherein estimation is performed, and the motion compensation image generation unit generates a motion compensation image using the reference image.

(3) The motion estimation unit inputs determination information indicating whether or not the input image is a qualified image as determination information of the motion estimation appropriateness determination unit, and based on the determination information, the input image The image processing apparatus according to (2), wherein when the image is determined to be a qualified image, motion estimation is performed by applying the input image and the reference image.
(4) The motion estimation unit inputs determination information indicating whether or not the input image is a qualified image as determination information of the motion estimation appropriateness determination unit, and based on the determination information, the input image Is determined not to be suitable for motion estimation, the motion estimation using the input image and the reference image is not performed, and the motion estimation is not performed in the motion estimation unit. The image processing apparatus according to (3), further including a motion amount correction unit that outputs an exceptional motion amount determined according to an algorithm to the motion compensated image generation unit.

(5) The signal processing unit inputs determination information indicating whether or not the input image is a qualified image as determination information of the motion estimation appropriateness determination unit, and based on the determination information, the input image The image processing apparatus according to any one of (2) to (4), wherein the signal processing is performed on the image.
(6) The motion estimation suitability determination unit may determine at least one of a blurred area, a saturated pixel area, and a noisy area in the input image, a camera movement at the time of capturing the input image, or a luminance range of the input image. The image processing apparatus according to any one of (2) to (5), wherein any one of the indices is applied to execute a determination process as to whether or not the input image is an image suitable for motion estimation.

(7) When the continuous estimation image set to different exposure time is input as the input image, the motion estimation appropriateness determination unit compares the effective pixel value areas of the exposure images to obtain a wider effective pixel area. The image processing apparatus according to any one of (2) to (6), wherein control is performed so that an image having the reference image stored in the frame memory is a reference image.
(8) The image processing apparatus further executes a motion estimation direction switching process executed to match the image position of the input image with the image position of the reference image using the image stored in the frame memory as the reference image. The image processing device according to any one of (2) to (7), further including: a motion estimation direction switching unit that performs the operation.

  (9) The image processing apparatus further includes an exposure ratio correction unit that inputs continuous shot images set at different exposure times as the input image and executes exposure ratio correction, and the motion estimation unit includes: The exposure ratio correction unit performs motion estimation between continuously shot images subjected to exposure ratio correction, calculates a motion amount between the continuously shot images, and the motion compensated image generation unit uses the motion amount. Then, a motion compensated image in which one image position of the continuously shot image is matched with the other image position is generated, and the signal processing unit combines the motion compensated image and a reference image stored in a frame memory. The image processing apparatus according to any one of (1) to (8), wherein:

(10) The reference image stored in the frame memory is an image having a predetermined exposure time, and the exposure ratio correction unit corrects the exposure ratio between the input image and the reference image stored in the frame memory. The image processing apparatus according to (9), wherein the motion estimation unit executes motion estimation between an input image that has been subjected to exposure ratio correction and a reference image in the exposure ratio correction unit.
(11) The motion estimation unit calculates a motion vector between images set at the same exposure time, inputs the motion vector calculated by the motion estimation unit, and performs linear computation on the motion vector to perform the motion compensation image. The image processing apparatus according to (9) or (10), further including a motion vector calculation unit that calculates a motion vector to be applied to generation of a motion compensated image in the generation unit.

  Furthermore, the configuration of the present disclosure includes a method of processing executed in the above-described apparatus and system, and a program for executing the processing.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and can be installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

As described above, according to the configuration of an embodiment of the present disclosure, high-quality noise reduction, a high dynamic range image, and the like are generated by image synthesis processing using a motion-compensated image based on highly accurate motion estimation. The configuration to be realized is realized.
Specifically, the amount of motion between the images is estimated, a motion compensation image is generated by matching one image with the position of the other image, and a motion compensated image is generated. An image subjected to noise reduction or the like is generated by the synthesis process. The motion compensated image generation unit generates a motion compensated image using a selected image selected according to a predetermined image selection criterion. For example, a motion estimation suitability determination unit that determines whether or not an input image is a qualified image suitable for motion estimation, and applies the input image to motion estimation only when the input image is a qualified image It is stored in the frame memory as a reference image.
With these processes, high-quality noise reduction and a high dynamic range image can be generated by image synthesis processing using a motion compensated image based on highly accurate motion estimation.

DESCRIPTION OF SYMBOLS 111 Still region detection part 112 Pixel value correction | amendment part 201 Frame memory 202 Motion estimation suitable degree determination part 203 Motion estimation part 204 Motion amount correction | amendment part 205 Motion compensation image generation part 206 Signal processing part 207 Frame memory stored image update control switch 251 Motion estimation Direction switching unit 331 to 334 Frame memory 335 High dynamic range image generation unit 336 Noise removal unit 337 Frame memory 341 Noise removal unit 351 to 354 Frame memory 355 High dynamic range image generation unit 401 Frame memory 402 Exposure ratio correction unit 403 Motion estimation unit 404 Motion Compensation Image Generation Unit 405 Signal Processing Unit 406 Frame Memory 501 Frame Memory Group 502 Motion Estimation Unit 503 Motion Vector Calculation Unit 504 Motion Compensation Image Generation Unit 505 Signal Processing Unit 506 Frame memory 600 Imaging device 601 Optical lens 602 Imaging device 603 Image processing unit 605 Control unit 606 Memory

Claims (13)

A motion estimator that estimates the amount of motion between two images of a first image and a second image that are captured images at different timings;
A motion compensated image generating unit that generates a motion compensated image by executing a motion compensation process for aligning the second image with the position of the first image based on the motion amount estimated by the motion estimator;
A signal processing unit that generates a synthesized image by synthesizing the motion compensation image and the first image;
The motion compensation image generation unit is an image processing device that generates a motion compensation image using a selected image selected according to a predetermined image selection criterion. The image processing apparatus includes:
A motion estimation suitability determination unit that determines whether the input image is a suitable image suitable for motion estimation;
The motion estimation suitability determination unit executes control to store the input image in a frame memory as a reference image to be applied to the motion estimation only when the input image is a qualified image,
The motion estimation unit performs motion estimation using a reference image stored in the frame memory;
The image processing device according to claim 1, wherein the motion compensation image generation unit generates a motion compensation image using the reference image.   The motion estimation unit inputs determination information indicating whether or not the input image is a qualified image as the determination information of the motion estimation suitability determination unit, and based on the determination information, the input image is a qualified image The image processing apparatus according to claim 2, wherein when it is determined that the input image and the reference image are applied, motion estimation is performed. The motion estimation unit inputs determination information indicating whether or not the input image is a qualified image as determination information of the motion estimation suitability determination unit, and the input image is estimated based on the determination information. If it is determined that the image is not suitable for the image, motion estimation is not performed by applying the input image and the reference image.
The image processing apparatus according to claim 3, further comprising a motion amount correction unit that outputs an exceptional motion amount determined according to a predetermined algorithm to the motion compensated image generation unit when motion estimation is not performed in the motion estimation unit.   The signal processing unit inputs determination information indicating whether or not the input image is a qualified image as determination information of the motion estimation appropriateness determination unit, and performs signal processing on the input image based on the determination information The image processing apparatus according to claim 2, wherein: The motion estimation suitability determination unit
The input image moves by applying at least one of a blur area, a saturated pixel area, a noisy area in the input image, a camera movement at the time of shooting the input image, or a luminance range of the input image. The image processing apparatus according to claim 2, wherein a process for determining whether the image is suitable for estimation is executed. The motion estimation suitability determination unit
When inputting continuously shot images set at different exposure times as the input image, a reference image for storing an image having a wider effective pixel area in the frame memory by comparing effective pixel value areas of the respective exposure images; The image processing apparatus according to claim 2, wherein control is performed. The image processing apparatus further includes:
The image processing apparatus includes a motion estimation direction switching unit that executes a motion estimation direction switching process that is executed to match the image position of the input image with the image position of the reference image using the image stored in the frame memory as a reference image. An image processing apparatus according to 1. The image processing apparatus further includes:
As the input image, it has an exposure ratio correction unit that inputs continuous shot images set at different exposure times and executes exposure ratio correction,
The motion estimation unit performs motion estimation between continuously shot images subjected to exposure ratio correction in the exposure ratio correction unit, calculates a motion amount between the continuously shot images,
The motion compensated image generating unit generates a motion compensated image in which one image position of the continuously shot image is matched with the other image position using the amount of motion.
The image processing apparatus according to claim 1, wherein the signal processing unit executes a synthesis process of the motion compensation image and a reference image stored in a frame memory. The reference image stored in the frame memory is an image having a predetermined exposure time,
The exposure ratio correction unit performs exposure ratio correction between the input image and a reference image stored in the frame memory,
The image processing apparatus according to claim 9, wherein the motion estimation unit performs motion estimation between an input image and a reference image that have been subjected to exposure ratio correction in the exposure ratio correction unit. The motion estimation unit calculates a motion vector between images set at the same exposure time,
10. A motion vector calculation unit that inputs a motion vector calculated by the motion estimation unit and calculates a motion vector to be applied to generation of a motion compensated image in the motion compensated image generation unit by linear calculation on the motion vector. An image processing apparatus according to 1. An image processing method executed in an image processing apparatus,
A motion estimation step in which a motion estimation unit estimates a motion amount between two images of a first image and a second image that are captured images at different timings;
A motion compensated image generating step for generating a motion compensated image by executing a motion compensation process for aligning the second image with the position of the first image based on the motion amount estimated in the motion estimating step. When,
A signal processing unit that performs a signal processing step of generating a synthesized image by synthesizing the motion compensated image and the first image;
The motion compensation image generation step is an image processing method for generating a motion compensation image using a selected image selected according to a predetermined image selection criterion. A program for executing image processing in an image processing apparatus;
A motion estimation step for causing the motion estimation unit to estimate a motion amount between the first image and the second image that are captured images at different timings;
A motion compensation image generation step for causing the motion compensation image generation unit to generate a motion compensation image by executing a motion compensation process for aligning the second image with the position of the first image based on the motion amount estimated in the motion estimation step. When,
Performing a signal processing step of causing a signal processing unit to generate a composite image by combining the motion compensated image and the first image;
In the motion compensation image generation step, a program for generating a motion compensation image using a selected image selected according to a predetermined image selection criterion.