WO2016147957A1

WO2016147957A1 - Imaging device and imaging method

Info

Publication number: WO2016147957A1
Application number: PCT/JP2016/057191
Authority: WO
Inventors: 啓志宮城; 村松　功一
Original assignee: リコーイメージング株式会社
Priority date: 2015-03-13
Filing date: 2016-03-08
Publication date: 2016-09-22
Also published as: JP2016171511A; JP6606838B2

Abstract

An imaging device according to the present invention is provided with: a control unit for controlling acquisition of a plurality of images while changing the relative position between a subject image and an imaging element; an image processing unit for performing image processing on the basis of the acquired plurality of images; and a detection unit for detecting differences of the subject image in at least parts of at least two of the acquired images. According to differences of the subject image, the control unit performs reacquisition of images or notification of the differences of the subject image.

Description

Imaging apparatus and imaging method

The present invention relates to image processing based on a plurality of images.

In video cameras and television systems, image processing (so-called super-resolution processing) is performed based on multiple images, so that high-resolution images that exceed the resolution (number of pixels) of the image sensor and reproducibility of hues, etc. However, it is possible to obtain an excellent high-quality image. In this case, shooting is performed while the image sensor is moved relative to the subject image under a condition in which the positional relationship between the subject and the camera or the state of the subject is fixed. Then, a plurality of images are synthesized by performing sampling point interpolation processing, color information interpolation, and the like on a plurality of images having different relative positions between the subject image and the pixel position. Thereby, a high-quality image can be obtained for the number of pixels of the image sensor.

When shooting multiple images, if there is a change (motion) in the positional relationship between the subject and the camera or the state of the subject, the motion must be estimated and corrected before being combined. Therefore, in a television system or the like, if there is an erroneous motion estimation, the generated composite image is disturbed. In order to prevent this, for example, whether or not there is an abnormal motion vector between the previous and next frames is detected for each frame. Image processing is not executed (see Patent Document 1).

JP 2009-140393 A

However, it is not possible to deal with various shooting situations only by correcting the super-resolution processing by motion vector detection. For example, when super-resolution processing is performed at the time of shooting, the photographer cannot determine whether or not the super-image processing has succeeded unless the generated image is viewed in detail. It has been done. In addition, a situation occurs in which the photographer notices that the photographer has failed in the editing work without noticing the failure on the spot. Therefore, it is necessary to improve such inconvenience in the super-resolution processing.

An imaging apparatus of the present invention includes a control unit that controls acquisition of a plurality of images while changing a relative position between a subject image and an imaging element, an image processing unit that performs image processing based on the acquired plurality of images, A detection unit that detects a difference in the subject image in at least a part between at least two acquired images, and the control unit acquires the image again or notifies the difference in the subject image according to the difference in the subject image. I do. For shooting to acquire a plurality of images, the image sensor can be shifted with respect to the subject image, or vice versa.

Such an imaging apparatus can be applied to, for example, a camera capable of performing a shooting operation for generating a high-quality image such as super-resolution shooting, and can improve resolution (number of pixels) and accurately. For a plurality of image acquisitions related to image quality improvement such as color interpolation, it is possible to detect a difference in the subject image, acquire an image again, and notify the difference. That is, if there is no difference in subject images, the present invention can be applied to shooting that can generate one image as it is from a plurality of images.

For example, the control unit can notify the difference in the subject image by displaying an image in which the portion in which the subject image difference occurs is identified. In addition, a light emitting unit may be provided, and the control unit may cause the light emitting unit to emit light so as to notify the difference between the subject images.

Here, the “subject image difference” is caused by a change including subject movement, brightness, hue, and the like. If there is a difference in at least part of the subject image between a plurality of images, the subject image has a difference. Suppose it has occurred. For example, the detection unit detects at least one of the movement of the subject image, the change in the brightness of the subject image, and the change in the color of the subject image as a difference between the subject images. “Re-acquisition of image” is based on image processing, such as re-shooting (exposure to image sensor and acquisition of image (data)) so as to obtain a plurality of images with no difference in subject images. The image acquisition operation is controlled to complete the acquisition of a plurality of images.

The detecting unit can detect a difference between subject images based on a difference between at least two images. For example, the detection unit obtains a difference in luminance or color information between corresponding pixels between at least two images, and detects a difference in subject images based on the obtained difference. The detection unit can obtain a difference between luminance histograms or color information histograms of at least two images, and can detect a difference between subject images based on the obtained difference. Further, the detection unit can obtain a difference between an average value of luminance or an average value of color information of at least two images and detect a difference between subject images based on the obtained difference.

The detection unit can detect at least one of a change in the position of the subject image between at least two images, a change in the brightness of the subject image, and a change in the color of the subject image as a difference between the subject images. It is. For example, the detection unit detects the difference between the subject images before completion of all the acquisition of the plurality of images based on the image processing, and when the difference between the subject images is detected, the control unit detects the plurality of images from the beginning. Re-acquisition may be executed.

Alternatively, when the difference between the subject images is detected, the control unit can execute re-acquisition from the last acquired image among the plurality of already acquired images. On the other hand, the detection unit detects the difference between the subject image of the reference image and the subject image of the other image after completing the acquisition of the plurality of images based on the image processing, and detects the difference between the subject images. It is also possible to execute re-acquisition for the image generated at that time.

The detection unit detects a difference between the subject images before completion of all acquisition of the plurality of images based on the image processing or after completion of all acquisition of the plurality of images based on the image processing, and the control unit detects the difference between the subject images. When detected, it is also possible to execute storage of at least one of the already acquired images and display of at least one of the already acquired images. The detection unit can also detect a difference in the subject image after image processing based on a plurality of images, and the control unit can notify the difference in the subject image when the difference in the subject image is detected.

On the other hand, it is also possible to detect a difference in the subject image even before shooting (before operating the release button) such as a through image display period. For example, before starting the acquisition of a plurality of images, the difference between the subject images is detected with a pre-shooting detection unit that detects a difference in the subject images between the plurality of images obtained by the image sensor while changing the relative positions. And a pre-shooting control unit for notifying the difference between the subject images. The pre-shooting detection unit and the detection unit, and the pre-shooting control unit and the control unit may be separate, or substantially the same, or may be included in the detection unit and the control unit. Such a configuration related to photographing can be configured regardless of the above-described acquisition of a plurality of images, difference detection, image re-acquisition, or notification.

An imaging method according to another aspect of the present invention executes acquisition of a plurality of images while changing a relative position between a subject image and an imaging device, and executes image processing based on the acquired plurality of images to perform another processing. An image is generated, a difference in at least a part of the subject image between at least two acquired images is detected, and an image acquisition or notification of the difference in the subject image is performed again according to the difference in the subject image.

According to another aspect of the present invention, a program recorded on a computer-readable medium is acquired by a control unit that controls acquisition of a plurality of images while changing a relative position between a subject image and an imaging element. An image processing unit that executes image processing based on the plurality of images and generates another image, and a detection unit that detects a difference in at least a part of the subject image between the acquired two images. Then, according to the difference between the subject images, the control unit is caused to function so as to acquire an image again or notify the difference between the subject images.

An image processing apparatus according to another aspect of the present invention performs image processing based on a plurality of images obtained by acquiring a plurality of images while changing a relative position between a subject image and an imaging element, An image processing unit that generates a difference, a detection unit that detects a difference in at least a portion of the subject image between at least two images, and a notification unit that notifies the difference in the subject image when a difference in the subject image is detected With.

An imaging apparatus according to another aspect of the present invention includes a control unit that controls acquisition of a plurality of images while changing a relative position between a subject image and an imaging element, and image processing that performs image processing based on the acquired plurality of images. And a detection unit that detects a difference in at least a part of the subject image between at least two images, and the control unit changes the number of acquisition times of the plurality of images according to the difference in the subject image.

For example, when a difference between subject images is detected, the number of acquisitions is increased by acquiring images again so that a plurality of images without differences are obtained. Alternatively, when it is impossible to finally generate one image due to the difference in the subject image, it is possible to end the image acquisition halfway and reduce the number of acquisitions.

According to the present invention, image generation failure in image processing based on a plurality of images can be prevented.

It is a block diagram of the digital camera which is 1st Embodiment. It is a schematic perspective view of an image blur correction mechanism. It is the figure which showed the flow of the super-resolution process. It is the figure which showed the shift position of the image sensor at the time of super-resolution imaging | photography. It is the figure which showed the high quality image partially. It is the figure which showed the difference of a pixel value It is the figure which showed the difference of the average value of the whole image. It is the figure which showed the histogram difference of the whole image. 5 is a flowchart of super-resolution imaging and super-resolution processing in the present embodiment. 10 is a flowchart of super-resolution imaging and super-resolution processing in the second embodiment. 10 is a flowchart of super-resolution imaging and super-resolution processing in the third embodiment. 14 is a flowchart of super-resolution imaging and processing in the fourth embodiment. 10 is a flowchart of super-resolution imaging and processing in the fifth embodiment. It is the figure which showed the display screen which notifies a to-be-photographed image change. 14 is a flowchart of super-resolution imaging and processing in the sixth embodiment.

Hereinafter, the digital camera according to the present embodiment will be described with reference to the drawings.

FIG. 1 is a block diagram of a digital camera according to the first embodiment. FIG. 2 is a schematic perspective view of the image blur correction mechanism.

The digital camera 10 includes a camera body 20 and a photographic lens 30 (lens barrel) that can be attached to and detached from the camera body 20, and an LCD 24 is provided on the back of the camera body 20 as a display device. Further, in accordance with an input operation to the release button 15 or a mode dial, a cross button, an execution button (none of which are shown), a shooting operation, a mode setting, and the like are performed.

A controller 40 constituted by a CPU, a DSP, etc., outputs a control signal to the aperture / shutter drive circuit 23, the image sensor drive circuit 60, etc., and a signal sent from the photographing operation switch 26 etc. that detects a release button operation or the like. Accordingly, operation control of the entire camera such as exposure control, shooting / recording operation, playback operation, function setting processing is performed. A camera operation control program is stored in advance in a recording medium such as a ROM (not shown).

The communication memory 33 stores data such as lens resolving power and is read out by the controller 40 when the photographing lens 30 is attached. The light emitting unit 90 can emit light in a predetermined shooting operation mode, and notifies the user and the like of shooting conditions.

The imaging optical system 31 including a focusing lens or the like causes light to be imaged on the image sensor 22 when light from the subject is incident, thereby forming a subject image on the image sensor 22. The image sensor 22 can be constituted by a CCD, a CMOS, or the like, and a color filter array (not shown) is disposed oppositely on the light receiving surface of the image sensor 22. Here, the color filter array has a configuration in which R, G, and B color filter elements are arranged (for example, a Bayer arrangement).

In the shooting mode, while displaying a through image (preceding image) on the LCD 24, pixel signals for one field or one frame are sequentially read out from the image sensor 22 at a predetermined time interval (1/60 seconds or the like). The read R, G, and B pixel signals are sent to the controller 40 via an AFE circuit (not shown).

The controller 40 performs gain processing, color conversion processing, white balance adjustment, and the like on the input series of pixel signals, and generates R, G, and B color image signals. Then, the controller 40 drives and controls the LCD 24 based on the generated R, G, B color image signals, whereby a real-time moving image is displayed on the LCD 24.

When the release button 15 is half-pressed, the AF driving circuit 62 drives the focusing lens based on the pixel signal read from the image sensor 22 according to the contrast method. Further, the controller 40 detects the brightness of the subject image based on the pixel signal read from the image sensor 22 and calculates the exposure value.

When the release button 15 is fully pressed, the drive control unit 41 of the controller 40 outputs a control signal to the aperture / shutter drive circuit 23 to perform exposure control. Thereby, a pixel signal for one frame is read from the image sensor 22. In the controller 40, still image data is generated based on the read pixel signal for one frame. Then, compressed image data or uncompressed image data is acquired and recorded in the image memory 25.

The image sensor 22 is mounted on the image blur correction mechanism 50 and is relatively movable along the x and y axis directions orthogonal to the optical axis (z axis) of the photographing optical system 31. As shown in FIG. 2, the image blur correction mechanism 50 includes a plate-like support portion 51 fixed to the camera body 20 and a movable stage 52 that can move relative to the support portion 51. The magnets M1, M2, and M3 are arranged to face the movable stage 52. On the movable stage 52, the yokes Y1, Y2, and Y3 are disposed so as to face the magnets M1, M2, and M3, respectively.

The movable stage 52 is provided with driving coils C1, C2, and C3. By passing an AC driving signal through the driving coils C1, C2, and C3 in the magnetic field, the movable stage 52 is moved along the plane orthogonal to the optical axis. It moves relative to the support portion 51. The drive control unit 41 of the controller 40 controls the drive signal generation unit 61 of the image sensor drive circuit 60, and thereby an AC drive signal is output from the drive signal generation unit 61.

Hall sensors H1, H2, and H3 are disposed in the vicinity of the driving coils C1, C2, and C3 of the support 51, and the hall sensors H1, H2, and H3 are movable by detecting the magnetic force of the magnets M1, M2, and M3. The position of the stage 52 (image sensor 22) in the plane orthogonal to the optical axis is detected. Further, the gyro sensor 28 provided in the camera body 20 detects a shake of the camera body 20, that is, a change in posture based on an inclination angle or an angular velocity.

The controller 40 drives and controls the image blur correction mechanism 50 along the plane orthogonal to the optical axis according to the shake of the camera body 20 detected by the gyro sensor 28 and the position of the image sensor 22 by the hall sensors H1, H2, and H3. The imaging area of the subject image is shifted so as to cancel the image displacement. By this camera shake correction operation, it is possible to obtain a subject image without image blurring.

In this embodiment, it is possible to acquire an image with higher image quality (hereinafter referred to as a high-quality image) using the image blur correction mechanism 50. Specifically, a plurality of shootings (hereinafter referred to as super-resolution shooting) are performed while shifting the shooting target area. At this time, the user shoots with the camera 10 fixed, such as by setting the camera 10 on a tripod. Then, image processing (synthesizing processing) is executed based on the obtained plurality of images (hereinafter referred to as captured images), and the obtained images are recorded / displayed.

In the case of super-resolution shooting, four images are acquired sequentially here. The super-resolution operation switch 27 detects ON / OFF of the super-resolution processing and the user's setting operation regarding the super-resolution processing.

Hereinafter, super-resolution imaging will be described with reference to FIGS.

FIG. 3 is a diagram showing a flow of super-resolution processing. FIG. 4 is a diagram illustrating the shift position of the image sensor during super-resolution imaging. FIG. 5 is a diagram partially showing a high-quality image. However, the flow of FIG. 3 is a flow for explaining the super-resolution processing, and in this embodiment, the operation of the camera 10 is controlled by the flow described later.

When the release button 15 is pressed and super-resolution imaging is started, the image sensor 22 moves to the first imaging position and performs imaging. That is, exposure to the image sensor 22 and acquisition of an image are performed (S101, S102). Then, the position of the image sensor 22 at the time of shooting the first image is used as a reference position, and the shooting position of the image sensor 22 is shifted to a different position at the time of shooting the second, third, and fourth images. (S103 to S107). The first to fourth images are automatically taken after the first release button 15 is pressed. Note that during super-resolution imaging, the camera shake correction function may be temporarily set to OFF regardless of the settings so far, and after super-resolution imaging is completed, the original settings may be restored.

FIG. 4 shows the position of the image sensor 22 at the time of shooting, and the horizontal and vertical directions of the image are represented by X and Y coordinates, respectively. For the sake of explanation, it is assumed that the image region IR is configured by a 6 × 6 pixel array. The image sensor 22 shifts at every photographing, that is, the image sensor 22 is formed in the image region IR by moving the position (pixel position) of the image sensor 22 relative to the subject image formed by the photographing optical system 31. The image varies with each shot.

Here, the image sensor 22 moves to a position shifted by +1 pixel in the X direction when the first image is taken as a reference position. In the third shooting, the image sensor 22 moves to a position shifted by +1 pixel and −1 pixel in the X and Y directions with respect to the reference position, and in the fourth shooting in the Y direction with respect to the reference position. The image sensor 22 moves to a position shifted by −1 pixel. As a result, captured images I1 to I4 having different image region IR positions with respect to the subject image are generated.

As described above, a Bayer array color filter array is disposed on the light receiving surface of the image sensor 22, and any one of R, G, and B color elements is disposed to face each pixel. Therefore, if attention is paid to one piece of image data before the color interpolation process, the pixel signal output from each pixel includes only R, G, or B color information. In FIG. 4, the symbols of the respective pixels are represented as “11R, 12G, 13R,.

On the other hand, when super-resolution imaging is performed, the light receiving area IR of the image sensor 22 is sequentially relatively moved by one pixel, so that the same portion is imaged by adjacent pixels having other color filter elements. Thereby, all the R, G, and B color information are included in the pixel signal of each pixel.

For example, in the first shooting, the image PA formed by the pixel 33R is shifted by one pixel of the image sensor 22, and in the second, third, and fourth shooting, the

pixels

32G, 22B, 23G is formed. Since the pixel signals output from the

pixels

32G, 22B, and 23G include G or B color information, it is possible to obtain a pixel signal that includes all the R, G, and B color information for the image PA. it can. For the G component, the average value of the pixel signals output from 32G and 23G is calculated here as the G component pixel value.

When four captured images I1 to I4 are obtained, super-resolution processing is performed to synthesize the captured images I1 to I4, and high-quality images different from the four captured images I1 to I4 are generated. That is, photographed image data based on the R, G, and B pixel values of each pixel is formed. The generated high-quality image is displayed on the LCD 24 and recorded in the image memory 25 (S108 to S110).

FIG. 5 is a diagram illustrating a high-quality image. The image area IR ′ of the high-quality image IH is composed of a 5 × 5 pixel array excluding the right end column that does not include all the R, G, and B information in the 6 × 6 pixel array shown in FIG. Yes. Each pixel has R, G, and B pixel values. For example, the image PA 'of the pixel 22 corresponding to the image PA in FIG. 4 includes R, G, and B color information.

The generation of a high-quality image that is realized by super-resolution imaging by performing super-resolution imaging a plurality of times by such movement of the image sensor 22 accompanied by the imaging region overlap does not change the subject (image). It is assumed that there is no difference in subject images between the images. Since it is assumed that the image sensor 22 is moved and photographed in a state where there is no movement / blur, brightness, hue change or the like in the subject image, the high-quality image has the same subject portion, the same brightness, and color. Must be made up of pixels with a based image.

However, during super-resolution shooting, if the subject moves or the subject's brightness, hue, etc. change to cause a difference in the subject image, the pixels of the high-quality image are images based on the same shooting conditions. do not become. As a result, an image in which an inappropriate image appears partially or entirely is generated.

Therefore, in this embodiment, when a change (difference) in the subject image occurs, complementary shooting is performed. That is, the image is acquired again. Specifically, when movement occurs in the subject image, when the brightness of the subject image changes, or when the hue of the subject image changes, shooting is performed again, based on a plurality of shot images that are not different with respect to the subject image. High quality images.

Here, three detection methods are prepared as methods for detecting a change in the subject image, and detection is possible based on a difference in pixel value, a difference in histogram, and a difference in average value. Hereinafter, each detection method will be described with reference to FIGS.

FIG. 6 is a diagram showing a difference between pixel values. For the two captured images I1 and I2 obtained by the first and second images, a difference in luminance value is calculated between corresponding pixels, and pixels whose difference is greater than or equal to the threshold T1 are counted. Then, when the count number of the pixels that are equal to or greater than the threshold T1 among the total number of pixels is equal to or greater than the threshold T2, it is determined that a change in the subject image has occurred. The threshold values T1 and T2 are determined in consideration of shooting conditions and the like, and are stored in advance in a ROM or the like.

For example, when the subject is moving during super-resolution imaging, pixels that do not become the image of the same subject are generated between the obtained captured images. In FIG. 6, the images of the pixels PX are not based on the same subject in the captured images I <b> 1 and I <b> 2 obtained at the time of the first image capturing and the second image capturing.

FIG. 7 is a diagram showing the difference between the average values of the entire image. An average luminance value is calculated for each of the two captured images I1 and I2 obtained by the first and second images. Then, a difference between the luminance average values is obtained, and when the difference amount is equal to or greater than the threshold value T3, it is determined that the subject image has changed. *

For example, if a change in illumination caused by flickering of a fluorescent lamp, etc., or a change in the brightness of the subject image caused by the operation of the shutter or aperture occurs during super-resolution shooting, the subject does not move. Regarding the brightness of the subject image, there is a difference between the captured images I1 and I2. Such a change in the subject image can be detected by calculating a difference between the average values.

FIG. 8 is a diagram showing a histogram difference of the entire image. Histogram calculations are performed on the two captured images I1 and I2 obtained by the first and second images. Then, the difference between the histograms HG1 and HG2 obtained by the histogram calculation (see the hatched portion in FIG. 8) is obtained. If the difference amount is equal to or greater than the threshold value T4, it is determined that the subject image has changed.

For example, if something unexpected (such as a bird) temporarily enters the shooting range during super-resolution shooting, the movement of the entire subject image changes relatively little, but the brightness, color, etc. Differences occur between I1 and I2. Even when the sun hidden in the clouds leaks sunlight due to slight movement of the clouds, a difference occurs between the captured images. Such a change in the subject image can be detected by calculating a histogram difference.

In this embodiment, the user can set any one detection method in mode setting or the like, or two or more detection methods can be combined and detected. As for the pixel difference value, the histogram difference value, and the average difference value, each difference based on the color information may be calculated instead of the luminance value, and those values may be calculated, or the luminance value and the color information may be used together. Also good.

FIG. 9 is a flowchart of super-resolution imaging and super-resolution processing in the present embodiment.

The image sensor 22 is moved to a predetermined position at the time of photographing the first image and the second image, and photographing is performed (S201 to S204). Then, it is detected whether or not a subject image change has occurred between the first photographed image and the second photographed image (S205). If the subject image has changed, the process returns to step S201, the image sensor 22 is moved to the first position, and the first image is taken again (that is, the first image is taken for the second time).

On the other hand, if there is no change in the subject image between the first and second photographed images, the process shifts to the third photograph (S206, S207) and the first and third photographed images. Whether or not the subject image has changed is determined (S208). If there is a change in the subject image, the process returns to step S201 and starts again from the first image.

If it is determined in step S208 that the subject image has not changed, the process proceeds to the fourth shooting (S209, S210), and the subject image changes between the first and fourth shot images. It is determined whether or not it has occurred (S211). If there is a change in the subject image, the process returns to step S201 and starts again from the first image.

If it is determined in step S211 that the subject image has not changed, super-resolution processing is executed based on the four captured images that have been acquired, and high-quality images are stored and displayed (S212 to S214).

As described above, according to the present embodiment, in super-resolution imaging, is there a change in the subject image between the captured image obtained by the first imaging and the captured image obtained by the second and subsequent imaging? Is detected every time shooting is performed, and if the subject image changes, super-resolution shooting is restarted from the beginning.

To obtain a high-quality image only from a photographic image with a subject image movement, that is, with a subject image difference, and without a subject image motion, an image that is disturbed or loses color balance It is possible to efficiently shoot, display and record a high quality image without bothering image processing, display and recording.

Also, since the change in the subject image is detected based on the first photographed image, if the subject image has moved due to shooting of a moving subject or the like, it can be detected appropriately. For super-resolution shooting, in order to prevent long-time shooting, a limit is set on the number of repeated shooting (re-acquisition of images). If the limit is exceeded, the super-resolution shooting process is forcibly terminated once. If it is done, it is more efficient.

Next, a digital camera according to the second embodiment will be described with reference to FIG. In the second embodiment, when a change in the subject image occurs, the shooting performed immediately before is performed again. About another structure, it is substantially the same as 1st Embodiment.

FIG. 10 is a flowchart of super-resolution imaging and super-resolution processing in the second embodiment.

After the first and second images are taken, it is detected whether or not a subject image change has occurred between the first and second shot images (S301 to S305). If there is a change in the subject image, the second shot, the detection of the change in the subject image are performed for the second time, the third time, and so on until a captured image with substantially no change in the subject image is obtained.

If no change in the subject image has occurred in step S305, the third image is shot, and it is detected whether or not the subject image has changed between the second and third shot images. (S306 to 308). If there is a change in the subject image, the third shot and the change detection of the subject image are performed for the second time, the third time, and so on until a captured image with substantially no change in the subject image is obtained.

If it is determined in step S308 that there is no change in the subject image, the fourth image is taken, and whether there is a change in the subject image between the third image and the fourth image. Whether or not is detected (S309 to S311). If it is determined that a change in the subject image has occurred, the fourth shot and the detection of the change in the subject image are detected for the second time, the third time, and so on until a captured image with substantially no change in the subject image is obtained. And do.

If it is determined in step S311 that the subject image has not changed, super-resolution processing is executed based on the four captured images obtained, and a high-quality image is stored and displayed (S312 to S314).

As described above, according to the second embodiment, in super-resolution imaging, a captured image obtained by imaging of the N−1th image (N is an integer of 2, 3, or 4 here), Whether or not there is a change in the subject image with the shot image obtained in the Nth shooting is detected every time shooting is performed, and if there is a change in the subject image, shooting without change in the subject image The Nth image is taken until an image is obtained.

Since a change in the subject image is detected between two captured images that are adjacent in time series, it is possible to appropriately detect changes in brightness and hue caused by illumination flicker or shutter / aperture variation. . In addition, by not performing shooting from the beginning, depending on the subject, super-resolution shooting may be able to be completed in a short time.

Note that a limit may be set for the number of repetitions of shooting.

Next, a third embodiment will be described with reference to FIG. In the third embodiment, “N (N is an integer of any one of 1, 2, 3, and 4) sheets” in which the shooting of four images is performed first and then the subject image changes after that. Retry only the "eye". About another structure, it is substantially the same as 1st Embodiment.

FIG. 11 is a flowchart of super-resolution imaging and super-resolution processing in the third embodiment.

4 images are taken continuously with the movement of the image sensor 22 to obtain 4 images and temporarily stored in a memory such as a RAM (S401 to S408). Then, after the super-resolution shooting is finished, for example, it is determined whether or not the subject image has changed in the second, third, and fourth shot images with reference to the first shot image (see FIG. S409). If the change in the subject image has occurred, the shooting for obtaining the shot image is retried (S413 to S421). When the subject image has not changed, a high-quality image is stored and displayed by super-resolution processing based on the four images (S410 to S412).

Thus, according to the third embodiment, super-resolution imaging (multiple imaging) is terminated first, and then a change in the subject image is detected and imaging is performed again. Thereby, depending on the subject, it may be possible to finish super-resolution shooting in a short time. Further, it is possible to efficiently process the case where there is no movement and only the brightness change occurs.

In the first to third embodiments, changes in the subject image are detected by extracting two consecutively photographed images or the second and subsequent photographed images based on the first photographed image. However, the present invention is not limited to this, and a change in the subject image may be detected based on a photographed image by another combination. Alternatively, a change in the subject image may be detected based on three or more captured images. Further, the present invention is not limited to shooting four images, and super-resolution shooting over a plurality of images may be performed.

Next, a digital camera according to the fourth embodiment will be described with reference to FIG. In the fourth embodiment, the complementary shooting as in the first to third embodiments is not performed, the user is notified that the subject image has changed, and the super-resolution processing is not performed.

FIG. 12 is a flowchart of super-resolution imaging and processing in the fourth embodiment.

As in the first embodiment, each time the second, third, and fourth shots are taken, whether or not the subject image has changed is determined based on the first shot image ( S501 to S511). If the subject image has not changed, a high-quality image is generated by super-resolution processing and displayed / recorded (S512 to S514).

On the other hand, when a change in the subject image occurs, the controller 40 does not execute the super-resolution process, stores each of the four captured images, and displays the first or other captured images on the LCD 24. (S515, S516). Then, the user is notified that the subject image has changed (S517). Here, a warning text (for example, “the subject image has changed!” Etc.) notifying the LCD 24 of the subject image change is displayed together with the first photographed image or only the warning text.

As described above, according to the fourth embodiment, when a change in the subject image is detected, the super-resolution processing is not performed, the user is notified of the subject image change, and each captured image is stored. The captured image of is displayed. This prevents unnecessary super-resolution processing that fails as a result from being executed, displayed, and recorded. Also, by detecting a change in the subject image for each shooting, it is possible to immediately stop super-resolution shooting and prompt the user to review the shooting environment. Since super-resolution processing is not performed even when the subject image changes that cannot be recognized by the user during shooting, unnecessary image processing, image recording, and image display are avoided. In addition, since each captured image is stored individually, it can be diverted to another purpose even if it is not suitable for super-resolution processing.

In addition, about the notification to the user of the subject image change by a screen display, structures other than a display are also possible, and the light emission part 90 (refer FIG. 1) provided in the camera main body 20 can be light-emitted. Alternatively, a predetermined sound or message such as a buzzer sound may be emitted.

Next, a digital camera according to the fifth embodiment will be described with reference to FIGS. In the fifth embodiment, super-resolution processing is performed regardless of the change in the subject image. On the other hand, if the subject image has changed, the user is notified of the change.

FIG. 13 is a flowchart of super-resolution imaging and processing in the fifth embodiment. FIG. 14 is a view showing a display screen for informing a subject image change.

Continuous super-resolution imaging is performed for four images, and the first to fourth captured images are temporarily stored in the memory, while an image is generated, displayed, and stored by super-resolution processing (S601˜). S611). On the other hand, it is determined whether or not a subject image change has occurred in the second, third, and fourth shot images with reference to the first shot image (S612). If a change in the subject image has occurred, the user is notified of this by using the generated high-quality image (S613).

FIG. 14 shows a high-quality image obtained by super-resolution imaging of two subjects SI and MI. Since movement or the like has occurred in the subject MI, image editing processing is performed on the image area of the subject MI so that the subject MI can be identified from the subject SI. Here, the subject image change is notified to the user by painting the subject MI with the same color.

As described above, according to the fifth embodiment, when a change in the subject image is detected, the user can know the changed portion from the image obtained by the super-resolution processing. Even in an image in which the subject image change is difficult to recognize, the user can know the part of the change. An arbitrary captured image may be displayed and displayed so that a portion where the subject image has changed in the image can be identified. Further, the outline of the image area where the change has occurred may be highlighted. According to this, although a high-quality image including an error due to a change in the subject image is strictly generated, the position and range of the error can be clearly recognized by the user. If it is a narrow part of the outermost peripheral part of the screen, it may be possible to cope by trimming that part. As a result, it is possible to prevent a situation in which the user does not notice an image with an error and releases or delivers the image.

Next, the sixth embodiment will be described with reference to FIG. In the sixth embodiment, a subject image change is detected in a through image display state before super-resolution imaging.

FIG. 15 is a flowchart of super-resolution imaging and processing in the sixth embodiment. If the super-resolution processing mode is set under the through image display mode, the processing is started. Alternatively, the processing may be started after an autofocus operation or the like is performed by half-pressing the release button 15 under the through image display mode or the super-resolution processing mode setting.

In step S701, it is determined whether or not the subject image has changed based on the pixel signal for one field / frame read at a predetermined time interval during the through image (preceding image) display. Here, since the position of the image sensor 22 is fixed, a change in the subject image is detected between pixel signals (pre-photographing images) for two consecutive fields / frames. Further, the threshold value used when detecting the subject image change described above is set to a value smaller than the threshold value described above.

If there is a change in the subject image, a warning message is displayed to notify that change, or the light emitting unit 90 emits light (S702). On the other hand, if the subject image change is not detected and the release button 15 is fully pressed, super-resolution shooting and super-resolution processing are executed, and a high-quality image is displayed and stored (S703 to S715).

As described above, according to the sixth embodiment, when a change in the subject image is detected before the super-resolution imaging is started, the user is notified of the change. As a result, it is possible to prevent unnecessary super-resolution imaging that is unsuccessful even when image processing is performed. It should be noted that the change in the subject image may be detected from the pre-photographing images with four or other numbers, or the change in the subject image may be detected by motion vector detection or the like. Further, the present embodiment may be configured not to perform subject image change detection and complementary shooting or user notification in super-resolution shooting. Note that as a modification of the sixth embodiment, even when the release button 15 is fully pressed, the exposure operation is not performed immediately, and a change in the subject image is detected for a certain period. Super-resolution processing may be started and a high-quality image may be displayed and stored.

In the first to sixth embodiments, super-resolution imaging by moving the image sensor by one pixel is performed to realize a high-quality image with improved color reproducibility. However, the movement amount of the image sensor is 1 The super-resolution processing may be performed by moving the image sensor with an amount of movement larger than that or a sub-pixel (1 pixel or less) accuracy. That is, it may be applied to super-resolution processing based on a plurality of captured images.

For super-resolution imaging, instead of moving the image sensor relative to the subject image, the subject image may be moved relative to the image sensor by driving the optical element eccentrically, and super-resolution imaging may be performed. An image processing unit may be provided separately from the controller. Digital cameras can be applied to both single-lens reflex cameras with a detachable lens (including so-called “mirrorless single-lens cameras”) and compact cameras with a built-in lens. The present invention can also be applied to an imaging device.

On the other hand, super-resolution processing can also be executed by image editing software or an image processing apparatus that reads out a captured image recorded by a camera or the like and performs image editing processing. In this case, in association with each captured image data, a plurality of sets of captured image information and the presence / absence of changes in the subject image are recorded, and a plurality of sets of captured images captured by the image editing software or the image processing apparatus are recorded. When there is a change in the subject image, the user can recognize that the subject image has changed by notifying the user of that fact.

As described above, in order to generate an image by super-resolution processing based on a plurality of images using a camera or the like, a plurality of images (exposure and exposure) are shifted while shifting the pixel position (image sensor position) at the time of shooting. When performing image acquisition), the user has conventionally experienced movement during shooting, or brightness, color change, etc. during shooting due to illumination, shutter, or aperture operation. Therefore, there is a possibility that the image processing is useless and the failed image is recorded and displayed as it is. If the high-resolution image generation by the super-resolution processing fails, the photographer must take another image while confirming the composition and the exposure condition. However, according to the embodiment of the present application, such inconvenience is eliminated, and the photographing operation can be performed efficiently.

10 Digital camera (imaging device)
22 Image sensor (image sensor)
40 controller (image processing unit, control unit, detection unit, pre-shooting detection unit, pre-shooting control unit)

Claims

A control unit that controls the acquisition of a plurality of images while changing the relative position between the subject image and the image sensor;
An image processing unit that performs image processing based on the plurality of acquired images;
A detection unit that detects a difference in the subject image in at least a part between the acquired at least two images,
An image pickup apparatus, wherein the control unit obtains an image again or notifies a subject image difference in accordance with the subject image difference.
2. The imaging apparatus according to claim 1, wherein the detection unit detects a difference between the subject images based on a difference between at least two images.
The said detection part calculates | requires the difference of the brightness | luminance or color information between the corresponding pixels between at least two images, and detects the difference of the said subject image based on the calculated | required difference. 3. The imaging device according to any one of 2.
The said detection part calculates | requires the difference of the brightness | luminance histogram or color information histogram of at least 2 image, and detects the difference of the said to-be-photographed image based on the calculated | required difference. The imaging device described in 1.
The detection unit obtains a difference between an average value of luminance of at least two images or an average value of color information, and detects a difference between the subject images based on the obtained difference. 5. The imaging device according to any one of 4.
The detection unit detects at least one of a change in position of a subject image between at least two images, a change in brightness of the subject image, and a change in color of the subject image as a difference between the subject images. The imaging apparatus according to claim 1, wherein:
The detection unit detects a difference between the subject images before completion of all acquisition of a plurality of images based on the image processing,
The imaging apparatus according to claim 1, wherein when the difference between the subject images is detected, the control unit executes re-acquisition of a plurality of images from the beginning.
The detection unit detects a difference between the subject images before completion of all acquisition of a plurality of images based on the image processing,
The said control part performs reacquisition from the image acquired last among the some images already acquired, if the difference of the said to-be-photographed image is detected, Any one of Claim 1 thru | or 6 characterized by the above-mentioned. An imaging apparatus according to claim 1.
The detection unit detects a difference between a subject image of a reference image and a subject image of another image after completion of acquisition of a plurality of images based on the image processing,
The imaging apparatus according to claim 1, wherein when the difference in the subject image is detected, the control unit executes re-acquisition of the image in which the difference has occurred.
The detection unit detects a difference between the subject images before completion of all acquisition of a plurality of images based on the image processing or after completion of acquisition of a plurality of images based on the image processing,
7. The control unit according to claim 1, wherein when the difference between the subject images is detected, the control unit executes at least one of storage and display of at least one acquired image. An imaging apparatus according to claim 1.
The detection unit detects a difference between the subject images after image processing based on the plurality of images;
The imaging apparatus according to claim 1, wherein when the difference between the subject images is detected, the control unit notifies the difference between the subject images.
A pre-shooting detection unit that detects a difference in a subject image between a plurality of images obtained by an image sensor while changing the relative position before starting the acquisition of a plurality of images;
The imaging apparatus according to claim 1, further comprising: a pre-shooting control unit that notifies the difference between the subject images when the difference between the subject images is detected.
The imaging according to any one of claims 1 to 12, wherein the control unit notifies the difference between the subject images by displaying an image in which a portion where the difference between the subject images occurs is displayed. apparatus.
The imaging apparatus according to claim 1, further comprising: a light emitting unit, wherein the control unit notifies the difference between the subject images by causing the light emitting unit to emit light.
Execute acquisition of multiple images while changing the relative position of the subject image and the image sensor,
Perform image processing based on the acquired multiple images to generate another image,
Detecting at least a portion of the subject image difference between the acquired at least two images,
An imaging method comprising: acquiring an image again or notifying the subject image difference according to the subject image difference.
The imaging device
Control means for controlling acquisition of a plurality of images while changing the relative position between the subject image and the image sensor;
Image processing means for executing image processing based on the plurality of acquired images and generating another image;
Function as detection means for detecting a difference in at least a portion of the subject image between at least two acquired images;
A program that functions as the control unit so as to acquire an image again or notify the difference of the subject image in accordance with the difference of the subject image.
An image processing unit that executes image processing based on a plurality of images obtained by acquiring a plurality of images while changing a relative position between the subject image and the image sensor, and generates another image;
A detection unit for detecting a difference in at least a part of the subject image between at least two images;
An image processing apparatus comprising: a notifying unit that notifies the difference between the subject images when the difference between the subject images is detected.
A control unit that controls the acquisition of a plurality of images while changing the relative position between the subject image and the image sensor;
An image processing unit that performs image processing based on the plurality of acquired images;
A detection unit that detects a difference in at least a part of the subject image between at least two images,
The imaging apparatus, wherein the control unit changes the number of acquisition times of the plurality of images according to a difference between the subject images.