WO2006052030A1 - 撮像装置と撮像システムおよび画像の撮影方法 - Google Patents
撮像装置と撮像システムおよび画像の撮影方法 Download PDFInfo
- Publication number
- WO2006052030A1 WO2006052030A1 PCT/JP2005/021316 JP2005021316W WO2006052030A1 WO 2006052030 A1 WO2006052030 A1 WO 2006052030A1 JP 2005021316 W JP2005021316 W JP 2005021316W WO 2006052030 A1 WO2006052030 A1 WO 2006052030A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- imaging
- resolution
- imaging device
- timing
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2625—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
Definitions
- the present invention relates to an imaging technique for synthesizing a high-resolution image from an imaging device having a small number of pixels, and in particular, to an imaging device having an imaging technique and high-resolution processing, an imaging system, and This is related to the method of taking images.
- Background art
- Patent Document 1 When a technique is known in which a plurality of images are captured while the image sensor is accurately displaced at intervals smaller than the pixel interval, and a single high-definition image is generated from the plurality of images. As described in Patent Document 1, there is an apparatus that takes an image by moving an optical system or an image sensor.
- the present invention has been made in view of the above-mentioned problems, and can perform pixel-shifted shooting without requiring precise control of the amount of displacement of the optical element or imaging element, and can achieve high resolution using a plurality of images.
- An object is to provide an imaging device, an imaging system, and an image capturing method. Disclosure of the invention
- an imaging apparatus includes an optical imaging unit that forms an image of a subject in an imaging apparatus that electronically obtains an image of the subject.
- An imaging unit that spatially discretizes an optically imaged image and converts it into a sampled image signal; and a high-resolution image is generated from the image signals of a plurality of frames sampled by the imaging unit.
- a resolution changing unit a state change detecting unit that detects a state change of the imaging apparatus itself; and an imaging timing determining unit that determines an imaging timing; and the imaging timing determining unit includes the state change Taking into account the change in the state of the imaging device itself detected by the detection means, the imaging timing for obtaining an image signal of an appropriate frame for generating a high resolution image by the high resolution means The Characterized in that it constant.
- the invention in () corresponds to the first embodiment shown in FIG.
- the optical imaging means 1 1 having the configuration of (1) forms an image of the subject, and the optical imaging unit 1 1 converts the optically imaged image into a spatially discrete sampled image signal.
- the means for imaging is the imaging unit 1 2
- the means for generating a high-resolution image from the image signals of a plurality of frames is the image processing unit 15, and the means for detecting the state change of the imaging device itself is the moving speed detection unit 1.
- 3 is the means for determining the timing of imaging, which corresponds to the imaging timing determination unit 14.
- the invention of (1) is provided with means for detecting a change in the state of the imaging apparatus itself, it is possible to perform imaging at a timing when the speed of the imaging apparatus itself is reduced. This eliminates the need for precise control of the amount of displacement of the optical element or image sensor, and makes it possible to perform pixel-shift photography using random movements such as camera shake and achieve high resolution using multiple images. To enable imaging it can. If the state change here is acceleration, the velocity is calculated from the history of direction and magnitude of acceleration.
- An imaging apparatus is an imaging apparatus that electronically obtains an image of a subject, optical imaging means that forms an image of the subject, and optical imaging An image sensor that spatially discretizes an image and converts it into a sampled image signal; a high-resolution means that generates a high-resolution image from image signals of a plurality of frames sampled by the image sensor; An imaging timing determining unit that determines an image timing; a unit that applies a spatial displacement to the imaging device; and a unit that detects a state change of the imaging device.
- the imaging timing determination unit includes: An imaging tie for obtaining an image signal of an appropriate frame for generating a high-resolution image by the high-resolution means in consideration of the state change of the image sensor detected by the means for detecting the state change. And determines the ring.
- the invention (2) corresponds to the third embodiment shown in FIG.
- the optical imaging means that forms an image of a subject having the configuration (2) is an optical image forming unit 11 that spatially discretizes the optically imaged image into a sampled image signal.
- the imaging device 12 converts the imaging device to convert, the image processing unit 15 generates the high-resolution image from the image signals of a plurality of frames, and the imaging timing determination unit 14 determines the imaging timing.
- the means for giving a spatial displacement to the image sensor corresponds to the image sensor moving unit 117, and the means for detecting the state change of the image sensor corresponds to the moving speed detector 13.
- the invention of (2) is provided with means for giving a spatial displacement to the image pickup device and means for detecting a change in the state of the image pickup device, not the image pickup device itself. For this reason, it is possible to take an image with a positional shift regardless of the method for holding and fixing the imaging apparatus.
- the imaging device of the present invention is characterized in that, in the inventions of (1) and (2), the state change is a speed.
- the configuration of the invention of (3) corresponds to the moving speed detector 13 shown in FIGS. 1 and 11 respectively. Since the invention of (3) detects the speed change, it is possible to easily detect the state change of the image pickup device or the image pickup element.
- the state change is acceleration in the invention of (1) or (2).
- the configuration of the invention of (4) corresponds to a configuration in which an acceleration detection unit is provided instead of the moving speed detection unit 13 shown in FIG. 1 and FIG. 11.
- the speed at that time can be calculated from the history of the direction and magnitude of the acceleration.
- the imaging device of the present invention is the above-mentioned invention (1) or (2). It is characterized in that imaging is performed when the speed is zero. In the invention of (5), since the image is taken when the speed is 0 in the image pickup apparatus or the image pickup device, a high-definition image without image blur can be taken.
- the imaging device of the present invention is the imaging device according to any one of (2) to (5), wherein the imaging element is displaced in one linear direction substantially perpendicular to the optical axis. It is characterized by.
- the invention (6) corresponds to the third embodiment shown in FIG. In the invention of (6), since the image element can be displaced in one linear direction substantially perpendicular to the optical axis, it is possible to efficiently acquire images of a plurality of frames with variations in photographing positions, and to achieve high resolution. Can be done easily.
- An imaging apparatus is an imaging apparatus that electronically obtains an image of a subject, optical imaging means that forms an image of the subject, and optical imaging An image that is spatially discretized and sampled
- An imaging device for converting the signal, a high-resolution means for generating a high-resolution image from image signals of a plurality of frames sampled by the imaging device, an imaging timing determination unit for determining the timing of the imaging, Means for spatially giving displacement to the image sensor; means for giving spatial displacement to the optical imaging means in a state different from the image sensor; and the image sensor and the optical imaging means.
- a relative change detecting means for detecting a relative change with respect to the imaging timing determining means, taking the relative change detected by the relative change detecting means into consideration, and obtaining a high resolution image by the high resolution means. It is characterized in that the imaging timing for obtaining an image signal of an appropriate frame to be generated is determined.
- the invention (7) corresponds to the imaging apparatus according to the fourth embodiment shown in FIG.
- the optical image forming means for forming the image of the subject is converted into a sampled image signal obtained by spatially discretizing the optically imaged image.
- the image pickup unit 1 2 is the image pickup device
- the image processing unit 15 is the means for generating a high-resolution image from the image signals of a plurality of frames
- the image pickup timing determination unit 14 is the means for determining the image pickup timing.
- Means for generating a signal for starting a series of processing relating to the imaging preparation start signal generator 1 1 6 is provided
- means for applying a spatial displacement to the image pickup device is an image pickup element moving unit 1 1 7.
- the means for giving the image means a spatial displacement in a different state in the same direction as the image pickup means is a means for the optical element moving unit 138 to detect a relative change between the image pickup element and the optical imaging means.
- Relative speed detector 1 33 corresponds to each .
- the imaging device of the present invention is characterized in that, in the invention of (7), the relative change is a relative speed, and imaging is performed when the relative speed is zero.
- the invention (8) corresponds to the fourth embodiment shown in FIG. 13 and the fifth embodiment shown in FIG.
- the relative change between the image sensor and the optical imaging means is the relative speed
- the detection of the relative speed being 0 is performed by detecting the relative speed detector 133, FIG. This corresponds to the case where the detection result of the relative speed detector 163 in FIG.
- the relative change between the image sensor and the optical imaging means detected by the relative change detecting means is a relative speed, and the image is taken when the relative speed is 0, high-definition without image blurring. You can shoot a good image.
- the relative change is a relative acceleration.
- the invention (9) relates to a relative velocity detector 133 as a means for detecting a relative change between the image pickup element and the optical imaging means in the image pickup apparatus according to the fourth embodiment shown in FIG. Instead, a relative acceleration detector is provided. In this way, the relative change between the image sensor and the optical imaging means can be detected by the change in acceleration, and the speed can be calculated from this acceleration. Therefore, image blurring can be performed regardless of the method for holding and fixing the image pickup device. It is possible to acquire multiple images with few images and to achieve high resolution.
- the imaging device of the present invention is the imaging device according to any one of (7) to (9), wherein the imaging device and the optical imaging means are one perpendicular to the optical axis. Displacement can be given in a linear direction.
- the invention 0) corresponds to the imaging device according to the fourth embodiment shown in FIG.
- the configuration in which the imaging device and the optical imaging means of the invention of (10) can be displaced in one linear direction substantially perpendicular to the optical axis is shown in FIG. This corresponds to the view La direction and the arrow L b direction of the displacement of the optical imaging means being one linear direction substantially perpendicular to the optical axis. Since the image sensor and optical imaging means can be displaced in one linear direction that is almost perpendicular to the optical axis, it is possible to efficiently acquire images of multiple frames with variations in shooting position, suitable for high-resolution processing. .
- an imaging device of the present invention is characterized in that, in the invention described in any one of (2) to (-10), the imaging device includes means for measuring a displacement amount of the imaging device. .
- the invention (11) corresponds to the imaging apparatus according to the third, fourth, and fifth embodiments shown in FIG.
- the means for measuring the displacement of the imaging element of the invention of (11) corresponds to the motion measuring unit 169 shown in FIG. In this way, since it has a means to measure the amount of displacement of the image sensor, it is not estimated by calculating the motion from the image, but it is actually measured, so the accurate relative position can be determined regardless of the type of image. Information can be obtained.
- the imaging device of the present invention is characterized in that, in the invention described in the above (2) or (7), the means for giving a spatial displacement is an inertia member.
- the invention (12) corresponds to the imaging devices according to the third, fourth, and fifth embodiments shown in FIGS. 11, 13, and 16.
- the elastic member according to the invention of (12) corresponds to the image sensor driving spring 124 shown in FIG.
- an elastic member such as a spring is used as means for spatially giving the image sensor a displacement. The vibration of the spring attenuates over time. And the movement speed is the slowest at different places each time. Since the direction is changed, it is possible to obtain motions that vary efficiently by shooting at that position.
- the imaging device of the present invention is characterized in that the resolution increasing means increases the resolution of an image at a desired magnification. It is characterized by having a function to determine whether or not it is possible and to notify the photographer.
- the invention (1 3) corresponds to the sixth embodiment described in FIGS. 18 to 23, but is also applied to other embodiments.
- the function of determining whether or not it is possible to increase the resolution of the image to the desired magnification of the invention of (1 3) and notifying the photographer is the high resolution determination section 1810 shown in FIG.
- the signal confirmation unit 221 shown in Fig. 22 corresponds. With such a configuration, when high resolution processing is performed, a high resolution image can be obtained with a high probability.
- an optical imaging means for forming an image of a subject, and the optically imaged image are spatially dispersed and sampled.
- Imaging means for converting to an image signal high-resolution means for generating a high-resolution image from image signals of a plurality of frames sampled by the imaging means, and state change detection for detecting a state change of the imaging device itself Means, an imaging timing determining means for determining the timing of imaging, an imaging device moving means for giving displacement to the imaging device, and holding or supporting the imaging device and the imaging device moving means
- the imaging timing determining means generates a high-resolution image by the high-resolution means in consideration of the state change of the imaging device detected by the state change detection means. It is characterized by determining an imaging timing for obtaining an image signal of an appropriate frame to perform.
- the invention of (14) corresponds to the second embodiment of the present invention shown in FIG. To do.
- the optical imaging means for imaging the subject image of the invention of (14) is an image obtained by sampling the optically imaged image spatially discretized by the optical imaging unit 1 1.
- the means for converting to a signal is the imaging unit 1 2
- the means for generating a high-resolution image from the image signals of a plurality of frames is the image processing unit 1 5
- the means for detecting the state change of the imaging device itself is the moving speed detection unit 1 3
- the imaging timing determination unit 14 4 is a means for generating a signal for starting a series of processing related to imaging
- the imaging start signal generator 7 6 is
- the imaging device moving unit 7 7 corresponds to the imaging device moving unit to be applied
- the imaging device fixing unit 78 corresponds to the fixing unit that holds or supports the imaging device and the imaging device moving unit.
- the invention of (14) has an imaging device moving means for giving a displacement to the imaging device, and a fixing means for holding or supporting the imaging device and the imaging device moving means. For this reason, by taking a picture while moving it in one direction that is neither horizontal nor vertical with respect to the imaging device, there is a position that has a horizontal and vertical shift without having a mechanism to move it separately in the horizontal and vertical directions. It is possible to take images with In addition, since the photographer does not hold the imaging device in his / her hand, only an image having a motion suitable for high resolution processing can be captured.
- An imaging system is an optical imaging means for forming an image of a subject, and the optically formed image is spatially dispersed and sampled.
- An image sensor for converting to an image signal, a high-resolution means for generating a high-resolution image from image signals of a plurality of frames sampled by the image sensor, and an imaging timing determination for determining an imaging timing Means, a means for giving a spatial displacement to the image sensor, a means for detecting a change in the state of the image sensor, an image pickup apparatus, and a fixing means for holding or supporting the image pickup apparatus.
- Imaging timing determination means In consideration of the state change of the image sensor detected by the means for detecting the state change of the image sensor, to obtain an image signal of an appropriate frame for generating a high resolution image by the high resolution means It is characterized by determining the imaging timing.
- the invention (15) corresponds to a configuration in which the fixing means (image pickup device fixing portion 78) for holding or supporting the image pickup device shown in FIG. 7 is applied to the image pickup device shown in FIG. Therefore, in the invention of (15), since the photographer does not hold the imaging device in his hand like the invention of (14), the image having a motion suitable for high-resolution processing. Only imaging is possible.
- An imaging system includes optical imaging means for forming an image of a subject and sampling by spatially separating the optically formed image.
- An image sensor that converts the image signal into a captured image signal, a high-resolution means that generates a high-resolution image from the image signals of a plurality of frames sampled by the image sensor, and an image capture timing that determines the image capture timing Determining means, means for giving a spatial displacement to the image sensor, means for giving a spatial displacement to the optical imaging means in a different state in the same direction as the image sensor, the image sensor and the A relative change detecting means for detecting a relative change with respect to the optical imaging means; an imaging device having: and a fixing means for holding or supporting the imaging device, wherein the imaging timing determining means includes Relative change detection In consideration of the relative change detected by the means, the imaging timing for obtaining an image signal of a frame suitable for generating a high-resolution image by the high-resolution means is determined. .
- the invention (16) corresponds to a configuration in which the fixing means (image pickup device fixing portion 78) for holding or supporting the image pickup device shown in FIG. 7 is applied to the image pickup device shown in FIG. Therefore, the invention of (1 6) Similar to the invention, since the photographer does not hold the imaging device in his / her hand, it is possible to capture only an image having a motion suitable for high-resolution processing.
- An imaging system includes an optical imaging means for forming an image of a subject, and sampling by spatially separating the optically formed image.
- An imaging device that converts the image signal into a captured image signal, a high-resolution means that generates a high-resolution image from the image signals of a plurality of frames sampled by the imaging device, and an imaging timing determination that determines the timing of imaging Means for spatially giving displacement to the image sensor, means for giving spatial displacement to the optical imaging means in a state different from the image sensor in the same direction, the image sensor and the optical
- a relative change detecting means for detecting a relative change with respect to the image forming means; a means for measuring a spatial position of the image pickup element at the time of shooting; an imaging apparatus having: and holding or supporting the imaging apparatus.
- Fixing means The imaging timing determination means obtains an image signal of a frame suitable for generating a high resolution image by the high resolution means in consideration of the relative change detected by the relative change detection means. It is characterized in that the
- the invention (17) corresponds to a configuration in which the fixing means (image pickup device fixing portion 78) for holding or supporting the image pickup device shown in FIG. 7 is applied to the image pickup device shown in FIG. Therefore, the invention of (1 7) is the above (15),
- the imaging system of the present invention is characterized in that, in the invention of (14) or (15), the state change is speed or acceleration.
- the imaging system of the present invention has the above (1 6) or (1 7
- the relative change is a relative velocity or a relative acceleration.
- the invention of (19) has a configuration in which the optical imaging means has means for spatially giving a displacement in a different state in the same direction as the image sensor.
- an imaging system that detects relative changes between the image sensor and the optical imaging means using relative speed or relative acceleration, it is possible to achieve high resolution using multiple images by performing pixel-shifted shooting.
- the imaging system of the present invention provides the above (1 4) to (19).
- the image pickup device can be displaced in one linear direction substantially perpendicular to the optical axis.
- the invention of (20) has the displacement mechanism described in FIG. For this reason, it is possible to obtain an imaging system capable of imaging at a position shifted in the horizontal and vertical directions without having a mechanism for moving the imaging apparatus separately in the horizontal and vertical directions.
- the imaging system of the present invention provides the above (1 4) to (2 0
- the means for giving a spatial displacement is an elastic member.
- the invention of (2 1) uses a spring-like elastic member as described in FIG. 9 as means for giving a spatial displacement to the imaging device.
- the spring vibration is damped over time. Since the moving speed is the slowest and the moving direction is changed at a different place each time, an image with motions that vary efficiently can be obtained by shooting at that position. The system is obtained
- the imaging system of the present invention provides the above (1 4) to (2 1 ),
- the high-resolution means has a function of determining whether or not it is possible to increase the resolution of an image at a desired enlargement magnification and notifying the photographer.
- the invention of (2 2) is a high resolution image pickup system such as the high resolution determination unit 18 10 10 shown in FIG. 18 and the signal checking unit 2 2 1 described in FIG. It has a function to determine whether or not it is possible to notify the photographer. For this reason, unnecessary high resolution processing can be omitted, so that when the high resolution processing is performed, an imaging system that obtains a high-resolution image with high probability can be configured.
- An image capturing method is an image capturing method based on the premise that the resolution of an image is increased using a plurality of images. And a step of determining an imaging timing, and imaging is performed at an appropriate timing in consideration of a state change of the imaging device.
- the moving speed detection unit 1 3 detects a change in the state of the imaging device itself, and the imaging timing determination unit 1 4 determines the imaging sunset.
- the invention of (2 3) which is a method of capturing an image using a configuration that performs the above-described step (procedure) for detecting a change in the state of the white body of the imaging device, determines the step of imaging imaging (steps). (Procedure) is set, so it is possible to shoot at the evening time when the speed of the imaging device itself is low. Therefore, it does not require precise control of the optical element or image sensor, and performs pixel-shifted shooting using random movements such as camera shake, resulting in high resolution using multiple images. If the state change here is acceleration, the speed can be calculated from the ⁇ history of acceleration direction and magnitude.
- a step for generating a signal for starting a series of processing relating to imaging, a displacement for the imaging device in response to a signal for starting a series of processing ', and a step further including It is characterized by.
- the invention of (24) is an example related to imaging by the imaging preparation start signal generator 7 6 described in FIG. 7 as an example.
- the invention of o (4) which is a method for capturing an image using a configuration in which a signal for starting a series of processes is generated and the image capturing device moving unit 77 applies displacement to the image capturing device, A signal to start a series of processing related to imaging is added, and a step to give displacement by receiving a signal to start a series of sd processing is added, so the pixel spacing ⁇ of the image element is also fine motion It is possible to realize a method for taking images that perform high-resolution processing using images with.
- the invention described in 4) is characterized in that the image pickup apparatus further includes a step for giving displacement in two perpendicular directions substantially perpendicular to the optical axis.
- the invention of 5) is a method of taking an image using a configuration such as the image sensor moving portion 8 1 shown in FIG. 9, for example.
- the invention of (25) has a procedure to give the imaging device a displacement in one linear direction almost perpendicular to the optical axis, so that there is a deviation in the horizontal and vertical directions. It becomes possible to take images.
- An image capturing method is a method of capturing an image based on the premise that the resolution of the image is increased using a plurality of images.
- a step of determining an imaging timing, and imaging is performed at an appropriate timing in consideration of the state change of the detected imaging element.
- a signal for starting a series of processing relating to imaging is generated by the imaging preparation start signal generating unit 116 shown in FIG. 11, and the imaging element moving unit ⁇ 7
- the invention of (26) can take an image with a misalignment by detecting the displacement of the image pickup device, not the image pickup device itself, regardless of the method of holding and fixing the image pickup device. Is possible.
- the invention of (2 6) further includes a step of giving displacement to the image sensor in one linear direction substantially perpendicular to the optical axis. It is characterized by.
- the invention (2.7) is a method of taking an image using a configuration in which a displacement is given in one linear direction substantially perpendicular to the optical axis with respect to the image sensor as shown in FIG. Since the invention of (27) has such a step, it is possible to obtain an image capturing method that enables imaging at a position that is shifted in the horizontal and vertical directions.
- An image capturing method is an image capturing method based on the premise that image resolution is increased using a plurality of images. And a step for generating a signal for starting image processing, and a signal for starting a series of processing for imaging, and spatially displaces the image sensor and is the same as a part of the imaging optical system or the entire image sensor.
- a signal for starting a series of processing related to imaging is generated by the imaging preparation start signal generating unit 1 16 shown in FIG. 13 and the imaging element moving unit 1 1.7 Relative displacement is given to the image sensor, and the optical element moving unit 1 38 gives a spatial displacement in a different state in the same direction as the image sensor to the whole or part of the imaging optical system.
- This is a method of taking an image using a configuration in which a relative change between the image pickup device and the optical imaging means is detected by 33 and the image pickup timing is determined by the image pickup timing determination unit 14.
- the optical imaging means is spatially displaced in a different state in the same direction as the imaging element, and the imaging element and the optical element are detected by the relative change detection means. A relative change with the imaging means is detected. For this reason, it is possible to acquire a plurality of images with little image blurring regardless of the method of holding and fixing the imaging device, and it is possible to capture a high-resolution image.
- An image capturing method is an image capturing method based on the premise that the resolution of an image is increased using a plurality of images.
- a step of generating a signal to start, a step of spatially displacing the image sensor, and a part or the whole of the imaging optical system is spatially displaced in a different state in the same direction as the image sensor.
- the invention of (29) is, for example, a shooting preparation start signal generation as shown in FIG. 7
- the raw unit 116 generates a signal for starting a series of processing related to imaging, the imaging element moving unit 117 spatially displaces the imaging element, and the optical element moving unit 138 partially or entirely of the imaging optical system.
- the image sensor and the optical imaging means are detected by the relative velocity detector 163, and the relative change is detected in the same direction as the image sensor.
- This is an image capturing method using a configuration in which the timing is determined and the spatial position of the image sensor is measured when the motion is measured by the motion measuring unit 169.
- the invention of (29) measures the spatial position of the image sensor when photographing is performed by the motion measuring unit 169, so that an accurate relative position can be obtained regardless of the type of image. Information can be acquired.
- the image capturing method of the present invention is the method according to (28) or (29), wherein a part or the whole of the image sensor and the imaging optical system is substantially the optical axis.
- the method further includes a step of giving displacement in one vertical linear direction.
- the invention (30) is, for example, an image capturing method using a configuration that gives displacement to the imaging device and the imaging optical system shown in FIG.
- the imaging device and the optical imaging means can be displaced in one linear direction substantially perpendicular to the optical axis, these are separately provided in the horizontal and vertical directions. Without having a mechanism to move, it is possible to take an image at a position that is shifted in the horizontal and vertical directions.
- the step of detecting the relative change is a step of detecting a relative velocity or a relative acceleration. It is characterized by being.
- the invention of (3 1) is, for example, an image capturing method using a configuration in which the relative speed detection unit 133 detects the speed in FIG. 13 or detects the relative acceleration instead of the relative speed detection unit 133.
- the invention of (3 1) is When detecting acceleration as a relative change between the image sensor and imaging optical system, the current speed can be calculated from the history of acceleration direction and size.
- the invention (3 2) is, for example, an image capturing method using a configuration having the motion measuring unit 169 shown in FIG. As described above, since there is a step of measuring the displacement amount of the image sensor, accurate information on the relative position between images can be acquired regardless of the type of image.
- the invention (3 3) is an image capturing method using a configuration provided with, for example, the high resolution determination unit 1810 shown in FIG. 18 and the signal confirmation unit 221 shown in FIG. In this way, since it is determined whether or not high resolution of the image is possible and unnecessary high resolution processing is not performed, high resolution images can be obtained with high probability when high resolution processing is performed. it can.
- FIG. 1 is a block diagram showing an embodiment of the first invention.
- FIG. 2 is a block diagram of the image processing unit in the embodiment of the first invention.
- Figure 3 is a flowchart of motion estimation.
- Fig. 4 is a conceptual diagram of the estimation of the optimal similarity of motion estimation.
- Figure 5 is a flowchart of high-resolution image estimation.
- Fig. 6 is a block diagram of super-resolution processing.
- FIG. 7 is a block diagram showing an embodiment of the second invention.
- FIG. 8 is a block diagram of the imaging device moving unit in the embodiment of the second invention.
- FIG. 9 is an explanatory view showing an example of the movement of the image pickup apparatus in the embodiment of the second invention.
- FIG. 10 is an explanatory diagram showing a trajectory that can be photographed for one pixel or less when the imaging apparatus is moved obliquely.
- FIG. 11 is a block diagram showing an embodiment of the third invention.
- FIG. 12 is an explanatory diagram showing a configuration example of an imaging unit and an imaging element driving unit in the embodiment of the third invention.
- FIG. 13 is a block diagram showing an embodiment of the fourth invention.
- FIG. 14 is a block diagram of the image processing unit in the embodiment of the fourth invention.
- FIG. 15 is an explanatory view showing an example of the positional relationship between the optical element and the image sensor in the embodiment of the fourth invention.
- FIG. 16 is a block diagram showing an embodiment of the fifth invention.
- FIG. 17 is a block diagram of the image processing unit in the embodiment of the fifth invention.
- FIG. 18 is a block diagram showing an embodiment of the sixth invention.
- FIG. 19 is an explanatory diagram showing an example of uneven distribution of motion when a plurality of images are taken.
- FIG. 20 is an explanatory diagram for the determination of the region to which the motion belongs.
- FIG. 21 shows the configuration of the high-resolution determination unit in the sixth embodiment.
- FIG. 22 is a block diagram of the image processing unit in the embodiment of the sixth invention.
- FIG. 23 is a block diagram of a modification of the embodiment of the sixth invention. BEST MODE FOR CARRYING OUT THE INVENTION
- FIG. 1 is a block diagram showing an image pickup apparatus according to the first embodiment.
- Super-resolution processing is a technique in which a plurality of images with misalignment at the sub-pixel level are taken and combined after canceling deterioration factors such as the optical system of these images.
- the imaging apparatus shown in FIG. 1 includes an optical imaging unit 11, an imaging unit 12, a moving speed detection unit 13, an imaging timing determination unit 14, and an image processing unit 15.
- the imaging unit 12 is an image sensor such as CCD or CMOS, for example. This imaging device shall be photographed with the photographer holding it. At the time of shooting, light from the subject is imaged on the imaging unit 12 by the optical imaging unit 1 1.
- the moving speed detection unit 1 3 detects the moving speed when the imaging apparatus is displaced when the photographer holds the imaging apparatus by hand, and the information on the moving speed is the imaging timing. Given to decision part 1 4.
- the displacement of the imaging device at this time may be consciously created by the photographer or may occur unconsciously.
- the imaging timing determination unit 14 sends a shooting signal to the imaging unit 12 when the moving speed is lower than a certain value, and the imaging unit 12 receives this signal and performs shooting.
- the captured image information is sent to the image processor 15 Is done.
- the image processing unit 15 uses the transmitted image information for a plurality of frames having a displacement to generate an image having a higher resolution than that at the time of shooting.
- the image processing unit 15 synthesizes a high-resolution image using a plurality of low-resolution images whose imaging positions have displacement.
- the method is described below, assuming that high-resolution processing is performed using, for example, super-resolution technology.
- the moving speed detector 13 detects the moving speed when the imaging device is displaced.
- the configuration of the present invention is not limited to the detection of the moving speed when such an imaging apparatus is displaced.
- the state change when the imaging device is displaced for example, acceleration is detected, the speed is calculated from the history of the direction and magnitude of the acceleration, and imaging is performed when the calculated speed becomes smaller than a certain value. It can be configured.
- the state change is a speed
- examples in which the state change is smaller than a certain value include a case where the speed becomes zero. If the image is taken when the speed reaches 0, a high-definition image without blurring can be obtained.
- the imaging apparatus shown in FIG. 1 has a moving speed detection unit 13 and an imaging timing determination unit 14, and the following procedure (when using a plurality of images to increase the resolution of an image)
- the image capturing method by step) is realized.
- it has a step of detecting a change in the state of the imaging device itself and a step of determining the timing of imaging, and taking an image at an appropriate timing in consideration of the change in the state of the imaging device.
- FIG. 2 is a block diagram showing an example of the image processing unit 15 when the super-resolution technique is used.
- the image processing unit 15 includes an image storage unit 21, a motion estimation unit 22, and a super-resolution processing unit 23.
- Image information taken by the imaging unit 12 is sent to the image storage unit 21.
- the image storage unit 21 uses multiple frames. The image information that was taken for each time is stored.
- the image storage unit 21 sends image information to the motion estimation unit 22 and calculates a relative displacement (hereinafter referred to as motion) based on a single image.
- motion a relative displacement
- the obtained motion is sent to the super-resolution processing unit 23.
- the super-resolution processing unit 23 is provided with image information of a plurality of frames taken from the image storage unit 21.
- the super-resolution processing unit 23 performs super-resolution processing using the information of these motions and the image information captured for each frame, and the image has a higher resolution than the image captured by the imaging unit 12.
- S1 Read one image as a reference for motion estimation, and use this as the reference image.
- S 2 Creates an image sequence in which the reference image is transformed by multiple motions.
- S3 Read one reference image for motion estimation with the base image.
- S4 A similarity value between an image sequence obtained by deforming a plurality of reference images and a reference image is calculated.
- S 5 A discrete similarity map is created using the relationship between the parameters of the deformation motion and the calculated similarity value.
- S 6 The extreme value of the similarity map is searched by interpolating the discrete similarity map created in S 5 to find the extreme value of the similarity map.
- the deformation motion with extreme values is the estimated motion. There are parabola fitting, spline interpolation, and other methods for searching for extreme values in the similarity map.
- S 7 Determine whether motion estimation is performed for all reference images. S8: If motion estimation has not been performed for all reference images, the frame number of the reference number is incremented by 1 and the process returns to S3 to continue reading the next image. Motion estimation is performed on all target reference images, and S 7 If the result of determination is Y, the process is terminated.
- Figure 4 shows a schematic diagram of an example of motion estimation performed by parabolic fitting.
- the vertical axis in Fig. 4 represents the similarity. The smaller the value, the higher the similarity.
- the black circles in Fig. 4 indicate discrete similarity values, and the gray circles indicate extreme values of similarity.
- the curve connecting the discrete similarity values is the interpolated similarity.
- Fig. 5 is a flowchart showing the algorithm for high-resolution image estimation processing. Next, the flowchart of Fig. 5 will be explained.
- S11 Read multiple low-resolution images ⁇ obtained by imaging for use in high-resolution image estimation (where n ⁇ l).
- S12 An initial high-resolution image is created by performing interpolation enlargement processing, assuming any one of a plurality of low-resolution images as a reference frame. This step can be omitted in some cases.
- S 13 Clarify the positional relationship between images based on the motion between the images of the reference frame and other frames obtained in advance by some motion estimation method.
- S14 Obtain the point spread function (PSF) taking into consideration the imaging characteristics such as the optical transfer function (0TF) and CCD aperture. PSF uses, for example, the Gauss function.
- S15 Minimize the evaluation function f (z) based on the information of S13 and S14. However, f (z) has the form shown in the following equation (1).
- y is a low-resolution image
- z is a high-resolution image
- A is an image transformation matrix that represents an imaging system including inter-image motion and PSF.
- the steepest descent method is used to minimize the evaluation function.
- S16 It is determined whether f (z) obtained in S15 is minimized. If f (z) is minimized, the process is terminated and a high-resolution image z is obtained.
- S17 If f (z) has not been minimized, the high-resolution image z is updated, and the process returns to S13.
- the steepest descent method is used to minimize the evaluation function f (z)
- Eq. (4) becomes Eq. (4), and the high resolution image when iterative computation for minimization is repeated n times can be expressed as Eq. (5).
- d (3) dz
- n A ⁇ (Az-y) (4 )
- n the number of iterations and is the contribution rate (weighting factor) of each term.
- FIG. 6 is a block diagram showing an example of the configuration of the super-resolution processing unit 23 when executing the algorithm.
- the super-resolution processing unit 23 includes an interpolation enlargement unit 61, a convolution integration unit 62, a PSF data holding unit 63, an image comparison unit 64, a multiplication unit 65, a pasting addition unit 66, an accumulation addition unit 67, and an update image generation Unit 68, image storage unit 69, iteration calculation determination unit 610, and iteration determination value holding unit 611.
- the interpolation enlargement unit 61 one image as a reference among images of a plurality of frames is given to the interpolation enlargement unit 61, and this image is subjected to interpolation enlargement to create an initial estimated image.
- interpolation enlargement methods include general bilinear interpolation and bicubic interpolation.
- the image subjected to the interpolation enlargement by the interpolation enlargement unit 61 is given to the convolution integration unit 62, and based on the motion for each frame obtained by the motion estimation unit 22.
- the PSF data given by the PSF data holding unit 63 is convolved and integrated at an appropriate coordinate position.
- the interpolated and enlarged image data is simultaneously sent to the image accumulating unit 69 where it is accumulated.
- the image data that has been subjected to the convolution operation by the convolution integration unit 62 is sent to the image comparison unit 64.
- the image comparison unit 64 the captured image given from the image storage unit 21 at an appropriate coordinate position is compared with the image data subjected to the convolution operation.
- the value for each pixel of the PSF data given from the PSF data storage unit 63 is multiplied by the residual.
- the calculation result is sent to the pasting and adding unit 66 and placed at the corresponding coordinate position.
- the image data from the multiplying unit 65 is shifted little by little while overlapping, so the overlapping portions are added.
- the data is sent to the accumulation addition unit 67.
- the accumulation / addition unit 67 accumulates the data sent in sequence until the processing for the number of frames is completed, and sequentially adds the image data for each frame according to the estimated motion.
- the image data added by the accumulation / addition unit 67 is sent to the update image generation unit 68.
- the updated image generation unit 68 is given the image data stored in the image storage unit 69 at the same time as the image data added by the storage addition unit 67, and the two image data are weighted and added. Update image data is generated.
- the generated update image data is given to the iterative operation determination unit 6 1 0, and it is determined whether or not to repeat the operation based on the iterative determination value given from the iterative determination value holding unit 6 1 1.
- the iterative calculation determination unit 610 when the calculation is repeated, the data is sent to the convolution integration unit 62. The series of processes after the convolution integration unit 62 is repeated. If not repeated, the generated image data is output.
- the image output from the iterative calculation determination unit 610 has a higher resolution than the captured image. Further, the PSF data held in the PSF data holding unit 63 needs to be calculated at an appropriate coordinate position during the convolution integration, so the motion estimation unit 22 determines the motion for each frame. It has come to be given.
- the image of each frame is larger than the spatial sampling interval of the imaging unit 12 described in FIG. It is necessary to have information on motion at fine intervals.
- This method uses the movement that occurs when the photographer supports the imaging device by hand to obtain this motion, and estimates the amount of motion with high accuracy from image processing as described above. .
- the speed of movement of the imaging device at this time is faster than the exposure time, the movement of the subject during the exposure period will increase, and motion blur will become prominent in the image. There's a problem.
- the moving speed of the imaging apparatus is detected by the moving speed detector 13 described in FIG. 1, and only when the detected speed is slower than the exposure time of the imaging unit 12, the image is taken. I do.
- the embodiment of the present invention it is required for high resolution of an image without performing precise position control at the time of photographing. Images can be taken and high resolution using multiple images becomes possible.
- FIG. 7 is a block diagram showing an imaging system according to the second embodiment of the present invention.
- the imaging system 10 includes an optical imaging unit 1 1, an imaging unit 1 2, a moving speed detection unit 1 3, an imaging timing determination unit 1 4, an image processing unit 1 5, an imaging start signal generation unit 76, an imaging device
- the moving unit 77 and the imaging device fixing unit 78 are configured.
- As the imaging device fixing unit 78 for example, a tripod can be used.
- the imaging device fixing unit 78 holds or supports the imaging device moving unit 77 and the set of imaging devices denoted by reference numerals 71 to 76.
- the imaging device moving unit 77 includes an imaging device driving unit 8 1 as shown in FIG.
- the imaging start signal given from the imaging start signal generation unit 7 is input to the imaging device driving unit 8 1 in the imaging device moving unit 77, and in response to this, the imaging device driving unit 8 1 moves the imaging device. start.
- the moving speed detector 13 receives the shooting start signal and starts detecting the moving speed.
- the movement of the imaging device 1 by the imaging device moving unit 77 at this time is assumed to move the imaging device only in one linear direction as shown in the explanatory diagram of FIG. 9, for example.
- the imaging device is reciprocated in the direction of the arrow (straight direction) by the imaging device drive unit 8 1 using an elastic member.
- Reference numeral 80 denotes a fixed portion of the imaging device driving portion 8 1 (an inertia member). Instead of the movement in the linear direction in the configuration of FIG. 9, if the angle is small, this may be replaced by the rotational movement (pan) of the imaging device.
- high resolution processing is performed using an image having a motion finer than the pixel interval of the image sensor.
- the moving speed detector 13 detects the moving speed when the imaging device is displaced.
- the configuration of the present invention is not limited to the detection of the moving speed when such an imaging apparatus is displaced.
- the state change when the imaging device is displaced for example, acceleration is detected
- the speed is calculated from the history of the direction and magnitude of the acceleration, and the calculated speed is a constant value. It can be configured to take an image when it becomes smaller than the above.
- the state change is a speed
- examples in which the state change is smaller than a certain value include a case where the speed becomes zero. If an image is taken when the state change becomes 0, a high-definition image without blurring can be obtained.
- the imaging system 10 in FIG. 7 includes a moving speed detection unit 13, an imaging timing determination unit 14, an imaging preparation start signal generation unit 7 6, and an imaging device moving unit 7 7.
- the configuration in FIG. 7 realizes an image capturing method according to the following procedure (step) when increasing the resolution of an image using a plurality of images.
- an image capturing method of this configuration includes a step of detecting a change in the state of the imaging device itself, a step of determining an imaging timing, a step of generating a signal for starting a series of processing related to imaging, Receiving a signal for starting the series of processes and giving a displacement to the imaging device, and imaging when the state change of the imaging device becomes smaller than a certain value.
- FIG. 10 is an explanatory diagram of an example using the imaging device drive unit 8 1 of FIG.
- the imaging device 1 is moved in one diagonal direction. Show the trajectory.
- Fig. 10 (b) By taking a picture as shown in Fig. 10 (b) on this trajectory, it can be seen that an image with a slight shift in the horizontal and vertical directions can be taken efficiently. Therefore, images of each frame are taken at slightly different positions while being displaced obliquely with respect to the imaging device 1, and the above-described image high-resolution processing is performed using the images.
- FIG. 11 shows a configuration diagram of an imaging apparatus according to the third embodiment of the present invention.
- the imaging apparatus shown in FIG. 11 includes an optical imaging unit 11, an imaging unit 12, a moving speed detection unit 13, an imaging timing determination unit 14, an image processing unit 15, an imaging preparation start signal generation unit 116,
- the image sensor moving unit 117 is configured.
- the switch of the shooting start signal generator 116 is turned on. By this operation, a shooting start signal is generated, and the shooting start signal is given to the image sensor moving unit 117 and the moving speed detecting unit 13.
- the image sensor moving unit 117 starts moving the image capturing unit 12, and at the same time, the moving speed detecting unit 13 starts detecting the moving speed of the image capturing unit 12.
- the information on the moving speed is given to the imaging timing determining unit 14, and imaging is performed when the moving speed becomes slower than a certain value.
- An example of the configuration of the imaging unit 12 and the imaging element moving unit 117 is shown in the explanatory diagram of FIG.
- the image sensor moving unit 117 includes an image sensor moving rail 121, an image sensor fixing unit (fixed claw) 122, an image sensor fixing release unit 123, and an image sensor moving spring 124.
- the imaging unit 12 corresponds to the imaging device 112.
- a shooting start signal is given to the image sensor fixing release unit 123, the image sensor fixing unit ⁇ 2 that has fixed the image sensor until then is detached from the image sensor 112, and the inertial force of the image sensor moving spring 124 causes the image sensor 112 to move to the rail. Amplitude motion starts along.
- the moving speed detecting unit 13 detects the moving speed when the image pickup unit 12 is displaced.
- the configuration of the present invention is not limited to the detection of the moving speed when such an imaging apparatus is displaced.
- the state change when the imaging device itself or the imaging unit is displaced, such as acceleration, is detected, and the speed is calculated from the history of the direction and magnitude of the acceleration.
- examples in which the state change is smaller than a certain value include a case where the speed becomes zero. If the image is taken when the speed reaches 0, a high-definition image without blurring can be obtained.
- an imaging system different from that in FIG. 7 can be configured.
- the photographer can increase the resolution of the captured image without holding the imaging device in his hand.
- the imaging apparatus of FIG. 11 includes a moving speed detection unit 13, an imaging timing determination unit 14, an image processing unit 15, an imaging preparation start signal generation unit 116, and an imaging element moving unit 117.
- the configuration shown in Fig. 11 realizes an image capturing method according to the following procedure (steps) when increasing the resolution of an image using multiple images. That is, a step for generating a signal for starting a series of processes related to imaging, a step for receiving a signal for starting a series of processes for imaging and spatially displacing the image sensor, and a state change of the image sensor A step of detecting and a step of determining an imaging timing, and taking an image at an appropriate timing in consideration of a state change of the image sensor.
- the imaging apparatus includes an optical imaging unit 11, an imaging unit 12, an imaging timing determination unit 14, an image processing unit 15, an imaging preparation start signal generation unit 116, an image sensor moving unit 117, a relative speed detection unit 133,
- the optical element moving unit 138 is configured.
- the imaging element moving unit 117 and the optical element moving unit 138 each start displacement.
- the relative speed detector 133 starts detecting the relative speed between the image sensor and the optical element.
- the imaging element moving unit 117 and the optical element moving unit 138 displace the imaging unit 12 and the optical imaging unit 11 with different initial velocities and accelerations in the same direction.
- Relative velocity information of the imaging unit and optical imaging unit 11 in the relative velocity detection unit 133 Is detected and sent to the imaging timing determination unit 14, and the imaging speed is given to the imaging unit 12 at a timing when the relative speed is smaller than a certain value, and imaging is performed.
- the imaging unit 12 and the optical imaging unit 11 have different accelerations, there is a moment when the relative velocity becomes 0. Therefore, the timing at or near that moment is detected, and the imaging timing determination unit 14 takes a picture. Will be performed.
- V is the speed of the optical imaging unit 11 at time t
- v 2 is the speed of the imaging unit 12 at time t
- ⁇ ⁇ 1 is optical
- ⁇ ⁇ 2 is the initial velocity of the imaging unit 12
- a is the acceleration of the optical imaging unit 11
- a 2 is the acceleration of the imaging unit 12
- V 2 V 02 + al (7) V0! One V 02
- each of the above parameters and the position at which the movement starts are set so that the positional relationship between the two can be captured, that is, the optical image of the subject is formed on the imaging element.
- FIG. 14 is a block diagram of an image processing unit 15 according to the fourth embodiment of the present invention, and has the same configuration as FIG. From the super-resolution processing unit 23, images taken by the imaging unit 12 An image Q with higher resolution is generated and output.
- FIG. 15 is an explanatory diagram showing an example of the positional relationship between the optical element and the image sensor in the image pickup apparatus according to the fourth embodiment of the present invention.
- Fig. 15 (a) is an explanatory diagram showing an example of the positional relationship between the optical element and the image sensor in the image pickup apparatus according to the fourth embodiment of the present invention.
- the image sensor 2 moves linearly in the direction of arrow La from the position shown in FIG. 15 (a) toward the imaging optical system 3.
- the imaging optical system 3 (optical element) moves linearly in the direction of the arrow Lb.
- the La direction and the L b direction are the same direction and are substantially perpendicular to the optical axis. Therefore, the image sensor and the optical imaging means can be displaced in one linear direction substantially perpendicular to the optical axis.
- the La direction and the L b direction are the same direction, and the imaging device 2 and the optical device have different accelerations. Therefore, at the time of imaging, the imaging device 2 is connected as shown in Fig. 15 (b). Arranged at the position of the image optical system 3.
- the relative change between the image pickup element and the optical element is detected as in the third embodiment example. Therefore, as described in the first embodiment, a plurality of images with little image blur can be acquired and high resolution can be achieved regardless of the method for holding and fixing the imaging device.
- the relative speeds of both the image sensor and the optical element are detected from the initial speed and acceleration. However, when a spatial displacement is applied to the image sensor and the optical element, It is also possible to adopt a configuration that detects relative changes other than acceleration of both.
- the imaging apparatus shown in FIG. 13 includes an imaging timing determining unit 14, an imaging preparation start signal generating unit 116, an imaging element moving unit 117, a relative speed detecting unit 133, and an optical element moving unit 138.
- the configuration shown in Fig. 13 realizes an image capturing method according to the following procedure (steps) when increasing the resolution of an image using multiple images.
- a step for generating a signal for starting a series of processing related to imaging a step for spatially displacing the imaging device, and a state in which the imaging device is partially or entirely different in the same direction as the imaging device
- a step of detecting relative changes between the imaging device and the imaging optical system and a step of determining the timing of imaging. Taking relative changes into account, perform imaging at the optimum timing.
- FIG. 16 shows a configuration diagram of an imaging apparatus according to the fifth embodiment of the present invention.
- the imaging apparatus according to the present embodiment includes an optical imaging unit 11, an imaging unit 12, an imaging timing determination unit 14, an image processing unit 15, an imaging preparation start signal generation unit 116, an imaging element moving unit 117, and an optical element.
- the moving unit 138, the relative speed detecting unit 163, and the motion measuring unit 169 are configured.
- the image processor 15 performs high-resolution processing.
- the configuration of the image processing unit 15 includes an image storage unit 171 and a super-resolution processing unit 172, as shown in the configuration diagram of FIG. Image data information from the imaging unit 12 is given to the image storage unit 171.
- the motion data measured by the motion measuring unit 169 is input to the super-resolution processing unit 172.
- the present embodiment does not require any means for estimating the position information in the image processing unit 15, so that the circuit scale can be reduced.
- the motion is actually measured rather than calculated from the image and estimated, it is possible to obtain accurate relative position information regardless of the type of image.
- an imaging system different from that in FIG. 7 can be configured.
- the photographer can increase the resolution of the captured image without holding the imaging device in his hand.
- the imaging device shown in FIG. 16 includes an imaging dimming determination unit 14, an image processing unit 15, an imaging preparation start signal generation unit 1 1 6, an image sensor moving unit 1 1 7, and a motion measurement unit 1 6 9. It has.
- the configuration shown in Fig. 16 realizes an image capturing method according to the following procedure (step) when increasing the resolution of an image using multiple images. That is, a step for generating a signal for starting a series of processes related to imaging, a step for spatially displacing the image sensor, and a part or the whole of the imaging optical system in the same direction as the image sensor.
- a step of applying a spatial displacement in different states, a step of detecting a relative change between the imaging device and the imaging optical system, a step of determining the timing of imaging, and the imaging device at the time of shooting The step of measuring the spatial position of the image sensor is performed, and imaging is performed at an optimal timing in consideration of the relative change between the image sensor and the optical system.
- FIG. 18 is a block diagram showing an imaging apparatus according to the sixth embodiment of the present invention.
- This imaging device has an optical imaging unit 1 1, an imaging unit 1 2, an imaging timing Configuration unit 14, image processing unit 15, imaging start signal generation unit 116, image sensor moving unit 117, optical element moving unit 138, relative speed detecting unit 163, motion measuring unit 169, and high resolution determining unit 1810 Has been.
- the motion measurement unit 169 measures the position of the spatial imaging unit 12 when each image is taken, and sends this information to the high resolution determination unit 1810.
- the high resolution determination unit 1810 determines whether or not the desired high resolution processing can be performed by the image processing unit 15 based on the sent information.
- a high resolution processing start signal is sent to the image processing unit 15, and the image processing unit 15 performs high resolution processing.
- the criterion for determining whether high resolution is possible is the uneven distribution of motion in the image.
- FIG. 19 is an explanatory diagram showing an example of motion unevenness suitable for high-resolution processing and uneven distribution suitable for high-resolution processing.
- Fig. 19 (a) it can be said that it is suitable for high resolution because it has motions distributed almost evenly within the pixel interval of the image sensor.
- Fig. 19 (b) since motion is unevenly distributed, it can be said that it is not suitable for high resolution processing.
- the degree of variation here depends on the image quality after synthesis. If you want to synthesize higher-quality images, it is better to have less uneven distribution, that is, even dispersion.
- FIG. 20 is an explanatory diagram for the determination of the region to which the motion belongs. For example, when enlarging an image twice vertically and horizontally, as shown in Fig. 20 (a), four areas I, II, It would be sufficient if there were images with motions belonging to III and IV, respectively. Therefore, the closer the number of images belonging to each region is, the more ideal it is for high resolution processing. Similarly, if you want to enlarge to 3 times in the vertical and horizontal directions, as shown in Fig. 20 (b), there are 9 areas divided into 3 vertical and horizontal areas. It is desirable that the number of images belonging to the region is close to equal.
- the high resolution determination unit 1810 includes a motion information holding unit 2101, an affiliation area calculation unit 2102, an affiliation area count 2103, and a comparison / determination unit 2104.
- the motion information of each frame image given from the motion measurement unit 169 is given to the motion information holding unit 2101, where the information for each frame is held and the motion information holding unit
- the information held in 2101 is given to the belonging area calculation unit 2102.
- the affiliation area calculation unit 2102 obtains the horizontal and vertical coordinate positions of the photographed image to be obtained from the given motion information, and determines to which area in FIG. 20 (a) the image belongs.
- the low-resolution image (captured) is based on the center position of the target pixel, A, B, C, and D of the low-resolution image (captured image) for which the region to which O belongs is to be obtained If there is no movement between the two images at the center position of the pixel in the image (Fig. 20 (a)), the regions I, II, III and IV are
- Zone II x a ⁇ x ⁇ x b , y b ⁇ y.
- Y c Region III x b ⁇ x ⁇ x c , y b ⁇ y ⁇ y c
- ⁇ (Shooting image) is determined to be in Area IV.
- the result determined as described above is given as a signal to the county 2103 for each affiliation area.
- the counter 2103 for each affiliation area increases the force count value by one.
- the comparison / determination unit 2104 generates a high-resolution signal and gives this high-resolution signal to the image processing unit 15 .
- the comparison / determination unit 2104 does not generate a high-resolution signal because it is not suitable for high-resolution when the area where the counter value is 0 occupies more than half of the whole. This high resolution signal is sent to the image processing unit 15.
- the image processing unit 15 inputs this high-resolution processing start signal to the signal confirmation unit 221.
- the signal confirmation unit 221 is provided with captured image information from the image storage unit 21 and motion information from the motion measurement unit 169.
- the signal confirmation unit 221 confirms that the three signals of the high-resolution processing start signal, image information, and motion information are ready, it sends the motion information and captured image information to the super-resolution processing unit 23, Super-resolution processing is started in the resolution processing unit 23. If the three signals are not available, the signal confirmation unit 221 issues an error signal to notify the photographer, and the motion information and captured image information are transmitted without starting the super-resolution processing. Do not send to 23.
- the motion estimation unit U serves as a substitute for the motion measurement unit 169 in FIG. It is good.
- This embodiment allows you to take pictures for higher resolution. Since it can be judged whether the motion distribution of the shadowed image is suitable for high resolution, high-resolution high-resolution images can be synthesized with high probability when high-resolution is performed.
- the configuration in which the signal confirmation unit 221 as described above issues an error signal to notify the photographer and does not start the super-resolution processing is not limited to the configurations in FIGS.
- the configuration shown in Fig. 1 to Fig. 17 is also applicable.
- FIGS. 18 to 23 show the configuration of the imaging device, but an imaging system can also be configured by fixing these imaging devices to the imaging device fixing unit 78 described in FIG. .
- the imaging devices shown in FIGS. 18 and 22 include an optical imaging unit 11, an imaging unit 12, an imaging timing determination unit 14, an image processing unit 15, an imaging start signal generation unit 116, and an imaging element movement.
- the configurations shown in Fig. 18 and Fig. 22 realize an image capturing method according to the following procedure (steps) when increasing the resolution of an image using a plurality of images.
- a step for generating a signal for starting a series of processing related to imaging, a step for spatially displacing the image sensor, and a part or the whole of the imaging optical system differ in the same direction as the image sensor.
- a step of spatially changing the state, a step of detecting a relative change between the imaging device and the imaging optical system, a step of determining the timing of imaging, and the imaging device at the time of shooting It has a step to estimate the spatial position and a step to determine whether or not the resolution of the image can be increased, and to notify the photographer.
- the relative change between the image sensor and the optical system is a constant value. When it becomes smaller, image is taken.
- a plurality of images with pixel shifts can be taken without the need for an accurate alignment mechanism of an optical system or an image sensor, and these images were used to capture images. It is possible to provide an imaging device and an imaging system capable of synthesizing an image having a resolution higher than that of the image, and an image capturing method.
- the present invention it is possible to capture a plurality of images with a pixel shift without requiring an accurate alignment mechanism of the optical system or the image sensor, and using these images, the resolution is higher than the captured image. Images can be combined.
- Patent Document 1
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
- Studio Devices (AREA)
Abstract
Description
Claims
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP05804081A EP1816854A4 (en) | 2004-11-15 | 2005-11-15 | PICTURE DEVICE, PICTURE SIZING SYSTEM AND PHOTOGRAPHIC PROCESSING |
US11/667,763 US20070296829A1 (en) | 2004-11-15 | 2005-11-15 | Imaging Device, Imaging System and Photography Method of Image |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2004330252A JP4311668B2 (ja) | 2004-11-15 | 2004-11-15 | 撮像装置と撮像システムおよび画像の撮影方法 |
JP2004-330252 | 2004-11-15 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2006052030A1 true WO2006052030A1 (ja) | 2006-05-18 |
Family
ID=36336682
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2005/021316 WO2006052030A1 (ja) | 2004-11-15 | 2005-11-15 | 撮像装置と撮像システムおよび画像の撮影方法 |
Country Status (4)
Country | Link |
---|---|
US (1) | US20070296829A1 (ja) |
EP (1) | EP1816854A4 (ja) |
JP (1) | JP4311668B2 (ja) |
WO (1) | WO2006052030A1 (ja) |
Families Citing this family (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2001227B1 (en) * | 2006-03-30 | 2016-07-20 | NEC Corporation | Image processing device, image processing system, image processing method and image processing program |
JP4757185B2 (ja) * | 2006-12-20 | 2011-08-24 | キヤノン株式会社 | 光学機器 |
DE102007037726B4 (de) | 2007-08-09 | 2010-07-08 | Lavision Gmbh | Verfahren zur berührungslosen Messung von Verformungen einer Oberfläche eines Messobjektes |
JP4758493B2 (ja) | 2009-04-03 | 2011-08-31 | シャープ株式会社 | 携帯端末装置、撮像画像処理システム、携帯端末装置の制御方法、プログラムおよび記録媒体 |
JP4991887B2 (ja) * | 2010-01-13 | 2012-08-01 | シャープ株式会社 | 撮像画像処理システム、撮像画像処理システムの制御方法、プログラムおよび記録媒体 |
WO2011148851A1 (ja) * | 2010-05-28 | 2011-12-01 | ソニー株式会社 | 撮像装置及び撮像方法 |
DE102010053458B4 (de) * | 2010-12-03 | 2023-06-22 | Testo SE & Co. KGaA | Verfahren zur Aufbereitung von IR-Bildern und korrespondierende Wärmebildkamera |
JP6136085B2 (ja) * | 2011-10-05 | 2017-05-31 | ソニー株式会社 | 画像取得装置、画像取得方法、およびコンピュータプログラム |
TW201316240A (zh) * | 2011-10-06 | 2013-04-16 | Rich Ip Technology Inc | 利用圖形用戶介面影像之觸碰處理方法及系統 |
JP5941758B2 (ja) * | 2012-06-06 | 2016-06-29 | オリンパス株式会社 | 内視鏡装置 |
JP5789765B2 (ja) | 2013-06-07 | 2015-10-07 | パナソニックIpマネジメント株式会社 | 画像取得装置、画像取得方法、およびプログラム |
JP2016052116A (ja) | 2014-08-28 | 2016-04-11 | パナソニックIpマネジメント株式会社 | 画像処理装置、画像処理方法およびコンピュータプログラム |
JP5893712B1 (ja) * | 2014-11-04 | 2016-03-23 | オリンパス株式会社 | 撮像装置、撮像方法、処理プログラム |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH04262681A (ja) * | 1991-02-18 | 1992-09-18 | Aiwa Co Ltd | 固体撮像装置 |
JP2001136350A (ja) * | 1999-11-02 | 2001-05-18 | Seiko Epson Corp | 画像入力装置、画像入力システム、および画像入力方法、並びにそのための記録媒体 |
JP2004015341A (ja) * | 2002-06-05 | 2004-01-15 | Olympus Corp | 電子的撮像装置 |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3690234A (en) * | 1971-01-18 | 1972-09-12 | Us Air Force | Apparatus for taking photographs at times of minimum image motion |
DE3477119D1 (en) * | 1983-06-15 | 1989-04-13 | Toshiba Kk | Solid state image sensor with high resolution |
US5214513A (en) * | 1991-07-11 | 1993-05-25 | Eastman Kodak Company | Apparatus for imparting motion to a solid-state image sensor |
JP3934786B2 (ja) * | 1997-05-27 | 2007-06-20 | 株式会社リコー | 画像入力装置 |
US6639625B1 (en) * | 1997-07-16 | 2003-10-28 | Minolta Co., Ltd. | Image sensing device |
US6650704B1 (en) * | 1999-10-25 | 2003-11-18 | Irvine Sensors Corporation | Method of producing a high quality, high resolution image from a sequence of low quality, low resolution images that are undersampled and subject to jitter |
JP2002218328A (ja) * | 2001-01-19 | 2002-08-02 | Ricoh Co Ltd | 画像入力装置、画像入力方法、およびその方法を実行するためのプログラムを格納したことを特徴とするコンピュータが読み取り可能な記録媒体 |
CA2349828A1 (en) * | 2001-06-06 | 2002-12-06 | Andrew M. Zador | Method, apparatus, and system for extracting denoised, high-resolution images, texture, and velocity from a lower-resolution, noisy detector |
WO2003032627A1 (en) * | 2001-09-11 | 2003-04-17 | Deltapix Aps | A method and apparatus for producing a high resolution image |
US7212230B2 (en) * | 2003-01-08 | 2007-05-01 | Hewlett-Packard Development Company, L.P. | Digital camera having a motion tracking subsystem responsive to input control for tracking motion of the digital camera |
US7447382B2 (en) * | 2004-06-30 | 2008-11-04 | Intel Corporation | Computing a higher resolution image from multiple lower resolution images using model-based, robust Bayesian estimation |
-
2004
- 2004-11-15 JP JP2004330252A patent/JP4311668B2/ja not_active Expired - Fee Related
-
2005
- 2005-11-15 US US11/667,763 patent/US20070296829A1/en not_active Abandoned
- 2005-11-15 EP EP05804081A patent/EP1816854A4/en not_active Withdrawn
- 2005-11-15 WO PCT/JP2005/021316 patent/WO2006052030A1/ja active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH04262681A (ja) * | 1991-02-18 | 1992-09-18 | Aiwa Co Ltd | 固体撮像装置 |
JP2001136350A (ja) * | 1999-11-02 | 2001-05-18 | Seiko Epson Corp | 画像入力装置、画像入力システム、および画像入力方法、並びにそのための記録媒体 |
JP2004015341A (ja) * | 2002-06-05 | 2004-01-15 | Olympus Corp | 電子的撮像装置 |
Non-Patent Citations (1)
Title |
---|
See also references of EP1816854A4 * |
Also Published As
Publication number | Publication date |
---|---|
EP1816854A4 (en) | 2010-08-04 |
US20070296829A1 (en) | 2007-12-27 |
EP1816854A1 (en) | 2007-08-08 |
JP2006140885A (ja) | 2006-06-01 |
JP4311668B2 (ja) | 2009-08-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2006052030A1 (ja) | 撮像装置と撮像システムおよび画像の撮影方法 | |
JP4926920B2 (ja) | 防振画像処理装置及び防振画像処理方法 | |
JP4607048B2 (ja) | 光学モーションセンサをジャイロスコープとして使用する撮像装置 | |
US8760526B2 (en) | Information processing apparatus and method for correcting vibration | |
US8379096B2 (en) | Information processing apparatus and method for synthesizing corrected image data | |
WO2013005316A1 (ja) | 画像処理装置、画像処理方法及び画像処理プログラム | |
JP5532297B2 (ja) | 画像処理装置、画像処理方法、及び、プログラム | |
JP3380402B2 (ja) | 撮像装置 | |
KR20070033272A (ko) | 흐림 현상 감소 시스템을 갖는 촬상 장치 | |
JP4869572B2 (ja) | 撮像装置 | |
JP2024069348A (ja) | 撮像装置、撮像装置の作動方法、及びプログラム | |
JP2009069748A (ja) | 撮像装置およびその自動焦点調節方法 | |
JP5424068B2 (ja) | 画像処理装置、画像処理方法、画像処理プログラム及び記憶媒体 | |
JP4529804B2 (ja) | 画素ずらし画像の生成機能を有する電子カメラおよび撮像方法 | |
JP4064001B2 (ja) | カメラ | |
JPH11155096A (ja) | 撮像装置、コンピュータ読み取り可能な記憶媒体及び画像処理装置 | |
JP6704803B2 (ja) | 撮像装置及びその制御方法 | |
JP7159486B2 (ja) | 撮像装置 | |
JP2012124939A (ja) | 撮像装置及び撮像装置の制御方法 | |
JP5401696B2 (ja) | 画像処理装置、画像処理方法及び画像処理プログラム | |
JP2018050147A (ja) | 撮像装置、撮像方法およびプログラム | |
JP4665541B2 (ja) | 撮像装置および手ブレ補正方法 | |
JP2021081611A (ja) | 撮像装置、撮像装置の制御方法およびプログラム | |
TW202007138A (zh) | 攝影裝置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS KE KG KM KN KP KR KZ LC LK LR LS LT LU LV LY MA MD MG MK MN MW MX MZ NA NG NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SM SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU LV MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2005804081 Country of ref document: EP |
|
WWP | Wipo information: published in national office |
Ref document number: 2005804081 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 11667763 Country of ref document: US |
|
WWP | Wipo information: published in national office |
Ref document number: 11667763 Country of ref document: US |