JP4401949B2

JP4401949B2 - Moving picture imaging apparatus and moving picture imaging method

Info

Publication number: JP4401949B2
Application number: JP2004363393A
Authority: JP
Inventors: 英夫河原
Original assignee: キヤノン株式会社
Priority date: 2004-11-26
Filing date: 2004-12-15
Publication date: 2010-01-20
Anticipated expiration: 2024-12-15
Also published as: US20060132612A1; JP2006179966A

Description

The present invention provides a video imaging apparatus such as a video camera that continuously outputs captured images at a predetermined cycle such as a field period, and after taking a plurality of images for each cycle, The present invention relates to a moving image capturing apparatus and a moving image capturing method that obtain a single image constituting a moving image by superimposing a plurality of images.

In conventional camera systems capable of shooting moving images, such as video cameras, automation and multi-functionalization are achieved in all respects, as seen in functions such as auto exposure (AE) and auto focus (AF). Can be easily performed. In recent years, downsizing of an image pickup apparatus and high magnification of an optical system have been achieved, and accordingly, there has been a tendency for a problem of deterioration in the quality of a photographed image due to shaking of the apparatus to become apparent. As countermeasures, various shake correction functions for correcting the shake of a photographed image caused by such a shake (hand shake) of the apparatus have been proposed, and it is further improved by installing such a shake correction function in the imaging apparatus. Can be easily taken.

For example, with regard to a shake correction function mounted on a video camera, a so-called optical shake correction method (for example, refer to Patent Document 1) that optically corrects shake, and an electronic device that performs shake correction by electrical processing. A formula shake correction method (see, for example, Patent Document 2) has been proposed.

Here, an outline of the electronic shake correction will be described with reference to FIG. 8A to 8C, an area indicated by 100 indicates an entire imaging area acquired by an imaging element (for example, a CCD sensor or a CMOS sensor). An area within a broken line 101 is a cutout frame that is actually converted into a standard video signal as a video signal (captured image) and output from the entire imaging area 100. Reference numeral 102 denotes a main subject photographed by the photographer. FIG. 8C shows a state where an image based on the standard video signal at this time is displayed on the monitor. In FIG. 8C, reference numeral 103 denotes a video area of a monitor that reproduces a video signal, and reference numeral 102 ′ denotes a main subject reproduced on the monitor 103. In other words, the video area 103 is reproduced on the monitor by outputting a part of the entire imaging area 100 acquired by the imaging element excluding its periphery as a standard video signal.

Next, FIG. 8B will be described. FIG. 8B shows a change in both images when a photographer who shoots the subject 102 shakes the video camera in the lower left direction in the figure indicated by arrows 104, 104 ′, 104 ″. In this state, the main subject 102 moves in the upper right direction in the figure indicated by the arrow 105 over the entire imaging region 100 of the imaging device, and in this state, the same position as the cutout frame 101 in FIG. When the clip is cut out using the cutout frame indicated by 101 ′ (the same coordinate position in the entire imaging region 100 plane), a video signal in which the subject 102 has moved by the vector amount indicated by the arrow 105 is generated. The amount of shaking of the video camera is detected when an image is picked up by the above method, and the position of the clipping frame is determined using the amount of displacement 106 of the image obtained from this amount of shaking, that is, the shaking correction target value. By moving from 101 'to the position of the broken line frame indicated by 101 ", the movement of the subject 102 image due to shaking is canceled out, so that the image shown in FIG. 8C can be obtained. Electronic shake correction uses this principle to realize image shake correction. That is, by electrically correcting the shake of the image pickup means, the shake correction of the moving image is performed, and a picked-up image with less blur can be taken.

A video camera provided with a shake correction means using the electronic shake correction method will be briefly described with reference to FIG. 9. The photographing light (light beam) incident through the lens 150 constituting a part of the optical system is as follows. An image is formed on an image sensor 151 such as a CCD, and converted into electric charges (electrical signals). An electric signal is read from the image sensor 151 based on a read signal output at a predetermined timing by a timing generator (TG) 152, and the electric signal is converted into a standard video signal such as NTSC by a signal processing circuit 153. It is output via the video output terminal 154.

On the other hand, the shake of the video camera is detected as an angular velocity by an angular velocity sensor 155 such as a vibration gyro in accordance with the timing of imaging, and is shaken by a shake correction amount calculation circuit 156 having a DC cut filter, an amplifier, and an integration circuit (not shown). A correction amount is calculated. More specifically, only the AC component, that is, the vibration component, is extracted from each input velocity signal by a DC cut filter, amplified by an amplifier, and further integrated by an integration circuit to convert the angular velocity into angular displacement. Then, a shake correction amount is calculated based on the obtained angular displacement. The calculated shake correction amount is converted into the movement amount of the pixel of the image sensor 151 by the read position control circuit 157, and the read (cutout) position is changed based on the move amount, and the shake correction amount is based on the shake correction amount. Thus, the readout timing of the image sensor 151 is changed.

In addition, the correction method using the optical shaking method that has been conventionally employed will be described. The configuration of the angular velocity sensor, the DC cut filter, the amplifier (amplifier), and the integrating circuit that are equivalent to the shaking detection means in the optical shaking method is the same. In the same manner as the above-described electronic shake correction method, an angular displacement is obtained by the shake correction amount calculation circuit 156, and based on this angular displacement, the optical shake is optically shifted to cancel the shake. For example, a method has been proposed in which the optical axis of incident light to the image sensor is shifted by displacing a shake correction lens within a plane orthogonal to the optical axis. By optically canceling the shake of the video camera as described above, optical shake correction is always performed during shooting (during exposure), and a captured image without blur can be taken.

JP-A-9-181959 JP-A-10-178582

However, the video camera provided with the shake correction means by the electronic shake correction method has the following problems. That is, since the shake correction during the exposure period (= during accumulation time) of the image sensor 151 is not performed, the shake during the exposure period in which the image sensor 151 accumulates charges (images) cannot be removed, and the shake is not performed. There was a limit to the accuracy of correction. The miniaturization of video cameras and the increase in the magnification of optical systems tend to progress further. In order to further improve the correction accuracy, the present inventors have studied to shorten the accumulation time for the purpose of reducing fluctuation during the exposure period. It is made by. In this case, the correction accuracy is improved as the influence of shaking during the exposure period is reduced, but the movement of the subject is also reduced at the same time, so that the depiction of the movement of the subject in the video movie becomes unnatural. There is.

To describe this problem with a specific example, for example, assuming that a car crosses the background of the main subject, the main subject and the background are corrected by the electronic shake correction, At the same time, if you focus on a car that crosses the background, the blur, that is, the movement of the car will also be reduced (reduced) by shortening the exposure time, and if it is viewed as a movie, the smooth movement due to the blur of the subject will be impaired. There may be a strange image in which a stationary car appears at a position accompanying the movement of the car for each field. On the other hand, if an attempt is made to reproduce the smooth movement of the vehicle by increasing the number of acquired images, the amount of data becomes enormous, and if a large-capacity storage medium is not attached, the time that can be imaged is shortened.

Further, in the optical shake correction method, there is a limit to downsizing because it is necessary to support the correction lens and to move the lens itself with high accuracy.

The present invention has been made in view of the above problems, and in an imaging apparatus capable of outputting an image at a predetermined cycle that repeatedly occurs, such as a field period, and shooting a moving image, for example, the continuous movement of a moving subject. One object is to provide a moving image capturing apparatus and a moving image capturing method capable of performing shake correction of a captured image without impairing reproducibility.

The moving image pickup apparatus of the present invention detects an angular velocity each time an image sensor that outputs a plurality of images in one vertical synchronization period and the plurality of images output from the image sensor in the one vertical synchronization period are acquired. An angular velocity sensor, a shake correction amount calculating means for calculating a shake correction amount based on the angular velocity corresponding to each of the plurality of images output from the image sensor during the one vertical synchronization period, and the one vertical synchronization period A storage unit for storing the shake correction amount corresponding to each of the plurality of images output from the image sensor, and converting the coordinates of the plurality of images based on the shake correction amount stored in the storage unit. Then, by performing image synthesis, the image processing apparatus includes a synthesis unit that generates one synthesized image per the vertical synchronization period .

In the moving image imaging method of the present invention , an angular velocity is detected each time an imaging step of outputting a plurality of images in one vertical synchronization period and each of the plurality of images output from the image sensor in the one vertical synchronization period are acquired. An angular velocity detection step, a shake correction amount calculation step of calculating a shake correction amount based on the angular velocity corresponding to each of the plurality of images output from the image sensor in the one vertical synchronization period, and the one vertical synchronization period Storing a shake correction amount corresponding to each of the plurality of images output from the image sensor, and converting coordinates of the plurality of images based on the shake correction amount stored in the storage step. Then, by performing image composition, a composition step of generating one composite image per one vertical synchronization period is included.

According to the present invention, in an imaging apparatus that outputs captured images continuously at a predetermined period, a plurality of images are captured during a predetermined period, and the fluctuation of the image is corrected based on the amount of fluctuation of each image. In addition, by correcting and synthesizing the captured image without impairing the reproducibility of the continuous movement of the subject by generating a single image by superimposing and synthesizing at least two image data after shake correction. Can do.

Preferred embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

<First Embodiment>
First, a first embodiment of the present invention will be described. FIG. 1 is a block diagram showing a schematic configuration of a camera system (imaging device) in the present embodiment. In the figure, as a configuration of an imaging unit provided in the camera system, reference numeral 200 denotes a lens that constitutes a part of an optical system, which is detachably attached to the camera body. Reference numeral 201 denotes an image sensor (e.g., a CCD sensor or a CMOS sensor) which is a photoelectric conversion unit, and 202 denotes a camera signal pre-processing circuit that converts an electrical signal from the image sensor 201 into an image signal. Reference numeral 203 denotes an image memory as an image storage means for storing an image signal (image data) output from the camera signal preprocessing circuit 202, and 204 denotes a conversion of the two-dimensional coordinates of the image signal read from the image memory 203. It is a coordinate conversion circuit as a coordinate conversion unit that constitutes a shake correction means for performing shake correction.

205 is an image synthesizing circuit as an image synthesizing means for synthesizing image signals acquired at different timings whose coordinates have been transformed by the coordinate transformation circuit 204, and 206 is a synthesized video signal, for example, a standard video signal such as NTSC. It is a camera signal processing circuit as a signal processing means for converting, and the converted standard video signal is configured to be output as a video image at a predetermined cycle, for example, 1/60 second interval, via the video output terminal 207. Yes. Although not shown in FIG. 1, the lens 200 is configured as a lens unit by combining a plurality of lenses, and the distance between these lenses can be changed by a drive motor (for example, a vibration motor or a stepping motor). The zoom position and focus can be adjusted, and the focal length can be changed.

Further, as a configuration of the shake correction mechanism provided in the camera system, reference numeral 208 denotes an angular velocity sensor such as a vibration gyro provided as a shake detection unit for detecting the shake amount of the camera system provided in the exterior body of the camera system. Reference numeral 209 denotes a shake correction amount calculation circuit as a shake correction amount calculation unit that calculates a shake correction amount based on an angular velocity signal (angular velocity information) output from the angular velocity sensor 208, and 210 is calculated by the shake correction amount calculation circuit 209. It is a shake correction amount memory as a shake correction amount storage unit for storing the shake correction amount. These shake detection unit, shake correction amount calculation unit, and shake correction amount storage unit constitute shake correction amount detection means. Reference numeral 211 denotes a timing generator (hereinafter abbreviated as “TG”) that generates a reference signal that is the basis of the operation timing of the image pickup apparatus. The image pickup device 200, the image memory 203, the coordinate conversion circuit 204, the image composition circuit 205, A reference signal that can serve as a trigger for starting the operation is supplied to the shake correction amount memory 210.

Here, the shake correction amount calculation circuit 209 will be described in detail with reference to FIG. FIG. 2 is a block diagram showing the internal configuration of the correction amount calculation circuit 209. In the figure, reference numeral 300 denotes a DC cut filter that constitutes a vibration signal extraction unit that blocks the direct current component in the angular velocity signal output from the angular velocity sensor 208 and passes only the alternating current component, that is, the vibration component. The vibration signal extraction unit is not limited to the DC cut filter 300, and a high-pass filter (HPF) that cuts off a signal in a predetermined band may be used. Reference numeral 301 denotes an amplifier that amplifies the angular velocity signal that has passed through the DC cut filter 300 to an appropriate sensitivity. Reference numeral 302 denotes an A / D converter that converts the angular velocity signal output from the amplifier 301 into a digital signal. Reference numeral 303 denotes a high-pass filter (hereinafter abbreviated as “HPF”) that blocks low-frequency components in the digital output from the A / D converter 302, and has a function capable of varying the characteristics in an arbitrary band.

Reference numeral 304 denotes an integrator circuit that integrates the angular velocity signal (angular displacement information) from the HPF 303 and outputs an angular displacement signal, and has a function capable of varying the characteristics in an arbitrary band. A pan / tilt determination circuit 305 determines whether the camera system is in a panning / tilting state based on the angular velocity signal and the angular displacement signal output from the integrator circuit 304. The circuit 305 is configured to perform panning control, which will be described later, based on information on angular velocity and angular displacement (levels of the angular velocity signal and the angular displacement signal). The panning / tilting state here means a state in which the operator of the camera system is photographing while intentionally performing panning or tilting as camera work. Control that limits the range of shake correction in order to prevent image distortion and to ensure quick response in the direction intended by the operator is called panning control.

The A / D converter 302, HPF 303, integrator circuit 304, and pan / tilt determination circuit 305 are actually constituted by a microcomputer to constitute a correction amount calculation unit. Then, the angular displacement signal obtained by the correction amount calculation unit based on the angular velocity signal detected by the shake detection unit is calculated by a shake correction target value, for example, correction amount = focal length × tan (correction angle) in later control. Value.

The panning control will be described in more detail. When the angular velocity signal output from the A / D converter 302 and the angular displacement signal output from the integrator circuit 304 are respectively input to the pan / tilt determination circuit 305, Even if the angular velocity is equal to or greater than a predetermined threshold value or the angular velocity is less than the predetermined threshold value, it is determined that the panning / tilting state is present when the angular displacement obtained by integrating the angular velocity signal is equal to or greater than the predetermined threshold value. In such a case, the low frequency cutoff frequency of the HPF 303 is shifted to the high frequency side, and the characteristics are changed so that the shake correction system does not respond to the low frequency. If it is determined that further in the state of the panning tilting the corrected position of the image correcting means centered to the center of the range of movement gradually, the constant is displaced to shorten consisting direction when the integral characteristic of the integrator circuit 304, integration It performs panning control so accumulated value becomes a criteria value (possible values in a state not detecting swing) the vessel circuit 304. During this time it has also performed the detection of the angular velocity signal and the angular displacement signal, when it is released from the state of panning tilting the operation to expand the correction range shake by lowering the cutoff frequency of the low frequency again Done and exit from panning control.

This operation will be described with reference to the flowchart of FIG. The following flowchart is executed by a program incorporated in the shake correction amount calculation circuit 209.

Step S301: This is the beginning of this flow, and is repeatedly started at a predetermined timing. Step S302: The amplified angular velocity signal is converted from an analog to a digital value. Step S303: The HPF 303 is calculated based on the initial setting value of the cutoff frequency or the previous value. Step S304: Integration is performed according to the initial set value of the integration time constant or the previous value. Step S305: Output an integration result, that is, an angular displacement signal. Step S306: It is determined whether the angular velocity signal is equal to or greater than a predetermined threshold value. Step S307: It is determined whether the integral value is greater than or equal to a predetermined threshold value. Here, if the angular velocity signal is equal to or greater than a predetermined threshold value, or if the integrated value is equal to or greater than the predetermined threshold value even if the angular velocity signal is less than the predetermined threshold value, the panning / tilting state is determined and the process proceeds to step S308. move on. On the other hand, when both the angular velocity signal and the integral value are less than the predetermined threshold value, it is determined that the normal control state or the panning / tilting end state is reached, and the process proceeds to step S310.

Step S308: The cutoff frequency value (f) used for the calculation of the HPF 303 is made higher than the current value by a predetermined value, and the attenuation rate of the low frequency signal is made larger than the current value. Step S309: The value of the time constant used for the integral calculation is shortened by a predetermined value from the current value so that the angular displacement output approaches the reference value. Step S310: The cutoff frequency value (f) used for the calculation of the HPF 303 is made lower by a predetermined value than the current value, and the attenuation factor of the low frequency signal is made smaller than the current value. Step S311: The value of the time constant used for the integration calculation is made longer than the current value by a predetermined value to increase the integration effect. Step S312: End of processing. By the above-described control, the correction target value is set to a steady state by preventing saturation of the integral value = correction target value, and a stable fluctuation correction signal can be obtained.

Subsequently, the operation of each unit of the above-described camera system will be described in order. For example, a light beam (photographing light) incident through the lens 200 at the time of video shooting forms an image on the image sensor 201, is converted into an electric charge (electric signal), and is accumulated. The charge accumulated in the image sensor 201 is read at a predetermined timing (hereinafter referred to as a second cycle) generated by the TG 211 a plurality of times within one field (predetermined cycle = hereinafter referred to as a first cycle). For example, it is converted into a digital signal by an A / D converter (not shown) and input to the camera signal preprocessing circuit 202. Then, the camera signal pre-processing circuit 202 performs predetermined signal processing such as forming a luminance signal and a color signal on the input digital signal, thereby forming an image signal. The image signal output from the camera signal preprocessing circuit 202 is sequentially stored in the image memory 203 at the predetermined timing (second period) generated by the TG 211. As a result, an image signal that is read from the image sensor 201 within one field period (first period) and further processed by the camera signal preprocessing circuit 202 is the second period generated by the TG 211. A plurality of images are stored in the image memory 203 based on the timing.

On the other hand, the shake of the camera system is detected as the angular velocity by the angular velocity sensor 208 in accordance with the timing of reading the electric charge from the image sensor 201, and the shake correction amount calculation circuit 209 calculates the shake correction amount as described above. The shake correction signal output from the shake correction amount calculation circuit 209 corresponds to the image signal read from the image sensor 201 at, for example, the synchronization timing based on the predetermined timing (second period) generated by the TG 211. In addition, it is sequentially stored in the shake correction amount memory 210 . By repeating this process, when the image signal is stored in the image memory 203 and the shake correction amount is stored in the shake correction amount memory 210 a predetermined number of times during one field period, for example, at the end of the field, the image memory 203 and The respective data are read from the shake correction amount memory 210 and input to the coordinate conversion circuit 204. The coordinate conversion circuit 204 converts the coordinates of the image signal read from the image memory 203 based on the shake correction amount read from the shake correction amount memory 210, that is, changes the position of the cutout area. The image signal whose coordinates have been converted is input to the image composition circuit 205, and the image composition circuit 205 adds and averages all the image signals obtained within a predetermined period, for example, one field period, thereby synthesizing the image. Two captured image data are generated and output in the first cycle via the video output terminal 207.

Next, an example of the timing at which the above-described operations are performed will be described with reference to FIG. First, each timing chart shown in the figure will be described. In the figure, reference numeral 411 denotes a synchronization reference signal, which corresponds to, for example, an NTSC vertical synchronization signal. Reference numeral 415 indicates a representative timing among the drive signals of the TG 211. Reference numeral 410 denotes an accumulation period of the image sensor 201. A high portion indicated by 429 is an accumulation period, and accumulation information is read out during a low period. Reference numeral 430 indicates a storage timing at which an image signal read out from the image sensor 201 and subjected to signal processing by the camera signal pre-processing circuit 202 is stored in the image memory 203, and corresponds to 431, 432, 433, and the like in this example. The storage operation is executed at the timing. Reference numeral 440 denotes a timing at which the shake correction amount is calculated from the angular velocity signal and the value is stored in the shake correction amount memory 210. In this example, the storage operation is executed at timings corresponding to 441, 442, 443, and the like. 450 reads out the stored image signal, reads out the shake correction amount stored at the same time, for example, the synchronization timing, performs coordinate conversion of the image signal based on this shake correction amount, and then acquires all of them acquired within one field period. The timing at which one image data is generated by combining these images is shown. In this example, reading is performed at a timing corresponding to 451, 452, 453, etc., and coordinate conversion and synthesis processing are executed.

Next, the operation will be described along the time axis. First, when the vertical synchronization period 412 indicated by the synchronization signal 411 is completed, accumulation and reading of the image sensor 201 are performed based on the timing of the TG drive signal 415 generated by the TG 211, and the camera signal pre-processing circuit 202 further performs image processing. A signal is stored in the image memory 203. For example, when the image sensor 201 is accumulated at the timing indicated by 421 as in this example, the readout operation is performed upon completion of the accumulation based on the TG drive signal 415, and signal processing is performed by the camera signal preprocessing circuit 202. After that, the image signal processed at the timing indicated by 431 is stored. Further, the angular velocity signal obtained from the angular velocity sensor 208 is converted into a shake correction amount by the shake correction amount calculation circuit 209 at the timing indicated by 441 at the same time, and then stored in the shake correction memory 210. After the predetermined number of times (four times in this example) is processed, the stored image signal is read from the image memory 203 at the timing 451 within the synchronization period of the synchronization signal 411 and stored at the same time. The shake correction amount is read from the shake correction amount memory 210, coordinate conversion of an image signal is performed based on the shake correction amount (output image read out according to a shake correction amount described later), and a predetermined field in one field is obtained after the coordinate conversion is completed. The timing of generating a single image data by combining a number of images, for example, all image signals (four image signals in this example) is shown.

As described above, each image signal captured in the image memory 203 obtained by a plurality of shootings during the field period is obtained by dividing each field period and acquiring each image signal, that is, based on the number of captured images. It is possible to reduce the influence of camera shake during the accumulation of the image signal as the accumulation time is shortened. Specifically, when the accumulation time is divided into ¼ by acquiring four images within one field period as in this example, the blur amount of the image is also reduced to ¼. ing. However, if these images are combined as they are, the combined image becomes an image in which the composition between the images is shifted by the amount of movement due to camera shake or the like. Therefore, it is necessary to correct the shift between the images at the time of synthesis. In this example, as described in detail below, the shift between the images is corrected by performing coordinate conversion of the images. In the present embodiment, it is described that four accumulations (second period) and read operations are performed in one field (during one synchronization period: first period), but the number of times is at least 2. The number of times is not limited to four times, and may be any number. Furthermore, it is not always necessary to combine all the image signals in each field period, and a part of the acquired image data may be combined.

Subsequently, coordinate conversion of an image signal will be described with reference to FIG. Reference numeral 500 denotes the entire area of the image stored in the image memory 203. It has a pixel array corresponding to the entire imaging region (entire imaging surface) of the imaging element 201 formed by arranging one photoelectric conversion element (imaging element) constituting a pixel unit in a lattice pattern in the plane, and TG 211 The read images are sequentially stored on the basis of the electric drive pulses generated. Also, areas indicated by 502 and 503 in the figure are cutout frames when output as video signals. That is, the entire imaging area 500 is not output as video as it is, but the area cut out from the area of the entire imaging area 500 by the cutting frame 502 (503) is output as a captured image.

Here, for example, a case where a video signal is cut out by a cutout frame indicated by 502 in FIG. 5 will be described as an example. When an image stored in the image memory 203 is read based on a signal from the TG 211, first, “ Memory images are sequentially read in the direction indicated by the arrow 505 from the pixel indicated by “S”. This readout is started within the synchronization period of the output video signal, and readout is performed at a speed higher than the normal readout speed, for example, up to one pixel before the pixel indicated by “A” before the end of the synchronization period. Then, in the actual video period after the end of the synchronization period, the charge from the pixel indicated by “A” to the pixel “F” in the cutout frame 502 is used as image information for one line of the video signal at a normal readout speed, for example. Reading is started. Further, during the horizontal synchronization period up to the next one line, pixels from “F” to “G” are read at a speed faster than the normal reading speed, and in the same manner as the reading from “A” to “F”. Reading one line from “G” is performed at a normal reading speed. By controlling the readout timing as described above, it is possible to selectively extract the area corresponding to the cutout frame 802 from the entire imaging area 500 stored in the image memory 203 and output the video as a captured image.

When the camera system shakes, for example, when the subject moves within the entire imaging area 500 by the amount indicated by the arrow 504 (= camera system shake), the cut frame is changed to the position indicated by the cut frame 503. In this way, a captured image without moving the subject can be obtained. More specifically, the pixel readout start position is moved from “A” to “B” in order to change the position of the cutout frame, and the cutout frame from the entire imaging region 500 is moved as in the case of the cutout frame 502. An image in 503 is selectively extracted and output as video. In this way, all the pixels read out from the image sensor 201 are stored in the image memory 203, and a part of the peripheral imaging area is read in advance in an amount corresponding to the shake correction information during the synchronization signal period that does not appear in the actual video period, By selectively reading a part of the image sensor 201 based on the shake correction amount of the camera system, a video signal from which the shake of the image accompanying the shake of the camera system is removed can be obtained.

In order to make it easier to understand the image shake correction and image synthesis described so far, a more detailed description will be given using the image schematically shown in FIG. In the figure, reference numerals 602a, 602b, 602c, and 602d schematically show a plurality of image signals (that is, the entire imaging surface 500) captured at equal intervals within one field period that is the first period. For example, images are taken in the order of 602 a → 602 b → 602 c → 602 d at the storage times 421, 422, 423, and 424 shown in FIG. 4 and stored in the image memory 203. In the figure, 611a is a main subject such as a person, 612a (612b, 612c, 612d) is a moving subject such as a car, 613a (613b) is a building, 621a (621b) is a window, and an arrow 630 is The movement direction of the image which has arisen by the rotation blur of an imaging device is shown. That is, the video signals 602a to 602d are acquired when the hand shake in the direction of the arrow 603 occurs.

Here, considering the shake correction signal described above, the obtained shake correction signal is obtained in the shake direction of the camera system, that is, the direction indicated by the arrow 630. Accordingly, by moving the coordinates of the image signal 602a (602b, 602c, 602d) picked up based on the shake correction amount corresponding to each of the image signals 602a (602b, 602c, 602d), each image signal is changed. The amount of movement (blurring amount) caused by the shaking of the camera system is corrected, whereby the shaking amount of each image signal can be corrected. That is, by converting the coordinates of the regions indicated by the broken lines of the image signals 603a, 603b, 603c, and 603d into the same coordinates, the movement due to the shaking of the camera system is canceled.

Further, the corrected image signals 603a, 603b, 603c, and 603d are superimposed and combined to form a frame image (captured image) indicated by the combined image signal 604. As a superposition specific method of image, a method of dividing the same monodentate gauge on the image luminance and chrominance data in addition respectively adding number, accumulated number of times in advance the light amount at the time of imaging of one frame It is sufficient to perform simple addition while suppressing to 1.

The synthesized image signal 604 synthesized in this way is stationary including the main subject 611 and the background, but the moving subject 612 is a blurred image (subject as a result of the overlay processing by the image synthesis circuit 205. The image has a fluctuation according to the movement of itself. Therefore, it is possible to reproduce a smooth depiction of a moving subject while reducing blurring of a captured image due to shaking of the imaging device such as camera shake. That is, according to the above-described embodiment, in the camera system, a plurality of image signals are acquired in a second period shorter than the first period in which the captured image is output, and shake correction is performed for each image signal. Later, by adopting a configuration in which one captured image is created by superimposing and combining these corrected image signals, shake correction of the captured image is performed without impairing the reproducibility of the smooth motion of the moving subject. be able to. Furthermore, by combining after performing shake correction, it is extremely unlikely that an object with no motion, for example, a stationary object such as a building is blurred.

<Second Embodiment>
A second embodiment of the present invention will be described.

In this embodiment, the angular velocity sensor 208 and the shake correction amount calculation circuit 209 for detecting the shake correction amount of the image are used in the first embodiment, but a singular point of each captured image is extracted. Thus, the feature is that the amount of motion between images is detected.

FIG. 7 is a block diagram illustrating a schematic configuration of the imaging apparatus according to the present embodiment. In the figure, reference numeral 700 denotes a singular point displacement amount calculation circuit as a singular point displacement amount calculation unit. As will be described in detail later, for example, the amount of movement between two images is calculated by the coordinate change of the singular point included in the image. To do. In addition, about the place which employ | adopts the same structure as the above-mentioned 1st Embodiment, detailed description is abbreviate | omitted by attaching | subjecting the same code | symbol.

Next, the operation of each part will be described in order. A light beam (photographing light) incident through the lens 200 forms an image on the image sensor 201, is converted into an electric charge (electric signal), and is accumulated. The electric charge accumulated in the image sensor 201 is read out at a predetermined timing (second period) that is generated a plurality of times within one field period (first period) generated by the TG 211, for example, A / It is converted into a digital signal by the D converter and input to the camera signal preprocessing circuit 202. Then, the camera signal pre-processing circuit 202 performs predetermined signal processing such as forming a luminance signal or a color signal on the input digital image signal to form an image signal, which is generated from the TG 211. The images are sequentially stored in the image memory 203 at a predetermined timing (second period). As a result, the image signal read out from the image sensor 201 within one field period and further processed by the camera signal preprocessing circuit 202 is the predetermined timing (second period) generated by the TG 211. A plurality of images are stored in the image memory 203.

Next, the detection of the image blur amount by the singular point displacement amount calculation circuit 700 will be described with reference to FIG. 6. The image signal read from the image memory 203 is input to the singular point displacement amount calculation circuit 700. The singular point displacement calculation circuit 700 extracts singular points. Specifically, for example, the edge of the window 621a that is a point with high luminance in the building 613a in the image signal 601a is extracted as a singular point by edge detection, and this singular point and the singular point in the next successive image signal 601b are detected. 641b is compared, and the two-dimensional position difference is corrected (coordinate conversion). Here, for convenience of explanation, the explanation is made with one singular point as one point. However, in practice, a plurality of singular points can exist in one image signal, and the average deviation amount of singular points is based on the information. Coordinate conversion may be performed by calculating a coordinate movement amount by calculation or the like. In general, it is desirable to have more singularities. This is because the more the background with no motion is included, the more accurately the motion of the image due to camera shake can be extracted.

In the above description, coordinate conversion of two image frames is described. However, since continuous shooting is actually repeated, coordinate conversion similar to the above is performed for image signals exceeding two images. By accumulating differences obtained by repeating the above, coordinate conversion of all image signals becomes possible. By performing feature point extraction as described above, it is possible to obtain a position correction amount of an image signal due to shaking, that is, a shake correction signal.

After that, at the time when storage of the captured image in the image memory 203 is completed a predetermined number of times within one field period, an image signal is read from the image memory 203 and input to the singular point displacement amount calculation circuit 700. As described above, the singular point displacement amount calculation circuit 700 extracts singular points from continuous image signals, and obtains a shake correction amount from the difference in coordinates. On the other hand, the image signal read from the image memory 203 is also input to the coordinate conversion circuit 204, and the coordinates are converted according to the shake correction amount calculated by the singular point displacement amount calculation circuit 700. These coordinate-converted images are input to the image composition circuit 205, and for example, a predetermined number of images are added and averaged to compose an image. The captured image obtained by the synthesis is output via the video output terminal 207. As described above, according to the present embodiment, it is possible to reproduce a smooth depiction of a moving subject while reducing blurring of a captured image due to shaking of the camera system such as camera shake.

It is a block diagram which shows schematic structure of the camera system in the 1st Embodiment of this invention. It is a figure for demonstrating the shake correction calculating circuit of the said camera system. It is a flowchart which shows operation | movement of the said shake correction calculating circuit. It is a timing chart which shows the operation timing of the said camera system. It is a figure which shows typically the image at the time of the shake correction of the said camera system, and the synthesized image. It is a figure explaining the image clipping at the time of shake correction. It is a block diagram which shows schematic structure of the camera system in the 2nd Embodiment of this invention. It is a figure for demonstrating the outline of an electronic shake correction. It is a figure which shows the structure of the conventional camera system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 200 Lens 201 Image pick-up element 203 Image memory 204 Coordinate conversion circuit 205 Image composition circuit 208 Angular velocity sensor 209 Shake correction amount calculation circuit 210 Shake correction amount memory 211 Timing generator 700 Singular point displacement amount calculation circuit

Claims

An image sensor that outputs a plurality of images in one vertical synchronization period ;
An angular velocity sensor that detects an angular velocity every time each of the plurality of images output from the image sensor in the one vertical synchronization period is acquired;
A shake correction amount calculating means for calculating a shake correction amount based on the angular velocity corresponding to each of the plurality of images output from the image sensor during the one vertical synchronization period;
A storage unit that stores the shake correction amount corresponding to each of the plurality of images output from the image sensor in the one vertical synchronization period;
Combining means for generating one composite image per one vertical synchronization period by performing image composition by converting the coordinates of the plurality of images based on the shake correction amount stored in the storage unit. A video imaging apparatus characterized by the above.
The moving image capturing apparatus according to claim 1 , wherein an image accumulation time of the image sensor is determined based on the number of images captured in the one vertical synchronization period .
An imaging step of outputting a plurality of images in one vertical synchronization period ;
An angular velocity detection step of detecting an angular velocity every time each of the plurality of images output from the image sensor in the one vertical synchronization period is acquired;
A shake correction amount calculating step of calculating a shake correction amount based on the angular velocity corresponding to each of the plurality of images output from the image sensor during the one vertical synchronization period;
A storage step of storing the shake correction amount corresponding to each of the plurality of images output from the image sensor during the one vertical synchronization period;
A synthesis step of generating one composite image per vertical synchronization period by converting the coordinates of the plurality of images based on the shake correction amount stored in the storage step and performing image synthesis; A moving image capturing method comprising: