JP2008085688A - Image pickup device, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform an assured correction operation in correcting a blur resulting from hand shaking coping with an increase in an amount of hand shaking without increasing a circuit scale or the like as much as possible. <P>SOLUTION: An image pickup device is equipped with an image pickup system including an image pickup lens optical system 12 for taking an image and a CCD 13, and a controller 22 which detects, from given research block displacement in time-series images obtained by periodical shooting, the amount and direction of hand shaking that have been generated in the image pickup system and captures an effective image according to the content detected from among images obtained in the image pickup system and records the images in a memory card 29 and besides sets variably a detecting condition depending on the detected amount of hand shaking. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program capable of performing sharp image shooting that eliminates the influence of camera shake by electronic camera shake correction.

  Conventionally, it is effective to detect the blurring amount and direction of the photographic optical axis and extract them from the image sensor by performing image processing such as pattern matching by comparing predetermined block ranges in periodically captured images. An electronic camera shake correction technique that eliminates the influence of blur by shifting the position of the recording area as needed is generally used in an imaging apparatus such as a video movie camera.

As one example of this type of technology, for example, when a moving object crosses the imaging screen of an imaging apparatus that is fixed and does not shake during blur correction, it recognizes that the moving object has crossed and shake correction. A technique for preventing malfunction of the device has been considered. (For example, Patent Document 1)
JP-A-8-251473

  In the electronic camera shake correction including the technique described in Patent Document 1, the degree of camera shake becomes large, and if the search range is exceeded, the amount and direction of camera shake cannot be detected. For this purpose, it is conceivable to set a sufficiently large camera shake search range in advance. However, if it does so, the circuit function for performing image processing must be strengthened, resulting in an increase in circuit scale and power consumption.

  The present invention has been made in view of the above circumstances, and the object of the present invention is to reliably deal with a case where the degree of camera shake increases without causing an increase in circuit scale or the like as much as possible. An imaging device, an imaging method, and a program capable of performing a correction operation are provided.

  The present invention provides an image pickup means for picking up an image, and a detection means for detecting the amount and direction of camera shake generated in the image pickup means from a time-series displacement of the image obtained by periodically driving the image pickup means. And a recording control means for displacing an effective image range to be cut out and recorded from the image obtained by the imaging means according to the content detected by the detection means, and the detection means according to the amount of camera shake detected by the detection means. And a condition setting means for variably setting the detection conditions.

  According to the present invention, when the degree of camera shake increases, the camera shake correction operation can be performed reliably without changing the search range itself by changing the camera shake detection condition.

(First embodiment)
A first embodiment when the present invention is applied to a digital camera will be described below with reference to the drawings.
FIG. 1 shows a conceptual configuration of an electronic circuit of the entire digital camera 10.
In the drawing, in the monitoring state in the photographing mode, the position of a part of the lenses in the photographing lens optical system 12, specifically the zoom lens and the focus lens, is appropriately moved by driving the motor (M) 11. A CCD 13 that is a solid-state imaging device is disposed at an imaging position behind the photographing optical axis of the photographing lens optical system 12 via a mechanical shutter (not shown).

  The CCD 13 is scanned and driven by a timing generator (TG) 14 and a CCD driver 15 and outputs a photoelectric conversion output corresponding to a light image formed at fixed intervals for one screen.

  In the AGC / SH (Auto Gain Control / Sample Hold) circuit 16, this photoelectric conversion output is automatically gain-adjusted according to the ISO sensitivity set for each primary color component of RGB in the state of an analog value signal. After being sampled and held, it is sent to the A / D converter 17.

  The A / D converter 17 converts analog image data into digital data and outputs the digital data to the color process circuit 18.

  The color process circuit 18 performs color process processing including pixel interpolation processing and γ correction processing on digital image data, generates digital luminance luminance / chrominance image data YUV by matrix conversion, and then performs DMA (Direct Memory Access). ) Output to the controller 19.

  The DMA controller 19 once writes the image data YUV output from the color process circuit 18 into a buffer inside the DMA controller 19 using the composite synchronization signal, the memory write enable signal, and the clock signal from the color process circuit 18 as well. DMA transfer is performed to a DRAM 21 used as a buffer memory via an interface (I / F) 20.

  A control unit 22 is provided to perform all operation control including DMA transfer to the DRAM 21. That is, the control unit 22 includes a CPU and a non-volatile memory such as a flash memory in which an operation program including an electronic camera shake correction operation described later is fixedly stored. The controller 23 controls the entire digital camera 10 while temporarily developing and writing necessary data in 23.

  Thus, after the DMA transfer of the image data to the DRAM 21 is completed, the control unit 22 reads the image data from the DRAM 21 via the DRAM interface 20 and writes it to the VRAM 25 via the display controller 24.

  The display controller 24 periodically reads the image data from the VRAM 25 and outputs it to the display unit 26.

  The display unit 26 includes, for example, a backlit color liquid crystal panel disposed on the rear surface of the housing of the digital camera 10 and a drive circuit thereof. The display unit 26 functions as an electronic viewfinder in the shooting mode, and the display controller 24. By performing the display based on the image data from the image, the image captured from the display controller 24 at that time is displayed on the monitor in real time, while the selected image is reproduced and displayed in the reproduction mode.

  In this way, a part of the key input unit 27 is configured at a timing at which a still image is to be captured in a so-called through image display state in which the image of the subject at that time is displayed in real time on the display unit 26 as a monitor image. When a shutter key is operated, a trigger signal is generated.

  In response to the trigger signal, the control unit 22 cancels the DMA transfer of the luminance and color difference signals for one screen currently captured from the CCD 13 to the DRAM 21, and is not shown at a shutter speed according to the appropriate exposure conditions. The mechanical shutter and the CCD 13 are driven to obtain luminance and color difference signals for one screen and transfer them to the DRAM 21. Thereafter, this path is stopped and the state is changed to the storage and storage state.

  In this storage and storage state, the control unit 22 outputs the image data YUV for one frame written in the DRAM 21 to 8 pixels × 8 pixels for each of Y, U, and V components via the DRAM interface 20. A unit called a basic block is read out and written to the image compression / decompression circuit 28. The image compression / decompression circuit 28 uses a predetermined image format, for example, ADCT (Adaptive Discrete Couse Transform) according to JPEG (Joint Photographic Coding Experts Group). : Adaptive discrete cosine transform) and encoding by data compression processing such as Huffman coding which is an entropy coding method.

  The obtained code data is read out from the image compression / decompression circuit 28 as a data file for one image, and is written in a memory card 29 that is detachably attached to the digital camera 10 as a recording medium.

  Then, along with the compression processing of the luminance and color difference signals for one frame and the writing of all the compressed data to the memory card 29, the control unit 22 activates the path from the CCD 13 to the DRAM 21 again.

  In addition, the key input unit 27, the sound processing unit 30, and the flash drive unit 31 described above are connected to the control unit 22.

  In addition to the shutter key described above, the key input unit 27 includes a power key, camera shake correction key, zoom key, shooting mode key, playback mode key, cursor key, set key, menu key, macro key, flash key, and the like. The signals accompanying these key operations are sent directly to the control unit 22.

  The sound processing unit 30 includes a sound source circuit such as a PCM sound source, and when recording sound, the sound processing unit 30 digitizes the sound signal input from the microphone unit 32 disposed on the front surface of the housing of the digital camera 10 and generates a predetermined data file format. For example, the data is compressed according to MP3 (Moving Picture Coding Experts Group-1 audio layer 3) standard, and an audio data file is created and sent to the memory card 29.

  On the other hand, the audio processing unit 30 uncompresses and converts the audio data file read from the memory card 29 during audio reproduction into an analog signal, and a speaker unit 33 provided on the back side of the digital camera 10 as with the display unit 26. Drive to emit loud sounds.

  The flash drive unit 31 charges a flash-use large-capacitance capacitor (not shown) at the time of still image shooting, and flash-drives the flash light-emitting unit 34 including a xenon discharge tube or the like based on the control from the control unit 22. .

  However, when shooting of a moving image instead of a still image is selected in the shooting mode, the above-described still image data is acquired and the image compression / expansion circuit 27 is obtained when the first shutter key is operated. A series of operations of compressing data and storing in the memory card 29 is started as a continuous execution at an appropriate frame rate, for example, 30 [frames / second], and the shutter key is operated for the second time, or When a predetermined time limit, for example, 30 seconds elapses, the series of still image data files are collectively set as a motion JPEG data file (AVI file).

  In the playback mode, the control unit 22 selectively reads out the image data stored in the memory card 29, and the image compression / decompression circuit 28 compresses the image data in a procedure that is completely opposite to the data compression procedure in the shooting mode. The image data is decompressed, and the decompressed image data is held in the DRAM 21 via the DRAM interface 20, and then the content held in the DRAM 21 is stored in the VRAM 25 via the display controller 24, and the image data is periodically transferred from the VRAM 25. The video signal is read out and generated on the display unit 26.

  If the selected image data is not a still image but a moving image, playback of the individual still image data constituting the selected moving image file is executed continuously in time at a predetermined frame rate, and the last still image is When the data reproduction is finished, the data is reproduced and displayed using only the still image data positioned at the head until the next reproduction instruction is given.

  Although not shown in FIG. 1, it is assumed that the digital camera 10 can also transmit / receive to / from an external device conforming to, for example, the USB (Universal Serial Bus) standard.

Next, the operation of the above embodiment will be described.
FIG. 2 shows the contents of processing related to moving image shooting executed in the shooting mode in a state in which the camera shake correction function is set to ON by operating the camera shake correction key. It is executed based on the operating program that is being used.

  In this place, the image data is acquired by the imaging operation of the CCD 13 and temporarily stored in the DRAM 21 as a buffer (step A01), and the previous image data similarly held in the DRAM 21 and within a predetermined search range (predetermined search range). Pattern matching is performed in the block), and the amount and direction of image motion are calculated (step A02).

  Whether or not the calculated amount of movement is large depending on whether or not the threshold set in advance corresponding to the search range of the CCD 13 is exceeded, in other words, matching within the search range is performed. It is determined whether or not the movement is so large that it cannot be taken (step A03).

  Here, if it is determined that the amount of motion is not large and pattern matching within the search range is possible, the frame rate (FR) of the imaging drive of the CCD 13 at that time is the frame at the time of normal moving image shooting. It is determined whether the rate is set to a higher frame rate, for example, 60 [frame / second], instead of 30 [frame / second] (step A04).

  When the frame rate is high, the case where it is determined that the amount of motion is large even at the previous timing is in a continuous state. On the other hand, if it is determined that the frame rate is not high, the amount of motion is not particularly large at the previous timing, so if this is determined in step A04, the calculation is performed in the immediately preceding step A02. Based on the amount and direction of the movement, an area to be recorded is cut out from the imaging data acquired in the immediately preceding step A01 as electronic camera shake correction processing and held in the DRAM 21 (step A08). The extracted image data is compressed by the image compression / decompression circuit 28 and then recorded on the memory card 29 (step A09). The above-described step A01 is again performed assuming that one image has been recorded. Return to processing from.

  If it is determined in step A03 that the amount of motion exceeds the threshold value, it is then determined whether or not the drive frame rate of the CCD 13 is set to a high speed at that time (step A10).

  If a high frame rate is not set here, the amount of movement due to camera shake is not large until the previous time, and it is assumed that the normal frame rate was set. The setting is changed (step A13).

  This is because, as shown in FIG. 3A, pattern matching is performed during the transition from one image frame F1 to the next image frame F2, which has a time difference t (for example, 1/30 [second]) according to the normal frame rate. P1 moves from the center position of the search range A for performing the distance L, and the center position P2 of the search range in the next frame F2 greatly deviates, so that accurate pattern matching cannot be performed. Indicates that

  Therefore, as shown in FIG. 3B, for example, by setting a frame rate that is twice as high as that of the image frame F1, the image data is obtained with the image frame F1 'when a time difference of 2 / t has elapsed from a certain image frame F1. Then, the center position P1 ′ of the search range A ′ in the image frame F1 ′ is within the search range where pattern matching can be performed by moving by L / 2 from the center position P1. Therefore, it is shown that even if a large camera shake occurs, it becomes possible to follow this.

  If it is determined in step A10 that a high-speed frame rate has already been set, the image data held at that time is essentially the same as the image frame F1 ′ in FIG. Therefore, it is possible to confirm whether or not it is the recording timing (step A11).

  Here, if it is determined that it is the recording timing, the process proceeds from the step A08 as it is. If it is determined that it is not the recording timing, then the amount of motion calculated in the immediately preceding step A02 is determined. Reflecting the direction, the next imaging data is acquired as shown in the image frame F1 ′, and the amount and direction of motion due to a new camera shake are calculated by pattern matching in the same manner as in step A02 ( Step A12), the process proceeds again to Step A11, and it is determined whether or not the correct recording timing has come.

  Therefore, as described above, when the high frame rate is set to be twice that of the normal image shooting frame rate, the frame timing is to record once every two frames, and the processing in step A12 is also performed. It is executed only once every two frames.

  If it is determined in step A04 that the amount of motion is not large and the frame rate is set to high speed, the amount of motion has been large due to camera shake until then, but at that time, the amount of motion is not so large. Next, the captured image data held at that time is acquired at the original image capturing and recording timing as in the image frame F1 ′ in FIG. 3B. Since there is a possibility that it is not, it is confirmed whether or not the recording timing is the same as in step A11 (step A05).

  Here, if it is determined that it is the recording timing, after the setting change is newly made to lower the imaging frame rate to the normal level (step A07), the process proceeds to step A08.

  On the other hand, if it is determined in step A05 that the recording timing is not reached, the next imaging data is then reflected as shown in the image frame F1 ′ while reflecting the amount and direction of motion calculated in the immediately preceding step A02. As in step A02, the amount and direction of movement due to a new camera shake are calculated by pattern matching (step A06), and the process proceeds to step A05 again to determine whether the correct recording timing has been reached. Determine whether.

  As described above, when the processing shown in FIG. 2 is repeatedly executed, in a state where it is determined that the amount of movement due to camera shake is large, a frame rate higher than the frame rate necessary for recording is temporarily set to perform camera shake correction processing. Therefore, even if the amount of motion due to camera shake is large, it is possible to reliably perform camera shake correction because image frames are acquired in a shorter time period and the amount and direction of motion are detected. Become.

  In the above-described embodiment, the image frame rate has been described as being changed and set in two stages of normal (constant speed) and high speed according to the recording frame rate. In practice, however, in addition to the normal frame rate, Thus, a high frame rate may be set in a plurality of stages.

  For example, as described in the present embodiment, when the taking lens optical system 12 of the digital camera 10 has a zoom function, the influence of camera shake becomes more conspicuous on the telephoto side where the focal length becomes longer. As the focal length becomes longer (= shooting angle of view becomes smaller), the amount of motion may be determined by setting a higher frame rate.

  Further, as shown in the operation of the above embodiment, after setting the image frame rate at a high speed in order to perform camera shake correction, if it is determined that the detected motion amount is not so large, the frame is gradually increased. By changing the setting so as to reduce the rate, it is possible to avoid unnecessary power consumption due to driving of the CCD 13 and image processing.

(Second Embodiment)
A second embodiment when the present invention is applied to a digital camera will be described below with reference to the drawings.
Note that the conceptual configuration of the electronic circuit of the entire digital camera according to the present embodiment is basically the same as that shown in FIG. 1, and the same parts are denoted by the same reference numerals. And the description will be omitted.

Next, the operation of the above embodiment will be described.
FIG. 4 shows the processing contents related to moving image shooting executed in the shooting mode in a state where the camera shake correction function is set to ON by operating the camera shake correction key. All the operation control is stored in advance by the control unit 22. It is executed based on the operating program that is being used.

  In this place, the image data is acquired by the imaging operation of the CCD 13 and temporarily stored in the DRAM 21 as a buffer (step B01), and the previous image data similarly held in the DRAM 21 and within a predetermined search range (predetermined search range). Pattern matching is performed in the block), and the amount and direction of image motion are calculated (step B02).

  Whether or not the calculated amount of movement is large depending on whether or not the threshold set in advance corresponding to the search range of the CCD 13 is exceeded, in other words, matching within the search range is performed. It is determined whether or not the movement is so large that it cannot be taken (step B03).

  Here, if it is determined that the amount of motion is not large and pattern matching within the search range is possible, then the resolution for handling image data when pattern matching is performed in the immediately preceding step B02 at that time. Is determined to be low (step A04).

  When the resolution is low, the case where it is determined that the amount of motion is large even at the previous timing is in a continuous state. On the other hand, if it is determined that the resolution is not low, the amount of movement is not particularly large even at the previous timing. Therefore, when this is determined in step B04, the movement calculated in the immediately preceding step B02 is directly performed. The area to be recorded is cut out from the imaging data acquired in the previous step B01 as the electronic camera shake correction process according to the amount and direction of the image (step B06). The extracted image data is compressed by the image compression / expansion circuit 28 and then recorded on the memory card 29 (step B07). Assuming that the recording of one image is completed as described above, step B01 is again performed. Return to processing from.

  If it is determined in step B03 that the amount of motion exceeds the threshold value, it is determined whether or not the resolution of the image data handled when detecting the amount of motion is set low at that time. Judgment is made (step B08).

  If a low resolution is not set here, the amount of motion due to camera shake is not large until the previous time, and the resolution of the image data handled by motion detection is low again, assuming that the normal resolution was set. The setting is changed so as to become (step B09).

  As shown in FIG. 5A, this is because P1 from the center position of the search range A in which pattern matching is performed during the transition from one image frame F1 to the next image frame F2 having a time difference t according to the normal resolution. Is moved by the distance L, and the center position P2 of the search range in the next frame F2 is greatly deviated, which indicates that accurate pattern matching cannot be performed.

  For this reason, as shown in FIG. 5B, for example, by setting the resolution of the image data handled by motion detection to be low so as to be ½, the amount of computation does not change, but the search range is relatively long. As a result, the apparent search range is expanded so that even if a large camera shake occurs, this can be followed.

  If it is determined in step B08 that the resolution at the time of motion detection has already been set low, the process directly proceeds to step B06.

  If it is determined in step B04 that the amount of motion is not large and the resolution is set to be low, the amount of motion has been large due to camera shake until then, but at that time, the amount of motion is not so large and low resolution is set. Assuming that there is no need to set, after changing the setting to increase the resolution handled when performing motion detection to a normal level (step B05), the process proceeds to step B06.

  As described above, when the processing shown in FIG. 4 is repeatedly executed, in a state where it is determined that the amount of movement due to camera shake is large, the camera shake is temporarily set so as to reduce the resolution of the image data handled for motion detection. Since the correction processing is executed, even if the amount of movement due to camera shake is large, the image frame is acquired with the apparent search range widened by the thinned image data, and the amount and direction of movement are determined. Therefore, the camera shake correction can be surely executed.

  In the embodiment described above, the resolution of the image data handled when performing motion detection has been described as being changed and set in two stages of a normal value and a low value. In practice, in addition to the normal resolution, A lower resolution may be set in a plurality of stages.

  For example, as described in the present embodiment, when the taking lens optical system 12 of the digital camera 10 has a zoom function, the influence of camera shake becomes more conspicuous on the telephoto side where the focal length becomes longer. As the focal length becomes longer (= the shooting angle of view becomes smaller), a lower resolution may be set and image processing such as pattern matching for motion detection may be executed.

  Further, as shown in the operation of the above embodiment, after setting the resolution of the image to be handled for performing the camera shake correction to be low, if it is determined that the detected motion amount is not so large, the image to be handled in stages. The setting may be changed to increase the resolution.

  In the first and second embodiments, the electronic camera shake correction operation is described as being performed at the time of moving image shooting. However, the present invention is not limited to this, and a through image is displayed at the time of still image shooting. Even if the same operation is executed together, a certain degree of camera shake correction effect can be expected.

  In addition, although each of the above embodiments has been described for a case where the present invention is applied to a digital camera, the present invention is not limited to this, and a mobile phone having a camera function or a PDA (Personal Digital Assistants: personal information portable terminal) ), Etc., can be similarly applied to various electronic devices.

  In addition, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention in the implementation stage. In addition, the functions executed in the above-described embodiments may be combined as appropriate as possible. The above-described embodiment includes various stages, and various inventions can be extracted by an appropriate combination according to a plurality of disclosed structural requirements. For example, even if several constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the effect can be obtained, a configuration from which the constituent requirements are deleted can be extracted as an invention.

1 is a block diagram showing a schematic configuration of a functional circuit of a digital circuit according to a first embodiment of the present invention. The flowchart which shows the content of the camera-shake correction process at the time of the video recording concerning the embodiment. The figure explaining change setting of detection conditions (frame rate) of motion amount detection concerning the embodiment. The flowchart which shows the content of the camera-shake correction process at the time of the video recording which concerns on the 2nd Embodiment of this invention. The figure explaining the change setting of the detection conditions (image resolution) of the motion amount detection according to the embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Digital camera, 11 ... Motor (M), 12 ... Shooting lens optical system, 13 ... CCD, 14 ... Timing generator (TG), 15 ... CCD driver, 16 ... AGC / SH circuit, 17 ... A / D conversion 18 ... color process circuit, 19 ... DMA controller, 20 ... DRAM interface (I / F), 21 ... DRAM, 22 ... control unit, 23 ... main memory, 24 ... display controller, 25 ... VRAM, 26 ... display 27: Key input unit, 28 ... Image compression / decompression circuit, 29 ... Memory card, 30 ... Audio processing unit, 31 ... Flash drive unit, 32 ... Microphone unit, 33 ... Speaker unit, 34 ... Flash light emitting unit.

Claims (10)

Imaging means for taking an image;
Detecting means for detecting the amount and direction of camera shake occurring in the imaging means from the displacement of the images on the time series obtained by periodically driving the imaging means;
A recording control means for displacing an effective image range to be cut out and recorded from the image obtained by the imaging means according to the content detected by the detection means;
An image pickup apparatus comprising: a condition setting unit that variably sets a detection condition in the detection unit according to the amount of camera shake detected by the detection unit.   2. The imaging apparatus according to claim 1, wherein the detection unit detects the amount and direction of camera shake generated in the imaging unit from a displacement of a predetermined search block in an image obtained by the imaging unit.   The condition setting means variably sets the detection condition so that when the amount of camera shake detected by the detection means increases, the amount of camera shake that can be detected in the predetermined search block by the detection means increases. The imaging apparatus according to claim 2.   When the amount of camera shake detected by the detection unit increases, the condition setting unit reduces the relative amount of camera shake displacement in the search block without changing the size of the predetermined search block. The imaging apparatus according to claim 2, wherein the detection condition is variably set.   2. The imaging apparatus according to claim 1, wherein the condition setting means variably sets a frame rate of an image obtained by the imaging means. 2. The imaging apparatus according to claim 1, wherein the condition setting unit variably sets a resolution of a predetermined search block of an image obtained by the imaging unit. The imaging means has a zoom function for changing the shooting angle of view,
Detection means for detecting the amount and direction of camera shake occurring in the imaging device from the displacement of a predetermined search block in the time-series image obtained by the imaging means;
2. The condition setting unit variably sets a detection condition in the detection unit according to a shooting angle of view by a zoom function of the imaging unit in addition to the amount of camera shake detected by the detection unit. The imaging device described.   The imaging apparatus according to claim 1, wherein the condition setting unit variably sets detection conditions in the detection unit over a plurality of stages. An imaging method with an imaging device that captures an image,
A detection step of detecting the amount and direction of camera shake occurring in the imaging device from the displacement of the captured image on a time series obtained periodically;
A recording control step for displacing an effective image range to be cut out and recorded from the captured image according to the content detected in the detection step;
An imaging method comprising: a condition setting step for variably setting a detection condition in the detection step according to the amount of camera shake detected in the detection step. A program executed by a computer incorporated in an imaging device that captures an image,
A detection step of detecting the amount and direction of camera shake occurring in the imaging device from the displacement of the captured image on a time series obtained periodically;
A recording control step for displacing an effective image range to be cut out and recorded from the captured image according to the content detected in the detection step;
A program that causes a computer to execute a condition setting step that variably sets a detection condition in the detection step according to the amount of camera shake detected in the detection step.

Images