JP2006319541A - Imaging device, moving picture processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain moving picture data wherein the view angle is given priority and moving picture data wherein hand-shape correction is given priority selectively and optionally. <P>SOLUTION: An imaging device equipped with a hand-shake correcting function files and records moving picture data obtained without pixel absence during photography of a moving picture, (i.e. moving picture data in which hand shake is not corrected) in a memory. Here, hand-shake information is extracted from the moving picture data recorded during the photography, filed, and recorded in the memory while related to the moving picture file. When a user designates hand-shake correction, hand-shake correction processing is carried out by using the hand-shake correction corresponding to the moving picture data and moving picture data having been corrected are re-recorded in the memory. Consequently, the moving picture data wherein the view angle is given priority and the moving picture data wherein the hand-shake correction is given priority can be obtained selectively and optionally. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image pickup apparatus such as a digital camera capable of shooting a moving image, and more particularly to an image pickup apparatus having an electronic camera shake correction function for electronically correcting camera shake during moving image shooting, and the image pickup apparatus. The present invention relates to a moving image processing method and a program.

  In general, an imaging apparatus capable of shooting a moving image has a function for correcting a shake that occurs when a user shoots with the camera body held by hand. This is called “camera shake correction function”, and there are optical type and electronic type. The optical camera shake correction function corrects camera shake by sensing the amount of camera shake using a sensor and moving the lens unit itself.

  On the other hand, the electronic camera shake correction function corrects the camera shake of the moving image data by performing image processing on the moving image data obtained from the image sensor. As a general technique, a portion other than the surrounding pixel region corresponding to the camera shake portion is cut out from each frame image constituting the moving image data, and the cut image is obtained as an image after camera shake correction. However, at this time, some pixels of the image are lost, so there is a drawback that the angle of view becomes narrow.

Patent Document 1 discloses that a predetermined field angle is ensured by interpolating a pixel missing portion due to such camera shake correction from another frame image. Japanese Patent Application Laid-Open No. H10-228867 discloses that moving image data is recorded without correcting camera shake during moving image shooting, and that camera shake correction is performed on the moving image data during reproduction.
JP 11-187303 A

  In Patent Document 1, the moving image data recorded without moving image stabilization at the time of moving image shooting is configured to perform camera shake correction at the time of reproduction. However, it can be finally obtained only with moving image data subjected to camera shake correction. However, depending on the user, there are various demands such as requiring moving image data before correction giving priority to the angle of view over moving image data subjected to camera shake correction, or requiring both moving image data.

  Therefore, an object of the present invention is to provide an imaging apparatus, a moving image processing method, and a program capable of optionally obtaining moving image data with priority on the angle of view and moving image data with priority on camera shake correction.

  An image pickup apparatus according to claim 1 of the present invention includes a moving image shooting unit that acquires image data of each frame continuously output from the image sensor as moving image data in a state in which there is no missing pixel due to camera shake correction, and the moving image shooting Camera shake information generation means for generating camera shake information for the moving image data obtained by the means, correction designation means for designating whether or not to perform camera shake correction, and generation of the camera shake information according to the designation of camera shake correction by the correction designation means Moving image processing means for performing the camera shake correction processing of the moving image data acquired by the moving image photographing means based on the camera shake information generated by the means, and reproducing and displaying or re-recording the moving image data after the camera shake correction. Features.

  According to such a configuration, information indicating camera shake of the moving image data is generated for the moving image data recorded in a state in which no pixel is missing at the time of shooting. Then, in accordance with a camera shake correction instruction from the user, the moving image data is subjected to camera shake correction based on the camera shake information, and is reproduced and displayed or recorded and saved.

  According to a second aspect of the present invention, in the image pickup apparatus according to the first aspect, the moving image data acquired by the moving image photographing unit is a pixel associated with camera shake correction at least until an instruction to change the recording state is given. Comprising a recording means for recording and storing in a state without omission, the correction specifying means allows the user to specify whether or not to correct the moving image data at any timing after the moving image data acquisition by the moving image capturing means, The moving image processing means performs a camera shake correction process on the moving image data recorded and stored in the recording means in a state in which no pixels are lost due to the camera shake correction when the camera shake correction is specified by the correction specifying means. It is characterized by performing.

  According to such a configuration, when a user instructs camera shake correction at an arbitrary timing, it is possible to perform camera shake correction on moving image data recorded and stored in a state where there is no pixel omission associated with camera shake correction at the time of shooting.

  According to a third aspect of the present invention, in the imaging apparatus according to the first aspect, a recording unit that records and stores the camera shake information generated by the camera shake information generating unit in association with the moving image data acquired by the moving image shooting unit. The correction designation means designates whether or not to correct camera shake for any moving picture data recorded and stored in the recording means, and the moving picture processing means designates that the camera shake correction is performed by the correction designation means. The moving image data is subjected to camera shake correction processing based on camera shake information recorded and stored in the recording unit corresponding to the moving image data.

  According to such a configuration, camera shake information is recorded and stored in association with moving image data acquired at the time of shooting. As a result, when camera shake correction is specified for any video data after shooting, camera shake correction processing can be performed using camera shake information corresponding to the video data.

  According to a fourth aspect of the present invention, in the imaging apparatus according to any one of the first to third aspects, the correction specifying unit specifies a degree of camera shake correction, and the moving image processing unit includes: The camera shake correction process is controlled such that the camera shake state after the correction of the moving image data changes according to the degree of camera shake correction specified by the correction specifying means.

  According to such a configuration, it is possible for the user to arbitrarily specify the degree of camera shake correction after shooting, and to perform camera shake correction processing of the moving image data.

  According to a fifth aspect of the present invention, in the imaging apparatus according to the second or third aspect, the image processing apparatus further comprises a storage method designating unit that designates a video data storage method at the time of re-recording. The corrected moving image data is re-recorded in the recording means in accordance with a storage method designated by the designation means.

  According to such a configuration, the user can arbitrarily instruct and re-record the method for storing the moving image data that has been subjected to camera shake correction.

  According to a sixth aspect of the present invention, in the imaging apparatus according to the fifth aspect, the storage method includes overwrite storage and new storage, and the moving image processing unit is configured to store the overwrite by the storage method designating unit. Is specified, the corrected moving image data is re-recorded on the recording means in place of the original moving image data, and when the new storage is specified, the corrected moving image data is stored in the recording means. It is characterized by re-recording in a different area from the original moving image data.

  According to such a configuration, overwriting or new saving can be designated as a method for saving the corrected moving image data. In this case, in the overwrite storage, it is possible to erase the moving image data before correction obtained at the time of shooting and re-record only the corrected moving image data. In the new saving, the corrected moving image data can be re-recorded while leaving the uncorrected moving image data obtained at the time of shooting.

  According to a seventh aspect of the present invention, in the imaging apparatus according to the first aspect, the specification for specifying a reference subject serving as a reference for determining camera shake on moving image data in which a plurality of subjects having different movements are photographed. And the camera shake information generating means generates camera shake information focusing on the movement of the reference subject specified by the specifying means.

  According to such a configuration, when a plurality of subjects having different motions are shot as a moving image, camera shake correction can be performed by paying attention to the motion of the reference subject among them.

  According to an eighth aspect of the present invention, there is provided the imaging apparatus according to the seventh aspect, further comprising a pixel ratio specifying unit that specifies a pixel ratio for subject determination, wherein the specifying unit is specified by the pixel ratio specifying unit. The reference subject is specified based on the pixel ratio.

  According to such a configuration, when the user arbitrarily designates the pixel ratio for subject determination, it is possible to perform camera shake correction by changing the reference subject according to the designated pixel ratio.

  According to a ninth aspect of the present invention, there is provided a moving image processing method in which image data of each frame continuously output from an image sensor is acquired as moving image data in a state where there is no pixel loss associated with camera shake correction; A step of generating camera shake information indicating a movement of camera shake, a step of specifying whether or not to perform camera shake correction, and when camera shake correction is specified, performs a camera shake correction process of the moving image data based on the camera shake information; Reproducing and displaying or re-recording the moving image data after the camera shake correction.

  According to such a moving image processing method, the same effect as that of the first aspect of the invention can be achieved by executing the processing according to each step.

  A program according to claim 10 of the present invention is a program executed by a computer installed in an imaging apparatus having a moving image shooting function, and each frame image continuously output from the imaging device to the computer. A function for acquiring data as moving image data in a state where there is no pixel loss associated with camera shake correction, a function for generating camera shake information indicating the movement of camera shake in the video data, a function for specifying whether or not to perform camera shake correction, and a camera shake When the correction is designated, the moving image data is subjected to camera shake correction processing based on the camera shake information, and a function of reproducing and displaying or re-recording the moving image data after the camera shake correction is realized.

  Therefore, when the computer executes the program for realizing each function, the same effects as those of the first aspect of the invention can be achieved.

  According to an eleventh aspect of the present invention, there is provided a moving image processing method in which image data of each frame continuously output from an image sensor is acquired as moving image data in a state where there is no pixel loss associated with camera shake correction; The step of generating camera shake information indicating the movement of the camera shake, the step of recording the generated camera shake information in association with the moving image data in a predetermined memory, the step of specifying whether or not to perform the camera shake correction, and the camera shake correction are specified. And performing a camera shake correction process on the moving image data based on the camera shake information corresponding to the moving image data recorded in the memory, and reproducing and displaying or rerecording the moving image data after the camera shake correction. It is characterized by.

  According to such a moving image processing method, the same effects as the invention of the third aspect can be achieved by executing the processing according to each step.

  A program according to a twelfth aspect of the present invention is a program executed by a computer mounted on an imaging device having a moving image shooting function, and each frame image continuously output from the imaging device to the computer. A function for acquiring data as moving image data in a state where there is no pixel loss associated with camera shake correction, a function for generating camera shake information indicating the movement of camera shake in the moving image data, and the generated camera shake information in association with the video data When the camera shake correction is specified, the function for recording in the memory of the video data, the function for specifying whether or not to perform the camera shake correction, and the camera shake of the video data based on the camera shake information corresponding to the video data recorded in the memory are specified. A function of performing correction processing and reproducing and displaying or re-recording moving image data after the camera shake correction is characterized.

  Therefore, when the computer executes the program for realizing each function, the same effect as that of the third aspect of the invention can be achieved.

  According to the present invention, in an imaging apparatus having a camera shake correction function, it is possible to perform camera shake correction on moving image data according to a user's request after shooting, and movie data giving priority to angle of view and moving image giving priority to camera shake correction. Data can optionally be obtained.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram showing an external configuration when a digital camera capable of shooting a moving image is taken as an example of the imaging apparatus according to the first embodiment of the present invention. FIG. (B) is a perspective view which mainly shows the structure of a back surface.

  The digital camera 1 has a photographing lens 3, a self-timer lamp 4, an optical finder window 5, a strobe light emitting unit 6, a microphone unit 7 and the like on the front surface of a substantially rectangular thin plate-like body 2 on the upper surface (for the user). On the right end side, a power key 8 and a shutter key 9 are provided.

  The power key 8 is a key operated every time the power is turned on / off, and the shutter key 9 is a key for instructing a photographing timing at the time of photographing.

  Also, on the back of the digital camera 1, a shooting mode (R) key 10, a playback mode (P) key 11, an optical viewfinder 12, a speaker unit 13, a macro key 14, a strobe key 15, a menu (MENU) key 16, a ring A key 17, a set (SET) key 18, a display unit 19, and the like are provided.

  The shooting mode key 10 is operated automatically from the power-off state to automatically turn on the power and shift to the still image shooting mode. On the other hand, by repeatedly operating from the power-on state, the still image mode and the moving image mode are switched. Set cyclically. The still image mode is a mode for photographing a still image. The moving image mode is a mode for shooting a moving image.

  The shutter key 9 is commonly used for these photographing modes. That is, in the still image mode, a still image is taken at the timing when the shutter key 9 is pressed. In the moving image mode, shooting of a moving image is started at a timing when the shutter key 9 is pressed, and shooting of the moving image is ended when the shutter key 9 is pressed again.

  When the playback mode key 11 is operated from the power-off state, the playback mode key 11 is automatically turned on to enter the playback mode.

  The macro key 14 is operated when switching between normal shooting and macro shooting in the still image shooting mode. The strobe key 15 is operated when switching the light emission mode of the strobe light emitting unit 6. The menu key 16 is operated when selecting various menu items including presence / absence of camera shake correction. The ring key 17 is integrally formed with item selection keys in the up, down, left, and right directions, and the set key 18 located in the center of the ring key 17 sets the item selected at that time. To operate.

  The display unit 19 is composed of a color liquid crystal panel with a backlight, and displays a through image on the monitor as an electronic viewfinder in the photographing mode, and reproduces and displays the selected image and the like in the reproduction mode.

  Further, the digital camera 1 is provided with an optical zoom function, and the enlargement ratio of the image can be changed by physically changing the focal length by operating the zoom keys 20a and 20b. Of the zoom keys 20a and 20b, one zoom key 20a is for the telephoto end and is used when the zoom magnification is changed to the telephoto side. The other zoom key 20b is for the wide end and is used when the zoom magnification is changed to the wide angle side.

  Although not shown, the digital camera 1 has a memory card slot for attaching / detaching a memory card used as a recording medium, a serial interface connector for connecting to an external personal computer, etc., for example, USB (Universal). Serial Bus) connector and the like are provided.

  FIG. 2 is a block diagram showing an electronic circuit configuration of the digital camera 1.

  In the digital camera 1, a lens optical system 22 including a focus lens and a zoom lens (not shown) constituting the photographing lens 3 is provided so as to be movable within a predetermined range in the optical axis direction by driving a motor 21. . A CCD (charge coupled device) 23 that is an image sensor is disposed behind the optical axis of the lens optical system 22. The CCD 23 receives light from each part of the subject input through the photographing lens 3 and outputs an electrical signal corresponding to the intensity of the light.

  In the recording mode, which is the basic mode, the CCD 23 is scanned and driven by a timing generator (TG) 24 and a driver 25, and outputs a photoelectric conversion output corresponding to a light image formed at regular intervals for one screen. The photoelectric conversion output of the CCD 23 is appropriately gain-adjusted for each primary color component of RGB in the state of an analog value signal, sampled and held by the sample hold circuit 26, and converted into digital data by the A / D converter 27. The

  Then, the image processing circuit 28 performs image processing including pixel interpolation processing and γ correction processing to generate a digital luminance signal Y and color difference signals U and V (Cb, Cr), and DMA (Direct Memory Access). ) Output to the controller 29.

  The DMA controller 29 once uses the luminance signal Y and the color difference signals U and V output from the image processing circuit 28 by using the composite synchronization signal, the memory write enable signal, and the clock signal from the image processing circuit 28 once. And the DMA transfer to the DRAM 31 used as the buffer memory via the DRAM interface (I / F) 30.

  The control unit 32 controls the entire digital camera 1 and is constituted by a microcomputer including a CPU, a ROM storing an operation program executed by the CPU, a RAM used as a work memory, and the like. . In the present embodiment, the control unit 32 is provided with a setting storage unit 32a for storing various setting information related to camera shake correction of moving image data.

  After the DMA transfer of the luminance and color difference signals to the DRAM 31, the control unit 32 reads the luminance and color difference signals from the DRAM 31 via the DRAM interface 30 and writes them to the VRAM 34 via the VRAM controller 33.

  The digital video encoder 35 periodically reads the luminance and color difference signals from the VRAM 34 via the VRAM controller 33, generates a video signal based on these data, and outputs the video signal to the display unit 19.

  As described above, the display unit 19 functions as a monitor display unit (electronic finder) at the time of shooting. By performing display based on the video signal from the digital video encoder 35, the display unit 19 captures from the VRAM controller 33 at that time. An image based on the image information is displayed in real time.

  As described above, when the image at that time is displayed in real time as the monitor image on the display unit 19, for example, when the shutter key 9 is pressed at a timing at which still image shooting is desired, a trigger signal is generated.

  In response to the trigger signal, the control unit 32 immediately stops the path from the CCD 23 to the DRAM 31 immediately after the DMA transfer of the luminance and color difference signals for one screen captured from the CCD 23 to the DRAM 31 is completed. , Transition to the record storage state.

  In this recording and storage state, the control unit 32 outputs the luminance and color difference signals for one frame written in the DRAM 31 to 8 pixels × 8 pixels for each of Y, Cb, and Cr components via the DRAM interface 30. The data is read out in units called basic blocks and written in a JPEG (Joint Photographic Coding Experts Group) circuit 37. The JPEG circuit 37 uses an ADCT (Adaptive Discrete Cosine Transform) and an entropy coding system. Data compression is performed by processing such as conversion.

  The obtained code data is read out from the JPEG circuit 37 as a data file of one image and written in the recording memory 38. The memory 38 includes a memory card that is detachably mounted as a recording medium in addition to an internal memory such as a flash memory built in the main body in advance. With the compression processing of the luminance and color difference signals for one frame and the completion of writing all the compressed data to the memory 38, the control unit 32 activates the path from the CCD 23 to the DRAM 31 again.

  The control unit 32 is further connected with an audio processing unit 39, a USB interface (I / F) 40, and a strobe driving unit 41.

  The sound processing unit 39 includes a sound source circuit such as a PCM sound source, digitizes a sound signal input from the microphone unit (MIC) 7 during sound recording, and performs a predetermined data file format such as MP3 (MPEG-1 audio layer). 3) Data compression is performed in accordance with the standard to create an audio data file and send it to the memory 38. On the other hand, when reproducing the audio, the audio data file read from the memory 38 is uncompressed and converted into an analog signal. The sound is output through a speaker unit (SP) 13 provided on the back side.

  The USB interface 40 performs communication control when image data and other information are transmitted / received to / from another information terminal device such as a personal computer connected by wire via a USB connector. The strobe drive unit 41 charges a strobe capacitor (not shown) at the time of shooting, and then drives the strobe light emitting unit 6 to flash based on control from the control unit 32.

  In addition to the shutter key 9 described above, the key input unit 36 includes a power key 8, a shooting mode key 10, a playback mode key 11, a macro key 14, a strobe key 15, a menu key 16, a ring key 17, and a set key. 18, zoom keys 20 a and 20 b and the like, and signals accompanying these key operations are sent directly to the control unit 32.

  Further, when shooting a moving image instead of a still image, when the shutter key 9 is pressed, the above-described JPEG circuit 37 compresses the captured moving image using a technique such as motion-JPEG (Joint Photographic Experts Group). To the memory 38. When the shutter key 9 is operated again, the recording of the moving image data is finished.

  On the other hand, in the playback mode which is the basic mode, the control unit 32 selectively reads out the image data recorded in the memory 38 and is compressed by a procedure completely opposite to the procedure of data compression in the recording mode by the JPEG circuit 37. Decompress image data. The decompressed image data is held in the DRAM 31 via the DRAM interface 30, and then the content held in the DRAM 31 is stored in the VRAM 34 via the VRAM controller 33. The image data is periodically read out from the VRAM 34. A video signal is generated and reproduced and output by the display unit 19.

  If the selected image data is not a still image but a moving image, the multiple frames of still image data that make up the moving image data are played back and displayed sequentially in chronological order, and all the still image data is played back. For example, the top still image data is displayed until the next playback instruction is given.

  Next, the operation at the time of moving image shooting by the digital camera 1 will be described.

  FIG. 3 is a diagram showing the relationship between the moving image file of the digital camera 1 and the camera shake information file.

  When the shutter key 9 is pressed while the moving image mode is set by pressing the shooting mode key 10, moving image shooting is started. As a result, the image data of each frame sequentially output from the CCD 23 serving as the image pickup device is captured as moving image data, and is filed and recorded in the memory 38 in a state where there is no missing pixel due to camera shake correction. The state of no pixel loss due to the camera shake correction is a state in which information on peripheral pixel portions that differ from frame to frame is captured without being lost by camera shake correction, for example, at a resolution lower than the maximum resolution of the CCD. There may be pixel missing in the same part in each frame, such as pixel thinning when capturing a moving image or pixel missing in the optical black detection area. When the shutter key 9 is pressed again, the recording of the moving image data ends.

  Here, when a user (photographer) shoots a moving image while supporting the camera body with his / her hand without using a tripod or the like, a so-called “shake” image blurring phenomenon occurs. In particular, when taking a moving body as a subject and moving while moving the camera body, a large amount of camera shake occurs.

  In the present embodiment, information on camera shake (direction and amount of camera shake) that occurs during movie shooting is extracted from movie data after shooting, and the camera shake information is filed and recorded in association with a movie file. . In the example of FIG. 3, the camera shake information file 1 is associated with the moving image file 1, and the camera shake information file 2 is associated with the moving image file 2. These camera shake information files 1 and 2 are used when camera shake correction is performed on the corresponding moving image files 1 and 2 after shooting.

  FIG. 4 is a diagram showing an example of a correction designation screen for designating whether or not camera shake correction is performed during playback display of moving image data, and FIG. 5 is an example of a storage method designation screen for designating a storage method when re-recording moving image data. FIG.

  The moving image data obtained by moving image shooting is recorded in the memory 38 without pixel blurring and without camera shake correction. Here, when it is desired to reproduce and display the moving image data in a state in which camera shake correction is performed after shooting, for example, by operating the menu key 16, a correction designation screen 51 as shown in FIG. 4 is displayed. The correction designation screen 51 is provided with items 52 and 53 for designating the presence or absence of camera shake correction. In this example, when the item 52 is selected with a cursor or the like, “camera shake correction” is set, and when the item 53 is selected with a cursor or the like, “no camera shake correction” is set.

  Further, when it is desired to permanently save the moving image data in a state in which camera shake is corrected, for example, the re-recording mode is set by operating the menu key 16 or the like. As a result, the moving image data is corrected for camera shake and re-recorded in the memory 38. At that time, a storage method can be specified on a storage method specification screen 61 as shown in FIG. The storage method designation screen 61 includes items 62 and 63 for designating “overwrite saving” or “new saving” as a saving method. In this example, when the item 62 is selected with a cursor or the like, “overwrite save” is set, and when the item 63 is selected with a cursor or the like, “new save” is set. The setting information at this time is stored in the setting storage unit 32a in the control unit 32 shown in FIG.

  Hereinafter, the processing of moving image data by the digital camera 1 will be described in detail. Note that the processes shown in the following flowcharts are executed when the control unit 32, which is a microcomputer, reads a predetermined program recorded in a recording medium such as a ROM or a RAM.

(A) Camera shake information addition processing First, camera shake information addition processing for adding camera shake information to moving image data will be described.

  FIG. 6 is a flowchart showing the operation of camera shake information addition processing by the digital camera 1. This camera shake information addition processing is executed automatically after shooting a moving image or according to a user instruction.

  First, a moving image file to which camera shake information is added is specified in the memory 38 (step A11). This is performed, for example, when the user explicitly indicates a moving image file to which camera shake information is added by a predetermined operation. When a moving image file to which camera shake information is added is specified, a frame position (pointer) for sequentially processing each frame image constituting the moving image file (moving image data) is initialized (step A12). The camera shake information file is created in such a procedure.

  That is, a plurality of frame images before the current frame position are compared, and a motion vector (movement direction and movement amount) is detected for each pixel (step A13). Subsequently, pixels having the same motion vector are grouped based on the detected motion vector of each pixel (step A14).

  Here, the group with the smallest moving amount among the groups composed of the predetermined number of pixels or more is specified as the reference subject, and the other groups are specified as the moving subjects (step A15).

  A specific example is shown in FIG. Assume that moving image data is composed of frame images of frame 1, frame 2,. Pa, Pb, and Pc in the figure are frames indicating rough positions of groups of pixels having the same motion vector, and do not strictly indicate pixel areas of groups of pixels. If these movement amounts are as follows, Pa and Pb are specified as the reference subject and Pc as the moving subject.

Pa movement amount: 10 pixels upward, 15 pixels right
Movement amount of Pb: 10 pixels upward, 15 pixels right
Movement amount of Pc: 10 pixels upward and 300 pixels right.

  When the reference subject is specified in this way, the motion vector of the reference subject is temporarily stored in a work memory (not shown) or the like as a camera movement vector caused by camera panning or camera shake at the time of shooting the current frame. (Step A16).

  In the example of FIG. 9, 10 pixels upward and 15 pixels right are stored as camera motion vectors. There is a possibility that the movement vector includes a movement of the camera panned intentionally by the user. In view of this, first, the Pann vector of the camera is obtained from the average value of the camera movement vectors obtained corresponding to a predetermined number of frame images before the current frame position (step A17). Then, the camera shake vector at the current frame position is obtained from the difference between the camera movement vector at the current frame position obtained at step A16 and the camera Pann vector obtained at step A17 (step A18).

  Unlike this example, for example, when the photographer has panned the camera so that the subject indicated by Pc is placed in the center of the frame, the amount of movement of Pc on the image data is the smallest, so Pc is It is specified as the reference subject, and the Pann vector and the hand movement vector are also different from each other focusing on the movement of the reference subject.

  The camera shake vector obtained in this way is associated with the number added to the frame and is additionally stored in the camera shake information file (step A19).

  Thereafter, similarly, the frame position to be processed next is sequentially updated (steps A20 and A21), and the camera shake vector at the frame position is additionally stored in the camera shake information file (steps A13 to A19). Then, when the processing for all the frames is completed (No in Step A20), the camera shake information file is recorded in the memory 38 in association with the moving image file (Step A22), and the camera shake information addition process is completed here. To do.

(B) Reproduction display processing Next, the reproduction display processing of moving image data will be described.

  FIG. 7 is a flowchart showing the operation of the reproduction / display processing of moving image data by the digital camera 1. This reproduction display processing is executed by the user instructing reproduction display by a predetermined operation after moving image shooting. At that time, the user can designate whether or not to correct the camera shake at an arbitrary timing through the correction designation screen 51 as shown in FIG. In the following description, it is assumed that “camera shake correction” is designated on the correction designation screen 51.

  First, a moving image file to be reproduced and displayed in the memory 38 is specified (step B11). This is performed, for example, when the user explicitly indicates a moving image file to be reproduced and displayed by a predetermined operation. Subsequently, a camera shake information file corresponding to the moving image file is specified (step B12). When the moving image file to be reproduced and the camera shake information file corresponding thereto are specified, the frame position (pointer) to be processed is initialized (step B13), and then the moving image data is reproduced and displayed in the following procedure. The

  That is, image data at the current frame position is extracted from the moving image file, and a camera shake vector at the current frame position is extracted from the camera shake information file (steps B14 and B15).

  Here, there are two display methods, “enlarged / reduced display” and “interpolated display” (step B16). In “enlarged / reduced display”, a portion other than the surrounding region corresponding to the camera shake vector is cut out from the image data, and the cut out image data is generated as a camera shake corrected image (step B17).

  A specific example is shown in FIG. In the figure, the surrounding area indicated by the oblique lines is a part that protrudes from the screen due to camera shake, and the other area is cut out as a part without camera shake. In this case, since the angle of view is narrowed by clipping, as shown in the example of FIG. 11, the clipped image data is displayed after being enlarged or reduced in accordance with the resolution of the display device (step B18). .

  On the other hand, in the “interpolation display”, the image data display position is adjusted according to the camera shake vector, thereby generating an image subjected to camera shake correction (step B19). In this case, depending on the display position, a portion that protrudes from a predetermined screen area occurs. Therefore, the protruding portion is deleted, and the missing portion is displayed after interpolation processing using the corresponding portion of another frame image. Suppose (step B20).

  It should be noted that “enlarged / reduced display” or “interpolated display” can be arbitrarily specified by the user when camera shake correction is specified on the correction specification screen 51 shown in FIG. 4, for example.

  Thereafter, in the same manner, the frame position to be processed next is sequentially updated (steps B21 and B22), the image data and the camera shake vector at the updated frame position are read, and the image data is subjected to camera shake correction. Are displayed according to a predetermined frame rate (steps B14 to B20). Then, when the processing for all the frames is completed (No in step B21), the reproduction display processing here ends.

(C) Re-recording process Next, the moving-image data re-recording process will be described.

  FIG. 8 is a flowchart showing the operation of the re-recording process of moving image data by the digital camera 1. This re-recording process is executed by the user instructing a re-recording mode by a predetermined operation after moving image shooting. At that time, the user can arbitrarily designate whether to perform “overwrite save” or “new save” through the save method designation screen 61 as shown in FIG.

  First, a moving image file to be re-recorded is specified in the memory 38 (step C11). This is performed, for example, when the user explicitly indicates a moving image file to be re-recorded by a predetermined operation. Subsequently, a camera shake information file corresponding to the moving image file is specified (step C12). When the moving image file to be re-recorded and the camera shake information file corresponding thereto are specified, the frame position (pointer) to be processed is initialized (step C13), and the moving image data is reproduced and displayed in the following procedure. The

  That is, image data at the current frame position is extracted from the moving image file, and a camera shake vector at the current frame position is extracted from the camera shake information file (steps C14 and C15).

  Here, a portion excluding the peripheral region corresponding to the camera shake vector is extracted from the image data, and the cut-out image data is generated as image data after camera shake correction (step C16). When the image data after the camera shake correction is re-recorded, the saving method designated on the saving method designation screen 61 in FIG. 5 is confirmed (step C17).

  As a result, if “overwrite saving” is designated as the saving method, the clipped image data, that is, the image after camera shake correction is enlarged to the original resolution, and then replaced with the image data of the frame portion. Is recorded in the memory 38 (step C18). On the other hand, if “new saving” is designated, the extracted image data, that is, the image after camera shake correction is enlarged or reduced in accordance with the resolution designated in advance, and then stored in the memory 38 as a new moving image file. Additional recording is performed (step C19).

  Thereafter, in the same manner, the frame position to be processed next is sequentially updated (steps C20 and C21), the image data and the camera shake vector at the updated frame position are read, and the image data is subjected to camera shake correction. (Steps C14 to C19). Then, when the processing for all the frames is completed (No in Step C20), the re-recording process here ends.

  In the re-recording process, the original image data is decompressed and the corrected image data is re-compressed as necessary.

  As described above, moving image data obtained by moving image shooting and camera shake information detected from the moving image data are recorded in the memory 38 in association with each other, and later recorded in the memory 38 as necessary. The moving image data can be corrected using camera shake information corresponding to the moving image data.

  In this case, since moving image data is recorded in the memory 38 in a state where no pixels are missing at the time of moving image shooting, it is possible to obtain moving image data with priority on the angle of view, and if camera shake correction is specified during playback display, Image data corrected for camera shake can also be obtained.

  Furthermore, the moving image data after the camera shake correction can be saved. At this time, the user can arbitrarily specify whether to perform overwriting or new storage. In this case, for example, when the capacity of the memory 38 is small, it is overwritten and saved, and when the original data, that is, the moving image data at the time of shooting with priority on the angle of view is to be saved, it is newly saved. You can use different storage methods accordingly.

  In the embodiment, various processes relating to moving image data, that is, moving image shooting processing, camera shake information addition processing, re-recording processing, and playback / display processing are individually executed. However, these processes are performed in parallel. That's fine.

  FIG. 12 shows a case in which a moving image shooting process, a camera shake information addition process, a re-recording process, and a reproduction display process are performed in parallel by the digital camera 1.

  The camera shake information addition processing is normally performed on the movie file obtained at the time after the movie shooting processing is completed. However, if the processing speed is sufficient, a time lag corresponding to a predetermined number of frames or at the same time as the movie shooting processing is performed. It may be performed in parallel with the video shooting process.

  The re-recording process is normally performed using the camera shake information file obtained at the time after the camera shake information adding process is completed. If there is a sufficient processing speed, the re-recording process may be performed simultaneously with the camera shake information adding process or at a predetermined time. It may be performed in parallel with the camera shake information addition process with a time lag corresponding to the number of frames.

  In addition, the playback display process is normally performed using the camera shake information file obtained at the time after the camera shake information adding process is completed. It may be performed in parallel with the camera shake information addition process with a time lag corresponding to the number of frames. Further, this reproduction display process may be performed simultaneously or in parallel with the re-recording process.

(Second Embodiment)
Next, a second embodiment of the present invention will be described.

  In the second embodiment, in addition to the presence / absence of image stabilization and the storage method described in the first embodiment, the user can arbitrarily specify the degree of followability to the movement of camera shake and the pixel ratio for determining the subject. Can be specified.

  FIG. 13 is a flowchart showing a setting process by the digital camera 1 in the second embodiment of the present invention. This setting process is executed, for example, when the “camera shake correction” item 52 is selected on the correction designation screen 51 of FIG.

  That is, when the user designates “camera shake correction” on the correction designation screen 51, a setting screen 71 as shown in FIG. 14 is displayed on the display unit 19 (step D11). The setting screen 71 is provided with items 72 and 73 for designating the degree of follow-up to the movement of camera shake, and items 74, 75 and 76 for designating the ratio of determination pixels.

  When setting the degree of camera shake followability, if the item 72 or the item 73 on the setting screen 71 is selected with a cursor or the like (Yes in step D12), the degree of followability corresponding to the selected item is displayed as setting information. 2 is stored in the setting storage unit 32a of the control unit 32 (step D13). In this example, the item 72 is selected to set “following” “low”, and the item 73 is selected to set “following” “high”. When the degree of followability is not specified by the user, for example, “low” followability is set as a default.

  Here, with reference to FIG. 15, the followability to the movement of the camera shake by the digital camera 1 will be described.

  Assume that the amount of movement of an image at the time of moving image shooting is in a state as shown in FIG. The amount of movement includes movement when the user intentionally pans the camera, in addition to movement due to camera shake not intended by the user. In addition, even intentional panning by the user may be unsightly due to excessive movement when actually played back, and it is desirable to remove such image movement as well as camera shake. . Here, an unfavorable image motion component including camera shake and excessive panning is referred to as a hand shake component (camera shake vector), and a preferable image motion component due to other appropriate panning is a pan component (pan vector). ). In addition, processing for removing a camera shake component from image motion is referred to as camera shake correction processing.

  However, it is difficult to separate the camera shake component to be removed and the other pan component before actually reproducing and viewing the moving image. Here, the tracking ability is a guideline for dynamically identifying where the shake component is from and where the panning component is, and changes the degree of camera shake correction when performing the shake correction processing according to this tracking ability. .

  FIG. 15B and FIG. 15C show a state in which the camera shake component is removed from the state of FIG.

  FIG. 15B shows the amount of motion of the image when the followability is set low. When the tracking capability is set low, the degree of camera shake correction for the image data of each frame is low (the proportion of motion components that are regarded as camera shake is less in the overall motion of the image), and small motion due to camera shake As a result, a portion that protrudes from the screen is corrected for each frame. On the other hand, FIG. 15C shows the amount of motion of the image when the followability is set high. When the tracking capability is set high, the degree of camera shake correction for the image data of each frame increases (the proportion of motion components that are considered to be camera shake in the overall motion of the image increases). A part that protrudes from the screen is corrected every frame including a large movement caused by camera panning.

  In the first embodiment described above, if the photographer has panned the camera so that the subject indicated by Pc in FIG. 9 is placed in the center of the frame, Pc may be specified as the reference subject. However, when the subject indicated by Pc is very small or is not the subject intended by the photographer, it is not preferable to use the subject as a reference subject. Therefore, a condition for specifying each subject portion as a reference subject can be designated by the pixel ratio of each subject portion in the entire image data.

  Returning to the flowchart of FIG. 13, when setting the pixel ratio for determining the subject, if any one of the items 74 to 76 on the setting screen 71 is selected with a cursor or the like (Yes in step D14), The pixel ratio corresponding to the selected item is stored as setting information in the setting storage unit 32a of the control unit 32 shown in FIG. 2 (step D15). Here, “ratio A” is set by selecting the item 74, “ratio B” is set by selecting the item 75, and “ratio C” is set by selecting the item 76. It is assumed that the ratio increases in the order of “Ratio A”, “Ratio B”, and “Ratio C”. Further, when the ratio of the pixel for determination is not designated by the user, for example, “ratio A” is set as a default.

  Next, camera shake information addition processing in the second embodiment will be described.

  FIG. 16 is a flowchart showing the operation of camera shake information addition processing in the second embodiment. Note that the basic processing flow is the same as that in FIG. 6, and the difference is that the processing is performed in consideration of the followability of camera shake correction and the setting information of the ratio of determination pixels.

  First, a moving image file to which camera shake information is added is specified in the memory 38 (step E11). This is performed, for example, when the user explicitly indicates a moving image file to which camera shake information is added by a predetermined operation. When a moving image file to which camera shake information is added is specified, a frame position (pointer) for sequentially processing each frame image constituting the moving image file (moving image data) is initialized (step E12). The camera shake information file is created in such a procedure.

  That is, a plurality of frame images before the current frame position are compared, and a motion vector (movement direction and movement amount) is detected for each pixel (step E13). Subsequently, pixels having the same motion vector are grouped based on the detected motion vector of each pixel (step E14).

  Here, among the groups configured with the number of pixels equal to or greater than the pixel ratio for determination set on the setting screen 71, the group with the smallest moving amount is specified as the reference subject and the other groups are specified as the moving subjects (step E15). ). In this case, the smaller the ratio of the determination pixels, the more the subject is determined including, for example, small insects.

  When the reference subject is specified in this way, the motion vector of the reference subject is temporarily stored in a work memory (not shown) or the like as a camera movement vector caused by camera panning or camera shake at the time of shooting the current frame. (Step E16).

  Subsequently, the number of averaged frames is determined in accordance with the degree of camera shake followability set in the setting storage unit 32a (step E17). As described with reference to FIG. 14, the degree of camera shake followability can be arbitrarily specified in advance by the user. The higher the followability, the greater the number of average frames, and the higher the result of camera shake correction. .

  When the number of averaged frames is determined, a camera Pann vector is obtained from the average value of the camera movement vectors obtained corresponding to the frame images for the number of averaged frames before the current frame position (step E18). Then, the camera shake vector at the current frame position is obtained from the difference between the camera movement vector at the current frame position obtained at step E16 and the camera pan vector obtained at step E18 (step E19).

  The camera shake vector obtained in this way is associated with the number added to the frame and is additionally stored in the camera shake information file (step E20).

  Thereafter, similarly, the frame position to be processed next is sequentially updated (steps E21 and E22), and the camera shake vector at the frame position is additionally stored in the camera shake information file (steps E13 to E20). Then, when the processing for all the frames is completed (No in Step E21), the camera shake information file is recorded in the memory 38 in association with the moving image file (Step E23), and the camera shake information addition process is completed here. To do.

  As described above, a camera shake information file in which the degree of camera shake tracking and the ratio of pixels for determination is arbitrarily set is created. As described above, the camera shake information file is used for camera shake correction of a moving image file during reproduction and display. In this case, for example, if the camera shake information file is set to have a high camera shake tracking capability, as described with reference to FIG. 15, not only camera shake but also movement including excessive panning of the camera is corrected.

  As shown in FIG. 17, it is also possible to add a plurality of camera shake information files to one moving image file. In this case, when the user designates a desired camera shake information file at the time of reproduction display or re-recording, camera shake correction processing for the moving image file is performed based on the designated camera shake information file.

  In the example of FIG. 17, with respect to the moving image file 1, a camera shake information file 1-a set with a high camera shake tracking capability and a camera shake information file 1-b set with a low camera tracking capability are associated. Yes.

  Further, another moving image file 2 includes a camera shake information file 2-a in which the camera shake followability is set high, a camera shake information file 2-b in which the camera shake followability is set low, and a camera shake information file 2 -C is associated. This camera shake information file 2-c is a file of camera shake information detected by the gyro sensor 42 shown in FIG. When the gyro sensor 42 is mounted on the digital camera 1, camera shake information can be detected from the movement of the camera during moving image shooting.

  In the second embodiment, the user designates the degree of camera shake followability (low / high) and determines the average number of frames according to the designated degree of camera shake followability. However, the degree of camera shake correction may be changed by another method.

  In each of the above embodiments, a digital camera has been described as an example. However, the present invention is not limited to this. For example, in addition to a digital movie camera, a mobile phone or a PDA (Personal Digital Assistant).

  In short, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying constituent elements without departing from the scope of the invention in the implementation stage. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the respective embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

FIG. 1 is a diagram showing an external configuration when a digital camera capable of shooting a moving image is taken as an example of the imaging apparatus according to the first embodiment of the present invention. FIG. 2 is a block diagram showing an electronic circuit configuration of the digital camera in the embodiment. FIG. 3 is a diagram showing a relationship between a moving image file and a camera shake information file of the digital camera in the embodiment. FIG. 4 is a diagram showing an example of a correction designation screen for designating the presence / absence of camera shake correction when reproducing and displaying moving image data in the embodiment. FIG. 5 is a diagram showing an example of a storage method designation screen for designating a storage method when re-recording moving image data in the embodiment. FIG. 6 is a flowchart showing the operation of camera shake information addition processing by the digital camera in the embodiment. FIG. 7 is a flowchart showing the operation of the reproduction / display processing of moving image data by the digital camera in the embodiment. FIG. 8 is a flowchart showing the operation of re-recording processing of moving image data by the digital camera in the embodiment. FIG. 9 is a diagram illustrating an example of each frame image of moving image data obtained during moving image shooting. FIG. 10 is a diagram showing an example in the case where a portion other than the camera shake region is cut out from each frame image of the moving image data. FIG. 11 is a diagram illustrating an example of displaying the cut-out image data in accordance with the resolution of the display device. FIG. 12 is a diagram showing a place where the digital camera performs moving image shooting processing, camera shake information addition processing, re-recording processing, and playback display processing in parallel. FIG. 13 is a flowchart showing setting processing by the digital camera in the second embodiment of the present invention. FIG. 14 is a diagram showing an example of a setting screen for the degree of camera-shake tracking and the ratio of determination pixels in the embodiment. FIG. 15 is a diagram for explaining camera shake followability by the digital camera in the embodiment. FIG. 16 is a flowchart showing the operation of camera shake information addition processing by the digital camera in the embodiment. FIG. 17 is a diagram showing a relationship between a moving image file and a camera shake information file of the digital camera in the embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Digital camera, 2 ... Body, 3 ... Shooting lens, 4 ... Self-timer lamp, 5 ... Optical finder window, 6 ... Strobe light emission part, 7 ... Microphone part, 8 ... Power key, 9 ... Shutter key, 10 ... Photographing Mode key, 11 ... Playback mode key, 12 ... Optical viewfinder, 13 ... Speaker unit, 14 ... Macro key, 15 ... Strobe key, 16 ... Menu (MENU) key, 17 ... Ring key, 18 ... Set key, DESCRIPTION OF SYMBOLS 19 ... Display part, 20a, 20b ... Zoom key, 21 ... Motor, 22 ... Lens optical system, 23 ... CCD, 24 ... Timing generator (TG), 25 ... Driver, 26 ... Sample hold circuit (S / H), 27 A / D converter, 28 Image processing circuit, 29 DMA controller, 30 DRAM interface (I / F), 31 DRAM, 3 Control unit 32a Setting storage unit 33 VRAM controller 34 VRAM 35 Digital video encoder 36 Key input unit 37 JPEG circuit 38 Memory 39 Audio processing unit 40 USB interface (I / F), 41... Strobe drive unit, 51... Correction designation screen, 61... Storage method designation screen, 71.

Claims (12)

Moving image capturing means for acquiring image data of each frame continuously output from the image sensor as moving image data in a state in which there is no pixel loss associated with camera shake correction;
Camera shake information generating means for generating camera shake information for the video data obtained by the video shooting means;
Correction designation means for designating whether or not to correct camera shake;
In accordance with the camera shake correction designation by the correction designating means, the camera shake correction processing is performed on the moving image data acquired by the moving image photographing means based on the camera shake information generated by the camera shake information generating means, and the moving image after the camera shake correction is performed. An imaging apparatus comprising: moving image processing means for reproducing and displaying or re-recording data. Recording means for recording and storing the moving image data acquired by the moving image photographing unit at least until a change of the recording state is instructed, in a state in which there is no pixel loss associated with camera shake correction,
The correction designating unit causes the user to designate whether or not to correct the moving image data at any timing after the moving image data is acquired by the moving image shooting unit,
The moving image processing means performs a camera shake correction process on the moving image data recorded and stored in the recording means in a state in which no pixels are lost due to the camera shake correction when the camera shake correction is specified by the correction specifying means. The imaging device according to claim 1, wherein the imaging device is performed. A recording unit that records and stores the camera shake information generated by the camera shake information generating unit in association with the moving image data acquired by the moving image capturing unit;
The correction designating unit designates whether or not to correct camera shake for any moving image data recorded and stored in the recording unit;
The moving image processing unit performs a camera shake correction process on the moving image data designated to be corrected by the correction designating unit based on camera shake information recorded and stored in the recording unit corresponding to the moving image data. The imaging apparatus according to claim 1. The correction designating means includes designating a degree of camera shake correction;
The image processing unit controls the camera shake correction processing so that a camera shake state after the correction of the moving image data changes according to a degree of camera shake correction specified by the correction specifying unit. The imaging device according to any one of 1 to 3. A storage method specifying means for specifying a storage method of moving image data at the time of re-recording is provided.
4. The imaging apparatus according to claim 2, wherein the moving image processing unit re-records the corrected moving image data in the recording unit in accordance with a storing method designated by the saving method designating unit. The saving method includes overwrite saving and new saving,
The moving image processing unit re-records the corrected moving image data in the recording unit in place of the original moving image data when the overwrite method is designated by the saving method designating unit, and the new saving is designated. 6. The imaging apparatus according to claim 5, wherein the corrected moving image data is re-recorded in a region different from the original moving image data of the recording means. On the video data in which a plurality of subjects having different movements are photographed, a specifying unit for specifying a reference subject serving as a reference for determining camera shake is provided.
The imaging apparatus according to claim 1, wherein the camera shake information generating unit generates camera shake information focusing on a movement of a reference subject specified by the specifying unit. A pixel ratio specifying means for specifying a pixel ratio for subject determination;
8. The imaging apparatus according to claim 7, wherein the specifying unit specifies the reference subject based on the pixel ratio specified by the pixel ratio specifying unit. Acquiring image data of each frame continuously output from the image sensor as moving image data in a state where there is no pixel loss associated with camera shake correction;
Generating camera shake information indicating the movement of the video data camera shake;
A step for specifying whether or not to correct camera shake;
Moving image processing based on the camera shake information when the camera shake correction is designated, and reproducing and displaying or re-recording the moving image data after the camera shake correction. Method. A program executed by a computer mounted on an imaging device having a video shooting function,
In the computer,
A function of acquiring image data of each frame continuously output from the image sensor as moving image data in a state where there is no pixel loss accompanying camera shake correction,
A function of generating camera shake information indicating movement of camera shake of the video data;
A function to specify whether or not to correct camera shake,
A program for performing a camera shake correction process on the moving image data based on the camera shake information and reproducing and displaying or re-recording the moving image data after the camera shake correction when camera shake correction is designated. . Acquiring image data of each frame continuously output from the image sensor as moving image data in a state where there is no pixel loss associated with camera shake correction;
Generating camera shake information indicating the movement of the video data camera shake;
Recording the generated camera shake information in a predetermined memory in association with the video data;
A step for specifying whether or not to correct camera shake;
A step of performing camera shake correction processing of the moving image data based on camera shake information corresponding to the moving image data recorded in the memory, and reproducing and displaying or rerecording the moving image data after the camera shake correction when camera shake correction is designated A moving image processing method comprising: and. A program executed by a computer mounted on an imaging device having a video shooting function,
In the computer,
A function of acquiring image data of each frame continuously output from the image sensor as moving image data in a state where there is no pixel loss accompanying camera shake correction,
A function of generating camera shake information indicating movement of camera shake of the video data;
A function of recording the generated camera shake information in a predetermined memory in association with the video data;
A function to specify whether or not to correct camera shake,
A function for performing camera shake correction processing of the moving image data based on the camera shake information corresponding to the moving image data recorded in the memory, and reproducing and displaying or rerecording the moving image data after the camera shake correction when camera shake correction is designated. A program characterized by realizing and.

Images