JP5164610B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus that records image data captured by an imaging unit on a recording medium, and a control method thereof.

  Conventionally, a digital camera that converts an image signal captured by an image pickup device such as a CCD into a digital signal, processes the digital signal into a digital video signal of luminance color difference, and then performs compression processing such as JPEG and records it on a recording medium Has been developed.

  Image sensors in digital cameras have become higher in resolution, and some have exceeded 10 million pixels. On the other hand, there are an increasing number of devices having a function of reading out image signals at high speed and recording them as moving images by skipping horizontal and vertical lines of the image sensor. In other words, in recent digital cameras, the above-described still image shooting mode for performing still image shooting and the moving image shooting mode for moving image shooting are provided, and both shooting modes are switched by a user operation. .

In recent years, it has become possible to shoot still images during moving image recording (Patent Document 1). This document describes a technique for solving the problem that a moving image compressed data file in moving image shooting interrupted by still image shooting is divided into two with a still image sandwiched when still image shooting is performed during moving image recording. Have been described. Specifically, one frame immediately before the still image shooting period is used as a substitute frame, and the substitute frame is continuously stored in the moving image file only during the still image shooting period so that the moving image is not divided. To do. Furthermore, this document also discloses a technique for synchronizing the start timing for resuming the moving image shooting process by the imaging unit after the still image shooting is completed with the moving image frame shooting period in the moving image shooting process before the moving image shooting process is stopped. doing. In addition, a technique that uses special image data for a substitute frame that replaces a missing moving image frame or uses a non-image screen is also known (Patent Document 2).
JP 2004-201282 A JP 2003-8948 A

  However, in the conventional technology as described above, a black image (blackout image), a frame immediately before / after a moving image, or a substitute frame (stop motion image) from a still image is copied and inserted into a file, so the file size is large. There was a problem of becoming.

  Further, even if the same stop motion image is input to the image codec, there is a problem that the compression rate cannot be increased as expected because the reference image is degraded from the input image.

  The present invention is for solving the above-described problems, and in order to take a still image during moving image shooting, the data size of the moving image data is reduced while the moving image recording is interrupted, and writing to the recording medium is performed. The purpose is to reduce the load.

In order to solve the above problems, an imaging apparatus of the present invention is an imaging apparatus that records image data captured by an imaging unit on a recording medium, and is a moving image obtained by continuously capturing images by the imaging unit. A moving image shooting control means for controlling a moving image shooting operation for recording the image as a moving image file on the recording medium, and a still image shooting operation for recording a still image obtained by taking an image with the imaging means as a still image file on the recording medium When a still image shooting operation instruction is input during a period in which the moving image shooting operation is being performed, the still image shooting control unit controls the still image shooting operation. recording the still image file on the recording medium, the video recording control means interrupts the moving image capturing operation, blanking the first image to replace the moving image corresponding to the period that the suspended Kkuauto insert the image, the black differential information after the out-image repeat that inserts an image compressed between frames is zero, the number of the inserted image is not less than the predetermined number and the inserted images are GOP Is the last frame of the next GOP, a stop motion image obtained from a still image corresponding to the still image shooting operation is used, and the difference information is zero after the I frame. A moving image file is generated and recorded on the recording medium by repeating the insertion of a stop motion image that has been compressed between the frames up to the number of frames corresponding to the GOP .

  According to the present invention, since still image shooting is performed during moving image shooting, the data size of moving image data can be reduced while moving image recording is interrupted, and the writing load on the recording medium can be reduced.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block configuration diagram of a digital camera 100 as an imaging apparatus in the present embodiment.

  In the figure, reference numeral 6 denotes a photographing lens, which is composed of one or a plurality of lenses. An exposure control unit 8 includes an aperture, a shutter, and the like, and adjusts the exposure according to the luminance of the subject. Reference numeral 10 denotes an image pickup device as an image pickup means such as a CCD sensor or a CMOS sensor having a number of pixels of about several million pixels, and outputs an analog image signal of each color component obtained by image pickup. Reference numeral 12 denotes an A / D converter for converting an analog image signal from the image sensor 10 into a digital signal. Reference numeral 14 denotes a DRAM which has a buffer function for temporarily storing a digital image signal from the A / D converter 12 and temporarily storing image data after signal processing and data after image compression, which will be described later. Normally, data is written to and read from the DRAM via a memory controller, but this is not shown here.

  Reference numeral 16 denotes a signal processing unit for processing the digital image signal obtained by the A / D converter 12 into a luminance signal and a color difference signal, and generates YUV data. An image codec unit 18 compresses the YUV data from the signal processing unit 16 into JPEG or MPEG format image data, or decompresses JPEG or MPEG format image data into YUV data.

  Reference numeral 20 denotes a recording medium interface for writing / reading the image-compressed data to / from a recording medium such as a CF (Compact Flash) card or an SD (Secure Digital) card. Reference numeral 21 denotes a removable recording medium such as the above-described CF card or SD card. The recording medium to be used is not limited as long as it is a randomly accessible recording medium. For example, an optical disk (DVD) or a hard disk may be used. Reference numeral 22 denotes a display unit composed of a color liquid crystal display or the like, which is used for displaying a viewer, displaying a review based on a photographed image, and reproducing and displaying an expanded image. The display unit 22 also includes a terminal for outputting a video signal to an external display device such as a TV.

  Reference numeral 24 denotes a resizing unit that resizes the YUV data output from the signal processing unit 16 to an arbitrary size. Reference numeral 26 denotes a CPU for controlling each processing unit described above and a data flow described later. Reference numeral 27 denotes a ROM that stores a program (firmware) related to the processing procedure of the CPU 26 and various types of information. As will be described in detail later, the ROM 27 stores in advance MPEG-encoded blackout image data, skipped macroblock image data, and wave data of pseudo shutter sound. Each processing unit is connected to the bus 33 of the CPU 26. Further, although not shown, a DMA controller is provided between each processing unit and the DRAM 14 for performing DMA transfer without using Read / Write with the CPU. Further, although not shown, data is directly transmitted and received from the A / D converter 12 to the signal processing unit 16, and from the signal processing unit 16 to the image codec unit 18 and the resizing unit 24, although not shown. There is a pass.

  Reference numeral 28 denotes an audio processing unit for recording and reproducing audio, 30 is a microphone for inputting audio when recording, and 32 is a speaker for reproducing audio. The audio processing unit 28 also performs audio A / D conversion and D / A conversion to record and reproduce audio information. Reference numeral 23 denotes an operation unit including a release switch 23a used for instructing to capture a still image, a moving image recording switch 23b for instructing ON / OFF of moving image recording, and various buttons and dial switches.

  Next, the moving image shooting operation in the present embodiment will be described. In this embodiment, the CPU 26 has a function of moving image shooting control, and the moving image shooting operation is executed by the CPU 26 controlling each processing unit in FIG.

  The light that has passed through the photographic lens 6 passes through the aperture and shutter exposure control unit 8 and is subjected to exposure control so as to form an image on the image sensor 10. When shooting a movie, the shutter of the mechanical mechanism is opened and the exposure is controlled by the aperture. The subject image is photoelectrically converted by the image sensor 10 and sent to the A / D converter 12 as an analog signal. The A / D converter 12 converts the analog image signal into a digital image signal, and the digital image signal is transferred to the signal processing unit 16 via the bus 33. At the time of moving image shooting, not all pixels are read out from the image sensor 10, but the pixels thinned out in the horizontal direction and the vertical direction are read out in accordance with the recording frame rate. This is to reduce the load on the resizing unit 24. In the signal processing unit 16, horizontal and vertical filter processing, aperture correction processing, gamma processing, and the like are performed according to the moving image shooting mode, and the processed image signal is transferred to the resizing unit 24.

  The resizing unit 24 resizes the image signal to the size of VGA (horizontal 640 pixels, vertical 480 lines), and transfers it to the DRAM 14. The VGA image data on the DRAM 14 is transferred to the display unit 22 and displayed as a through image on a liquid crystal mounted on a TV or a camera. In the present embodiment, the state up to this display process is referred to as a standby state during moving image shooting.

  In this state, when the moving image recording switch 23b provided in the operation unit 23 is pressed, the CPU 26 causes each processing unit to start a moving image recording operation.

  That is, in parallel with the image codec unit converting the VGA image data on the DRAM 14 into MPEG data, the audio processing unit 28 converts the audio captured by the microphone 30 into wave format audio data. Then, the CPU 26 uses these MPEG data and audio WAVE data to add a header or the like to generate a moving image file in the AVI format. The generated moving image file is recorded on the recording medium 21 via the recording medium interface 20. This moving image recording process is executed until the moving image recording switch 23b is pressed again.

  Hereinafter, a state satisfying both the standby state during moving image shooting and the state in which the moving image recording process is executed is referred to as a moving image shooting operation state. Although the video file format AVI format is described here, it may be a MOV format or an MPEG format. The audio data format may be a compressed audio data format.

  Next, the still image shooting operation in this embodiment will be described. In the present embodiment, the CPU 26 has a still image shooting control function, and the still image shooting operation is executed by the CPU 26 controlling each processing unit in FIG.

  In the stand-by state of still image shooting, almost the same processing as that of the above-described moving image standby state is performed, and a through image of an image captured by the image sensor 10 is displayed on the display unit 22.

  The release switch 23a provided in the operation unit 23 is composed of a two-stage switch. The switch SW1 is turned on when the switch SW1 is half-pressed, and the switch SW2 is turned on when the switch is fully pressed. When the switch SW1 is turned on, a shooting preparation state is set, and when the switch SW2 is turned on, a still image shooting operation state is set.

  More specifically, in the shooting preparation state in which the switch SW1 is ON, automatic ranging (AF) and automatic photometry (AE) necessary for still image shooting are performed. When the switch SW2 is turned on to switch from the ready state to the still image shooting operation, the front-curtain electronic shutter is closed, the reading of the image sensor 10 is stopped, and the exposure mode is changed. Then, the aperture is set according to the exposure setting calculated in advance from the luminance of the subject in the shooting preparation stage, and the shutter is opened and closed at the shutter speed calculated after discharging the excess charge of the sensor.

  After the shutter is completely closed, the image sensor 10 is switched from the exposure mode to the readout mode, and the data of all the pixels is read out in a plurality of fields.

  The read signal is converted into a digital signal by the A / D converter 12, and the RAW data from the A / D converter 12 is transferred to the DRAM 14 via the bus. When reading from the image sensor 10 is completed, development processing is started.

  In this development processing, first, RAW data stored in the DRAM 14 is read and transferred to the signal processing unit 16. The signal processing unit 16 processes horizontal / vertical filter processing, aperture correction processing, gamma processing, and the like in a still image shooting mode, and transfers the processed YUV data to the image codec unit 18. The image codec unit 18 converts the YUV data into JPEG data and writes it back to the DRAM 14.

  The development process is from the reading of the RAW data from the DRAM 14 to the writing back of the JPEG data to the DRAM 14.

  Thereafter, the generated still image data (JPEG data) is written in the recording medium 21 as a still image file. In addition, the YUV data from the signal processing unit 16 during the development processing is also transferred to the resizing unit 24. The resizing unit 24 resizes the transferred YUV data to about VGA size for review image display, and writes it back to the DRAM 14. When the above development processing is completed, the VGA size YUV data is transferred to the display unit 22 and displayed on the liquid crystal screen on the TV or camera for displaying the review image for confirming the shooting, thereby displaying the review image. it's shown.

  As described above, the moving image shooting operation and the still image shooting operation in the present embodiment have been described, but in the embodiment, still image shooting is permitted even during moving image recording. That is, it is possible to execute the still image shooting process and the still image recording process by the still image shooting operation while continuing the moving image recording process. Hereinafter, the processing of the CPU 26 when taking a still image during moving image recording will be described with reference to the flowchart of FIG.

  Step S200 in FIG. 2 shows a state in which a moving image is shot and recorded according to the moving image shooting operation described above. That is, an AVI file that is a moving image is being constructed on the recording medium 21 by pressing the moving image recording button 23 b in the operation unit 23.

  In step S202, whether or not the half-pressing detection switch SW1 of the release switch 23a is ON is monitored and determined during the moving image recording. When the switch SW1 is OFF, the normal moving image recording operation is continued.

  When the release switch is pressed halfway and the switch SW1 is turned ON, the process proceeds to step S204, and AF (distance measurement) and AE (photometry) for still image shooting preparation are performed in parallel with moving image recording.

  Next, in step S206, the switch SW1 is monitored, and when the switch SW1 is kept on, the AF and AE operations are continued. On the other hand, when the switch SW1 is turned OFF, the still image shooting preparation is completed. However, even in that case, the CPU 26 returns to step S202 and continues the moving image recording process.

  Next, when the switch SW1 is in an ON state and the preparation for photographing is completed, the process proceeds to step S208, and monitoring and determination are performed as to whether or not the release switch is fully pressed, that is, whether or not the switch SW2 is ON. To do.

  If the switch SW2 is OFF and the switch SW1 remains ON, the AF and AE are kept locked. If it is determined that the switch SW2 is turned on, the process proceeds to step S210, and a still image shooting operation is executed.

  The still image shooting operation in step S210 is substantially the same as the above-described still image shooting operation. The difference is that since still image shooting is performed during moving image recording, recording of the moving image is temporarily interrupted. At this time, the driving mode of the image sensor 10 is switched, and a moving image frame based on a moving image cannot be generated. In the present embodiment, a substitute frame and a skipped macroblock image are inserted in place of the missing video frame. This process will be described in detail using the moving image file generation flow when the still image shooting operation of FIG. 3 is performed. When the still image shooting operation is completed, the interrupted moving image shooting operation is prepared in step S212, and the moving image shooting operation state is returned in step S214.

  The moving image file generation flow will be described with reference to FIG. This procedure is executed almost in parallel with the still image shooting operation. Step S301 indicates that the process is started in response to a determination of Yes in step S208 of FIG. First, in step S301, moving image shooting is interrupted, and the CPU 26 starts this processing.

  Next, the CPU 26 proceeds to step S302, and starts counting the number N1 of blackout images set in advance as corresponding to a period during which the moving image shooting operation is interrupted to capture a still image (initial value i = 1). .

  In step S303, the CPU 26 determines whether it is the first frame of the GOP in order to insert the blackout image and the skipped macroblock image as the first substitute frame of the moving image frame.

  If the substitute frame is the first frame of the GOP (YES in step S303), the blackout image I frame encoded by the image codec unit 18 is output in step S304.

  If the substitute frame is not the first frame of the GOP (NO in step S303) and i = 1 (YES in step S305), the blackout image P frame encoded by the image codec unit 18 is output (step S306).

  If the substitute frame is not the first frame of the GOP (NO in step S303) and i is not 1 (NO in step S305), a skipped macroblock image (SMB image) is output (step S307).

  Here, the I frame is called an intra-coded frame and is a key frame. The P frame is a Predicted Frame and is a difference frame (including a B frame). A blackout image is a non-image pure black MPEG data, and is an image data having a very small data size because the black image is MPEG-compressed.

  A skipped macroblock image is a type of image compressed between frames. For example, the difference information between frames, for example, an image having a motion vector of zero. The skipped macroblock image is image data having a very small data size of about several tens of bytes even in the VGA size. This skipped macroblock image is the skipped macroblock image 27b stored in the ROM 27 of FIG. 1, and is MPEG-encoded with the VGA size.

  The skipped macroblock image 27b is read from the ROM 27 and inserted as a first substitute frame on the moving image file (AVI data).

  Further, when a stop motion image described later is a P frame, a blackout image is required as a reference frame, and therefore the image codec unit 18 generates a blackout image. FIG. 5 is a diagram showing the structure of a moving image file when a still image is shot during moving image recording.

  In FIG. In order to generate the stop motion image P frame of No. 8, frame No. 7 information is required. Frame No. 7 is a frame number. Six pieces of information are required. Frame No. 6 is a frame number. Five pieces of information are required. Frame No. 5 is a frame number. Four pieces of information are required.

  Therefore, frame No. 8 is a frame number. 4 information is necessary, so 4, the image codec unit 18 generates a blackout image P frame. Alternatively, the frame number for the image codec unit 18 to generate the stop motion image P frame. 4 information is temporarily stored.

  Thus, frame No. 5-Frame No. 7 is a skipped macroblock image. Frame No. 7 4 using the information of frame No. 8 is generated.

  When the stop motion image is an I frame, a blackout image is not necessary as a reference frame, and therefore the image codec unit 18 does not generate a blackout image. In this case, the MPEG-compressed blackout image 27a is read from the ROM 27 and inserted as a first substitute frame on the moving image file (AVI data). Here, the MPEG-compressed blackout image 27a is read from the ROM 27. However, the CPU may generate the blackout image 27a.

  The length (period) of the first alternative frame is equal to the longest exposure time in the camera green mode (AUTO mode), and is about 1/8 second (125 mSec). When a moving image is captured at 30 frames / second, the number of frames in 1/8 second is “4”. Therefore, the blackout image in the ROM 27 (or the blackout image generated by the image codec unit 18) is inserted into the AVI file once and the skipped macroblock image is inserted three times. Instead of the blackout image, an image having a message such as “still image shooting” may be used.

  If N1 blackout images have not been reached in step S308 (NO in step S308), i is incremented in step S309, and the process returns to step S303. If the number of blackout images reaches N1 in step S308 (YES in step S308), the process proceeds to step S310.

  Next, in step S310, the pseudo shutter sound data 27c placed on the ROM 27 is read and inserted into the audio data of the moving image file (AVI data). At the same time, the pseudo shutter sound data 27 c is output to the sound processing unit 28 and reproduced from the speaker 32. The time length of the pseudo shutter sound data is substantially the same as the time length of the previous blackout image data.

  At this time, if the pseudo data is suddenly replaced with the sound data recorded from the microphone 30, a high-frequency component is generated due to a sudden switching of data, and noise sound data is generated due to aliasing. Therefore, during the period of replacement with pseudo data (which is also the period during which the pseudo shutter sound is reproduced by the speaker 32), the sound processing unit 28 first mutes (silences) the period. A pseudo shutter sound is inserted on the muted audio data. As a result, high-quality pseudo shutter sound can be inserted into the audio data without generating high-frequency noise due to aliasing.

  In step S311, the process waits for the completion of the still image developing process in the above-described still image shooting operation. When the development is completed, the process proceeds to step S312.

  After that, the CPU 26 JPEG-encodes the developed still image by the image codec unit 18 and generates one still image file from the encoded data as the processing in step S210 in FIG. The generated still image file is stored in the recording medium 21 separately from the moving image file.

  In parallel with this processing, or thereafter, in this step S312, the CPU 26 generates a review image of VGA size from the developed still image by the resizing unit 24, and starts counting the number N2 of stop motion images. (Count initial value j = 1).

  Next, it is determined whether it is the first frame of the GOP. If it is the first frame (YES in step S313), the review image data is encoded as a stop motion image I frame by the image codec unit 18 and output (step S314).

  If it is not the first frame of the GOP (NO in step S313) and j = 1 (YES in step S315), the review image data is encoded and output as a stop motion image P frame by the image codec unit 18 (step S316). .

  If it is not the first frame of the GOP (NO in step S313) and j is not 1 (NO in step S315), a skipped macroblock image (SMB image) is output (step S317).

  If j has not reached N2 (NO in step S318), j is incremented in step S319, and the process returns to step S313. When j reaches N2 (YES in S step S318), the process proceeds to step S212 (step S320).

  The encoded VGA size image (stop motion image) is the second alternative frame data. Then, following the blackout image that is the first alternative frame, the second alternative frame is the skipped macroblock image or stop motion image for the number of video frames for which video recording was interrupted during the development processing period. compensate. That is, the second substitute frame is repeatedly inserted into the AVI file an appropriate number of times.

  The audio data during the second alternative frame period is inserted into the AVI file by converting the sound from the microphone 30 into WAVE data by the audio processing unit 28. In other words, when this AVI file is played back, a sound obtained by synthesizing the recorded sound with the pseudo shutter sound is recorded during the blackout period of the first alternative frame during the period interrupted by the still image shooting. Then, during the stop motion image period of the second alternative frame, a frame corresponding to an actually captured image is reproduced and sound recorded during that time is reproduced.

  Next, in step S320, the process returns to step S212 in FIG. This is an operation of switching the reading of the image sensor 10 for moving images, simultaneously switching the settings of the signal processing unit 16, the resizing unit 24, the image codec processing unit 18 and the like for moving images and switching the data path. In step S214, the first moving image shooting operation state is restored, and shooting for moving image recording is continuously performed.

  FIG. 4 shows a still image shooting sequence during moving image recording. FIG. 4A shows a state (sequence) in which the operating state of the camera changes during shooting. (B) of the figure shows a state (sequence) in which an image for displaying a through image at the time of photographing is changed on the display unit 22. Further, (c) in the figure shows the recording state of the generated moving image file (AVI format) in association with the actual shooting and recording sequence. In other words, it shows a state (sequence) in which the image display on the liquid crystal or TV monitor changes when a moving image file (AVI format) is reproduced. Further, FIG. 6D shows the timing when the created AVI file is written to the medium.

  As will be described later, the recording process to the medium in the moving image shooting operation and the still image shooting operation does not coincide with the timing of FIG. In FIG. 4, the period from the left end of the figure until the switch SW1 of the release switch is turned on indicates the moving image shooting operation state.

  In a period S400 in FIG. 6A, the state of the moving image shooting operation is shown. In this period, the through image of the moving image is displayed on the display unit 22 in the period S418 in FIG. 4B, and the moving image and the audio data are recorded as a moving image file in the period S428 in FIG.

  In the period S402 in FIG. 9A until the next switch SW1 is turned on and the switch SW2 is turned on, a state in which AE and AF for still image shooting preparation are performed while moving image shooting and recording are shown. Yes.

  In a period S420 in FIG. 6B, the state of performing AE and AF for shooting preparation for the still image is displayed on the display unit 22, and in the period S430 in FIG. 4C, the shooting preparation for the still image is performed. AE and AF are recorded in a moving image file.

  In the period until the switch SW2 is turned on, the timings (a), (b), and (c) in the figure are aligned, but not after the switch SW2 is turned on. The sequence will be described below.

  A period S404 in FIG. 6A shows the mode switching state of the image sensor 10 immediately after the switch SW2 is turned on, and the mode is switched from the moving image readout mode to the still image exposure mode. The next period S406 indicates an exposure period in a still image state. For example, the exposure time in the first embodiment is controlled to vary the exposure time from 1/2000 seconds to 1/8 seconds.

  The next period S408 indicates a readout period in which all pixels for still images are read out from the image sensor 10 in a plurality of fields, and for example, a time of about 300 mSec is required.

  The next period S410 indicates the period of the still image development process, and the contents of the development process are the process of writing back the processed JPEG data to the DRAM 14 from the reading of the RAW data in the DRAM 14 described above.

  The next period S412 indicates a period for generating a review image. The YUV data from the signal processing unit 16 having the above-described contents is resized to the VGA size by the resizing unit 24 and written back to the DRAM 14 as a review image.

  In the period from S404 to S412, as shown in FIG. 5B, a blackout image (no image) is displayed on the display unit 22.

  This is to clearly notify the user that the camera is capturing still images and processing the current state during still image shooting processing, and displaying a non-displayed image different from the EVF through image display, The display is the same as the normal still image shooting operation.

  A period S414 indicates a period in which the second alternative frame data to be inserted into the moving image file is created from the review image generated in the period S412. During this period, the VGA size YUV data is compressed into MPEG data by the image codec section 18. Furthermore, the number of missing moving picture frames is made a skipped macroblock image, and a moving picture file is generated together with the audio data at that time.

  A period S424 in FIG. 4B indicates a period during which the review image is displayed on the display unit 22 during the period S414.

  A period S432 in FIG. 6C is a period obtained by combining the mode switching period of the image sensor 10 in the period S404 and the exposure period in the period S406 when a still image is captured on a moving image file with a blackout image of a non-display image. The period supplemented with the skipped macroblock image is shown.

  Although the exposure time is variable depending on the luminance level of the subject, the blackout period S432 is set to the longest exposure time (50 + 80 = 130 mSec) in the first embodiment. Furthermore, a pseudo shutter sound is also fitted in accordance with this period S432.

  Next, a period S434 in FIG. 6C shows a period during which the review frame image inserted in the above-described period S414 is inserted into the missing video frame using the substitute frame data and the skipped macroblock image. This is called stop motion on the file. This period is a period from immediately after the blackout alternative frame period to when the video is returned to.

  In addition, after the still image shooting is completed, it is as follows. First, in a period S416 in FIG. 6A, the state is shown in which the operation returns to the moving image shooting operation again. A period S426 in FIG. 5B shows the display of the moving image through image on the display unit 22 again, and a period S436 in FIG. 4C indicates that the moving image is recorded on the moving image file. Show.

  Normally, the actual shutter operation when shooting a still image is a “closed to open” operation. In order to image the still image shooting operation on a moving image file, the operation is “moving image → shutter closing → stop motion image display of still image → shutter closing → shutter opening → moving image”.

  However, during the moving image shooting operation, when the user recognizes a still image by stop motion, it is more suitable for moving image recording to immediately return to the moving image. Therefore, the sequence of “moving image → shutter closing → still image stop motion image displaying → moving image” is optimal without the shutter operation after the previous stop motion image displaying.

  As described above, the shutter closes the blackout image as a substitute frame and a skipped macroblock image as described above, but the display period of the stop motion image is preferably displayed in a predetermined fixed period.

  Furthermore, while the stop motion display period is to check the captured image, if it is too long, the moving image capturing operation will be hindered. Therefore, this display period is preferably within the second predetermined time, and it is desirable from the viewpoint of camera operation that the second predetermined time is a period until the moving image is restarted during the still image processing period.

  That is, within the period of interruption until returning to the moving image shooting operation, a frame other than the blackout image substitute frame is substituted with the still motion stop motion image.

  In the present embodiment, as a result of the above, when a still image was shot during moving image recording, the still image was displayed as a stop motion image for a period of about 600 mSec.

  Next, recording on a medium using FIG. 4D will be described. In the present embodiment, the moving image AVI file has a data structure in which audio data and video data are arranged every second. The CPU 26 generates the AVI file on the medium by writing the AVI data in the AVI buffer area as a file and writing it to the medium.

  A period S440 in FIG. 4D indicates a period in which the AVI data generated by the moving image shooting is written on the medium. A period S442 in FIG. 6D shows a moving image recording period after the timing when the switch SW1 is turned on, and the recording of the moving image AVI file continues as it is in this period as in S440. This period does not end at the timing when the switch SW2 is turned on, that is, the timing at which the reading of the moving image from the sensor is interrupted and the generation of the AVI data of the moving image is also interrupted.

  This is because the AVI data still remains in the AVI buffer area on the DRAM 14 at this timing, so that the data remaining in the AVI buffer continues to be written to the medium.

  The time when all the data in the AVI buffer area is written to the medium is the timing at which the transition from the period S442 to the period S444 in FIG. A period S444 in FIG. 4D shows the timing for writing blackout image data and skipped macroblock image data corresponding to the blackout period of the above-described period S432 to the medium.

  Since the AVI data for blackout display is generated immediately from the timing when the switch SW2 is turned ON, it can be written immediately after the AVI data remaining in the AVI buffer area is written.

  Further, when the writing speed of the medium is sufficiently fast and the writing to the medium can be performed almost simultaneously with the generation of the AVI file, no data is accumulated in the AVI buffer. Accordingly, the AVI data in which the blackout image data and the skipped macroblock image data are inserted in the period S444 described above can be written immediately after the switch SW2 is turned on.

  Since the data capacity of the AVI in which the blackout image data and the skipped macroblock image data are inserted is small, writing can be performed in a relatively short time. The writing speed to the media varies depending on the media used. When the writing speed is fast, there is no data to be written to the medium during the next period S446 after the writing of the AVI data of the blackout image data and the skipped macroblock image data in the period S444. This period is a time during which writing does not occur.

  The next period S446 indicates the timing for writing a JPEG data file (still image file composed of all the pixels obtained as a result of the development processing corresponding to the recording mode) obtained by shooting a still image to the medium. Since the still image JPEG data is generated at the timing of the above-described period S410, the writing to the medium is performed after that timing. Immediately after the writing of the still image JPEG data in the period S446, the timing at which the AVI data created from the review image after shooting the still image is written to the medium in the period S448 is shown.

  The review image AVI in this period S448 records the second alternative frame period in the period S434 described above. When the AVI data of the review image in the period S448 is written, the AVI data of the moving image generated in the previous period S416 is accumulated in the AVI buffer. Therefore, the AVI data of the moving image is written to the medium at the timing of period S450.

  The above is the sequence of data writing to the media when still image shooting is performed during moving image recording. When the recorded AVI data is reproduced as described above, it is possible to easily display a situation where a still image is captured during moving image recording to the user.

  When the release switch is half-pressed, that is, when the switch SW1 is ON and the switch SW2 is OFF, the mark of the button of the release switch may be combined with the corner in the moving image being imaged. In this case, it becomes possible to notify the user that the user is about to shoot a still image immediately before the blackout image is displayed by still image shooting during moving image playback.

  Next, the structure of a moving image file when a still image is shot during moving image recording will be described with reference to FIG. As described above, there is audio data for one second, followed by a moving image frame.

  In this embodiment, the first I frame of the GOP is generated in units of 15 frames. When the SW2 of the operation unit 23a is turned on during moving image shooting, the moving image shooting is interrupted. Then, a blackout image P frame is output from the image codec unit 18 and one frame is inserted into the AVI. Subsequently, three skipped macroblock images (SMB images) are inserted into the AVI (blackout period).

  Next, when the image codec unit 18 completes the still image shooting development process, the image codec unit 18 generates a stop motion image P frame from the review image and inserts it into the AVI.

  Subsequently, an SMB image is inserted during the stop motion period. For the first frame of the GOP during the stop motion period, a stop motion image I frame is generated from the review image and is inserted into the AVI.

  When the stop motion period ends and the moving image shooting is resumed, the normal moving image shooting operation is resumed, and the moving image frame and the audio data are sequentially recorded in the AVI file.

  As described above, according to the present embodiment, an assisting frame of a moving image file generated by capturing a still image during moving image shooting is set as a skipped macroblock image. With this file structure, the file size of the moving image file can be reduced. In addition, the writing load on the storage medium can be reduced.

<Second Embodiment>
The digital camera of this embodiment is the same as that of the first embodiment shown in FIG. 1, and the processing procedure when a still image is shot during moving image recording is also the same as the flowchart of FIG.

  FIG. 6 is a flowchart showing a moving image file generation processing procedure when the still image shooting operation of step S210 in FIG. 2 is performed.

  In step S601, the moving image shooting is interrupted, and in step S602, counting of the number N2 of stop motion images is started (count initial value j = 1).

  Next, it is determined whether the frame is the first frame of the GOP. If it is the first frame (YES in step S603), the frame image data immediately before starting the still image shooting moving image is encoded by the image codec unit 18 as a VGA sized stop motion image I frame and output. (Step S604).

  If it is not the first frame of the GOP (NO in step S603), the VGA size skipped macroblock image 27b (SMB image) is output from the ROM 27 (step S605).

  If j has not reached N2 (NO in step S606), j is incremented in step S607, and the process returns to step S603. If j reaches N2 (YES in step S606), the process proceeds to step S212 (step S608).

  The encoded VGA size image (stop motion image I frame) is the substitute frame data. This alternative frame may be generated from a review image or an image immediately before the moving image shooting is interrupted, or may be an arbitrary image.

  FIG. 7 is a configuration diagram of a moving image file when a still image is captured during moving image recording in the present embodiment. First, there is data for one second of audio data, and a moving image frame continues.

  Frame No. 4, the video shooting is interrupted due to the SW2 being turned on, and a skipped macroblock image skipping the immediately preceding frame (frame No. 3) continues for the stop motion period.

  The first frame of the GOP existing during the stop motion period (here, the GOP is assumed to be 15 frames) is the frame number. 16. For this frame, an I frame is generated from the image codec unit 18 for the frame (frame No. 3) immediately before the moving image shooting is interrupted.

  When the stop motion period ends, the frame No. The normal moving image shooting is resumed from 20, and the moving image frame is generated by the image codec unit 18. Audio data may be recorded during the stop motion period, or a pseudo shutter sound may be inserted.

  As described above, according to the present embodiment, an encouraging frame of a moving image file generated by capturing a still image during moving image shooting is a skipped macroblock image. This can reduce the file size of the moving image file. In addition, the writing load on the storage medium can be reduced. In addition, since an I frame is inserted for each GOP during the stop motion period, the editing unit is fixed and simple editing is possible. Moreover, although described as I frame and P frame in the embodiment, this embodiment can be applied to an interlaced image. In this embodiment, the MPEG system is used as the format of the moving image data. However, other formats such as H264 can be similarly applied as long as they perform inter-frame compression.

<Third Embodiment>
The digital camera of this embodiment is the same as that of the first embodiment shown in FIG. 1, and the processing procedure when a still image is shot during moving image recording is the same as the flowchart of FIG.

  FIG. 8 is a flowchart showing a moving image file generation processing procedure when the still image shooting operation of step S210 in FIG. 2 is performed.

  The moving image file generation flow will be described with reference to FIG. This procedure is executed almost in parallel with the still image shooting operation. Step S801 indicates that the process is started in response to a determination of Yes in step S208 of FIG. First, in step S801, moving image shooting is interrupted, and the CPU 26 starts this processing.

  Next, the CPU 26 proceeds to step S802, and starts counting the number N1 of blackout images set in advance as corresponding to the period during which the moving image shooting operation is interrupted to capture a still image (initial value i = 1). .

  In step S803, the CPU 26 determines whether it is the first frame of the GOP in order to insert the blackout image and the skipped macroblock image as the first substitute frame of the moving image frame.

  If the substitute frame is the first frame of the GOP (YES in step S803), the blackout image I frame encoded by the image codec unit 18 is output in step S804.

  If the substitute frame is not the first frame of the GOP (NO in step S803) and i = 1 (YES in step S805), the blackout image P frame encoded by the image codec unit 18 is output (step S806).

  If the substitute frame is not the first frame of the GOP (NO in step S803) and i is not 1 (NO in step S805), a skipped macroblock image (SMB image) is output (step S307).

  The blackout image I frame or the blackout image P frame may be created by the CPU 26 in advance.

  The skipped macroblock image 27b is read from the ROM 27 and inserted as a first substitute frame on the moving image file (AVI data).

  When the stop motion image described later is an I frame, it is not necessary to generate a blackout image by the image codec unit 18 because a blackout image is not required as a reference frame. FIG. 9 is a diagram showing the structure of a moving image file when a still image is shot during moving image recording.

  In FIG. 9, the frame No. required when the moving image is resumed. In order to generate the 26 moving image P frames, the frame No. Sixteen pieces of information are required.

  Frame No. 17 to frame No. 25 is supplemented with a skipped macroblock image.

  When the stop motion image is an I frame (in the case of frame No. 16), a blackout image is not required as the reference frame, and therefore the image codec unit 18 does not generate a blackout image.

  In this case, the MPEG-compressed blackout image 27a is read from the ROM 27 and inserted as a first substitute frame on the moving image file (AVI data). Here, the MPEG-compressed blackout image 27a is read from the ROM 27. However, the CPU may generate the blackout image 27a.

  If N1 blackout images have not been reached in step S808 and if it is not the last frame of the GOP (NO in step S808), i is incremented in step S809, and the process returns to step S803.

  If the number of blackout images is N1 or more in step S808 and the last frame of the GOP (YES in step S808), the process proceeds to step S810.

  Next, in step S810, the pseudo shutter sound data 27c placed on the ROM 27 is read and inserted into the audio data of the moving image file (AVI data). At the same time, the pseudo shutter sound data 27 c is output to the sound processing unit 28 and reproduced from the speaker 32. The time length of the pseudo shutter sound data is substantially the same as the time length of the previous blackout image data.

  At this time, if the pseudo data is suddenly replaced with the sound data recorded from the microphone 30, a high-frequency component is generated due to a sudden switching of data, and noise sound data is generated due to aliasing. Therefore, during the period of replacement with pseudo data (which is also the period during which the pseudo shutter sound is reproduced by the speaker 32), the sound processing unit 28 first mutes (silences) the period. A pseudo shutter sound is inserted on the muted audio data. As a result, high-quality pseudo shutter sound can be inserted into the audio data without generating high-frequency noise due to aliasing.

  In step S811, the process waits for the completion of still image development processing in the above-described still image shooting operation. When the development is completed, the process proceeds to step S812.

  After that, the CPU 26 JPEG-encodes the developed still image by the image codec unit 18 and generates one still image file from the encoded data as the processing in step S210 in FIG. The generated still image file is stored in the recording medium 21 separately from the moving image file.

  In parallel with this processing, or thereafter, in this step S312, the CPU 26 generates a review image of VGA size from the developed still image by the resizing unit 24, and starts counting the number N2 of stop motion images. (Count initial value j = 1).

  Next, it is determined whether the frame is the first frame of the GOP. If it is the first frame (YES in step S313), the review image data is encoded and output as a stop motion image I frame by the image codec unit 18 (step S814).

  If it is not the first frame of the GOP (NO in step S813), a skipped macroblock image (SMB image) is output (step S815).

  If j has not reached N2 (NO in step S316), j is incremented in step S317, and the process returns to step S813. If j reaches N2 (YES in step S816), the process proceeds to step S212 (step S818).

  As described above, according to the present embodiment, an encouraging frame of a moving image file generated by capturing a still image during moving image shooting is a skipped macroblock image. This can reduce the file size of the moving image file. In addition, the writing load on the storage medium can be reduced. In addition, since the I frame is inserted in GOP units during the stop motion period, the editing unit is fixed and simple editing is possible. In the embodiment, since the blackout image is generated in units of GOPs, only the I frame that does not use the blackout image as the reference frame is required for the stop motion image, so that it is not necessary to generate the P frame. Therefore, the load on the image codec unit 18 can be reduced. Note that the number of GOPs is not limited to one and may be more than that.

  In this embodiment, the I frame and the P frame are described, but this embodiment can also be applied to an interlaced image. In this embodiment, the MPEG system is used as the format of the moving image data. However, the present invention can be similarly applied to other formats such as H264 as long as inter-frame compression is performed.

  Further, the stop motion image that is the second period may be generated in units of GOP as in the first period.

  Accordingly, even when the correlation between the stop motion image and the moving image at the time of the next moving image restart is small, the rate control is facilitated because the moving image recording is restarted from the I frame.

<Fourth Embodiment>
The digital camera of this embodiment is the same as that of the first embodiment shown in FIG. 1, and the processing procedure when a still image is shot during moving image recording is the same as the flowchart of FIG.

  FIG. 10 is a flowchart showing a moving image file generation processing procedure when the still image shooting operation of step S210 in FIG. 2 is performed.

  The moving image file generation flow will be described with reference to FIG. This procedure is executed almost in parallel with the still image shooting operation. Step S1001 indicates that the process starts in response to the determination of Yes in step S208 of FIG. First, in step S1001, moving image shooting is interrupted, and the CPU 26 starts this processing.

  Next, the CPU 26 proceeds to step S1002, and starts counting the number N1 of blackout images set in advance as corresponding to a period during which the moving image shooting operation is interrupted to capture a still image (initial value i = 1). .

  In step S1003, the CPU 26 determines whether it is the first frame of the GOP in order to insert the blackout image as the first substitute frame of the moving image frame.

  If the substitute frame is the first frame of the GOP (YES in step S1003), the blackout image I frame encoded by the image codec unit 18 is output in step S1004.

  If the substitute frame is not the first frame of the GOP (NO in step S1003), the blackout image P frame encoded by the image codec unit 18 outputs an inter macroblock (inter MB) (step S1005).

  The inter MB in this process is an image in which all macroblocks are intra macroblocks that are not subjected to predictive coding based on other frames in a P frame compressed between frames.

  Normally, difference information between frames is stored in the P frame. Also, the difference between the current frame and the previous frame is calculated in units of macroblocks, and the difference information is used as an inter macroblock (inter MB). If the difference information is 0, it is possible to replace the inter macro block with the skipped macro block as in the embodiment described above. However, in the processing of the present embodiment, each macroblock of a P frame encoded in advance is assumed to be a macroblock that is not forward-referenced, that is, an intra macroblock calculated within the frame. Since this inter MB is not forward-referenced, the blackout image P frame can be prepared in advance.

  The blackout image I frame or the blackout image P frame may be created by the CPU 26 in advance.

  The blackout image 27b is read from the ROM 27 and inserted as a first substitute frame on the moving image file (AVI data).

  FIG. 11 is a diagram showing the structure of a moving image file when a still image is shot during moving image recording.

In FIG. 1 to No. Up to 3 are images before the video is interrupted.
Frame No. after video interruption 4 to No. 6 is a blackout image and is composed of inter-macroblocks (inter-MB) of P frames.

  Frame No. 4 is generated by the image codec unit 18 in a blackout image P frame inter MB.

  The length (period) of the first alternative frame is equal to the longest exposure time in the camera green mode (AUTO mode), and is about 1/8 second (125 mSec). When a moving image is captured at 30 frames / second, the number of frames in 1/8 second is “4”. Therefore, the blackout image in the ROM 27 (or the blackout image generated by the image codec unit 18) is inserted into the AVI file four times.

  Instead of the blackout image, an image having a message such as “still image shooting” may be used.

  If N1 blackout images have not been reached in step S1006 (NO in step S1006), i is incremented in step S1007, and the process returns to step S1003. If the number of blackout images reaches N1 in step S1006 (YES in step S1006), the process proceeds to step S1008.

  In step S1008, the pseudo shutter sound data 27c placed on the ROM 27 is read out and inserted into the sound data of the moving image file (AVI data). At the same time, the pseudo shutter sound data 27 c is output to the sound processing unit 28 and reproduced from the speaker 32. The time length of the pseudo shutter sound data is substantially the same as the time length of the previous blackout image data.

  At this time, if the pseudo data is suddenly replaced with the sound data recorded from the microphone 30, a high-frequency component is generated due to a sudden switching of data, and noise sound data is generated due to aliasing. Therefore, during the period of replacement with pseudo data (which is also the period during which the pseudo shutter sound is reproduced by the speaker 32), the sound processing unit 28 first mutes (silences) the period. A pseudo shutter sound is inserted on the muted audio data. As a result, high-quality pseudo shutter sound can be inserted into the audio data without generating high-frequency noise due to aliasing.

  In step S1009, the process waits for the completion of the still image development process in the above-described still image shooting operation. When development is completed, the process proceeds to step S1010.

  After that, the CPU 26 JPEG-encodes the developed still image by the image codec unit 18 and generates one still image file from the encoded data as the processing in step S210 in FIG. The generated still image file is stored in the recording medium 21 separately from the moving image file.

  In parallel with this processing or thereafter, in this step S1010, the CPU 26 generates a VGA size review image from the developed still image by the resizing unit 24, and starts counting the number N2 of stop motion images. (Count initial value j = 1).

  Next, it is determined whether it is the first frame of the GOP. If it is the first frame (YES in step S1011), the review image data is encoded as a stop motion image I frame by the image codec unit 18 and output (step S1012).

  If it is not the first frame of the GOP (NO in step S1011), the review image data is encoded and output as a stop motion image P frame inter MB by the image codec unit 18 (step S1013).

  If j has not reached N2 (NO in step S1014), j is incremented in step S1015, and the process returns to step S1011. If j reaches N2 (YES in step S1014), the process proceeds to step S212 (step S1016).

  As described above, according to the present embodiment, an encouraging P frame of a moving image file generated by taking a still image during moving image shooting is an inter MB image. This eliminates the need for a reference image, thereby reducing the load on the image codec unit 18.

  Although described as I frame and P frame, this embodiment can also be applied to interlaced images. In this embodiment, the MPEG system is used as the format of the moving image data. However, other formats such as H264 can be similarly applied as long as they perform inter-frame compression.

It is a figure which shows the block diagram of the digital camera in embodiment of this invention. It is a flowchart which shows the process sequence at the time of image | photographing a still image during the moving image recording in embodiment of this invention. It is a flowchart which shows the production | generation procedure of the moving image file at the time of performing the still image shooting operation | movement in the 1st Embodiment of this invention. It is a figure which shows the sequence at the time of image | photographing a still image during the moving image recording in embodiment of this invention. It is a block diagram of the moving image file at the time of taking the still image during the moving image recording in the first embodiment of the present invention. It is a flowchart which shows the production | generation procedure of the moving image file at the time of performing the still image shooting operation | movement in the 2nd Embodiment of this invention. It is a block diagram of the moving image file at the time of the still image photography in the moving image recording in the 2nd Embodiment of this invention. It is a flowchart which shows the production | generation process sequence of the moving image file at the time of performing the still image shooting operation | movement in the 3rd Embodiment of this invention. It is a block diagram of the moving image file at the time of the still image photography during the moving image recording in the 3rd Embodiment of this invention. It is a flowchart which shows the production | generation process sequence of the moving image file at the time of performing the still image shooting operation | movement in the 4th Embodiment of this invention. It is a block diagram of the moving image file at the time of the still image imaging | photography during moving image recording in the 4th Embodiment of this invention.

Explanation of symbols

8 Image Codec Unit 10 Image Sensor 16 Signal Processing Unit 21 Recording Media 23 Operation Unit 26 CPU
27 ROM

Claims (9)

An imaging device that records image data captured by an imaging unit on a recording medium,
A moving image shooting control unit for controlling a moving image shooting operation for recording a moving image obtained by continuously capturing images with the imaging unit as a moving image file on the recording medium;
Still image shooting control means for controlling a still image shooting operation for recording a still image obtained by taking an image with the imaging means as a still image file on the recording medium;
When an instruction for a still image shooting operation is input during a period in which the moving image shooting operation is being performed, the still image shooting control unit controls the still image shooting operation and records the still image file on the recording medium. The moving image shooting control unit interrupts the moving image shooting operation , inserts a blackout image as the head of an image that replaces the moving image corresponding to the interrupted period, and the difference information is zero after the blackout image. Inserting a compressed image between frames is repeated, and when the number of inserted images is equal to or greater than a predetermined number and the inserted image is the last frame of a GOP, the I that is the head of the next GOP The frame is a stop motion image obtained from a still image corresponding to the still image shooting operation, and is compressed between frames whose difference information is zero after the I frame. Imaging apparatus characterized by recording on the recording medium to generate a moving image file by repeating the insertion of the image to the number of frames corresponding to the GOP.   The imaging apparatus according to claim 1, wherein the inter-frame compressed image in which the difference information is zero is a skipped macroblock image. Period of the interruption, of the still image capturing operation, reads the image data for a still image from at least the imaging unit, according to claim 1 or 2, characterized in that it comprises a period until the development process is completed Imaging device. Display means for displaying a photographed image, and means for generating a review image for display on the display means based on image data obtained by photographing the still image after capturing the still image includes, as a stop-motion image in the substitute image period of the interruption, the imaging device according to any one of claims 1 to 3, characterized by using the review image. Further comprising a voice input means, the moving image capturing operation, together with the audio information obtained by the voice input unit, any one of claims 1, characterized in that it comprises an operation for recording a moving image on the recording medium 4 The imaging device described in 1. It said generating means, said blackout image and difference information subsequent thereto is within the period in which the image has been compressed between the zero frame is inserted, characterized that you insert the audio information of a pseudo shutter sound The imaging device according to claim 5 . The period for inserting the audio information before Symbol pseudo shutter sound, the stop voice input means, the image pickup apparatus according to claim 6, characterized in that the silence detection status. The video recording control means, and with the stop-motion image, still image data obtained by imaging, claim 4, characterized in that the recording resize to an image size to be recorded as a moving image The imaging apparatus according to any one of 7 . A moving image shooting operation for recording a moving image obtained by continuous imaging by an imaging unit on a recording medium as a moving image file, and a still image obtained by imaging by the imaging unit as a still image file on the recording medium A method of controlling an imaging apparatus having a function of executing a still image shooting operation to be recorded,
If an instruction for a still image shooting operation is input during a period in which the moving image shooting operation is being performed, the moving image shooting operation is interrupted, and the still image file is controlled to store the still image file on the recording medium. Recording, inserting a blackout image as the head of an image that replaces the moving image corresponding to the interrupted period, and repeatedly inserting an inter-frame compressed image whose difference information is zero after the blackout image If the number of inserted images is equal to or greater than a predetermined number and the inserted image is the last frame of a GOP, the I frame that is the head of the next GOP is extracted from the still image corresponding to the still image shooting operation. A frame corresponding to the GOP that the obtained stop motion image is inserted and an inter-frame compressed image whose difference information is zero is inserted after the I frame. Control method for an imaging apparatus characterized by generating a moving image file recorded on the recording medium by repeating up.

Images