JP2008283276A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus performing high-speed photographing and normal photographing without requiring large-capacity recording media. <P>SOLUTION: The imaging apparatus includes: a camera section 21 which outputs a video signal; a video encoding section 23 which encodes the video signal by a first encoding system and outputs the encoded video signal as main-content encoded data when the video signal has a normal frame rate, and encodes one predetermined frame output from the camera section 21 in a period corresponding to one frame of the normal frame rate by the first encoding system and outputs the encoded frame as the main-content encoded data, and also encodes the remaining frames by a second encoding system and outputs the encoded remaining frames as difference encoded data when the video signal has a fast frame rate; a recording processing section 36 which records the main-content encoded data as a main-content file and the difference encoded data as a difference file in a recording medium 9; and an imaging frame rate control section 31 which switches the frame rate of the camera section 21 to the fast frame rate on receiving an instruction for high-speed imaging. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image pickup apparatus that records data obtained by compressing and encoding video on a recording medium, and that decodes and reproduces the recorded compressed encoded data from the recording medium.

  2. Description of the Related Art In recent years, imaging devices that record video and audio data as files on a randomly accessible recording medium such as an optical disk or a semiconductor memory have become common. An image pickup apparatus provided with such a recording medium is used, for example, in a broadcasting station or a video production company to perform video editing work. In this field of video production, as one of the special effects of video, for example, a short time event such as the moment when a water drop lands or the action of an insect wing is imaged at high speed and recorded on a recording medium. High-speed imaging devices that realize high-definition slow-down by playing are used.

  A normal imaging device captures images (for example, 30 frames per second) at the number of frames per second (hereinafter, the number of frames per second is abbreviated as “frame rate”) determined by the video standard. However, the high-speed imaging device drives the imaging device at a high speed, captures an image at a frame rate (for example, 120 frames per second) higher than that defined in the video standard, and records video data on a recording medium. By reproducing the recorded video data from this recording medium at a normal frame rate (for example, 30 frames per second), high-definition slow reproduction is realized.

  On the other hand, recording of a high-speed imaging device has a very large recording capacity per recording time compared to normal recording, and video data is generally recorded by compression encoding due to the limitation of the recording capacity of the recording medium. Is. If the recording size per unit recording time is small, the operation cost of the recording medium can be drastically reduced. That is, it is important to perform efficient encoding and recording.

  In addition, high-speed imaging devices are generally expensive and are often dedicated devices that are different from ordinary imaging devices. When high-speed imaging and normal imaging are performed simultaneously, two imaging devices are used. Or, it is necessary to perform an operation such as extracting only necessary frames from video data captured at a high speed by editing operations, which takes time and cost.

In order to solve these problems, a technique has been proposed that includes a plurality of light receiving elements, a plurality of encoding units, and a plurality of decoding units, and records and reproduces a plurality of compression-encoded video data (for example, Patent Document 1). In this conventional imaging apparatus, for example, two light receiving elements that capture images at a normal frame rate capture images while shifting the time axis for capturing images for a period of a normal half frame between the light receiving elements. A plurality of video data captured by this method is compressed and encoded, two video data (normal moving image data and supplemental moving image data) are recorded, the two compressed video data are decoded, and alternately at a normal frame rate. High-definition slow motion can be realized by playing.
JP 2006-197336 A

  However, the conventional imaging apparatus employs the inter-frame predictive coding method as the compression coding. However, since the two encoders perform the coding independently without interacting with each other, the compression efficiency is high. Unfortunately, the recording size per recording time is enormous. This is because the time interval for encoding by each encoder is a time interval of a normal frame rate unit, and the frame rate when two video data are alternately output (for example, twice the normal frame rate). This is because the time interval is long and the advantage of encoding using inter-frame predictive encoding is small, and images close in time are encoded separately by two encoders. That is, when recording a video signal captured at high speed, a large-capacity recording medium is required, and the operating cost of the high-speed imaging apparatus and production cannot be reduced.

  An object of the present invention is to provide an imaging apparatus that can perform high-speed imaging and normal imaging without requiring a large-capacity recording medium.

  In order to solve the above problems, an imaging apparatus according to the present invention includes a camera unit capable of switching between a normal frame rate and a high-speed frame rate higher than the normal frame rate, and a frame rate of the camera unit at the normal frame rate. In a state, the video signal output from the camera unit is encoded by the first encoding method and output as main encoded data. In the state of the high-speed frame rate, one frame of the normal frame rate is output. The predetermined one frame out of the frames output from the camera unit during a period corresponding to the above is encoded by the first encoding method and output as the main encoded data, and the remaining frames are encoded by the second encoding method. A video encoding unit that encodes and outputs as differential encoded data, and the main encoded data as a main file, A recording processing unit that records the difference encoded data as a difference file on a recording medium, a management file processing unit that stores a management file that records the relationship between the main file and the difference file, and an external An input unit that receives a high-speed imaging instruction or a normal imaging instruction, and when the input unit receives a high-speed imaging instruction, the frame rate of the camera unit is set to the high-speed frame rate, and the input unit receives a normal imaging instruction And an imaging frame rate control unit that sets the frame rate of the camera unit to the normal frame rate.

  According to the present invention, a video signal is encoded by a single video encoding unit, and is divided into a main file and a difference file and recorded, thereby making it possible to efficiently reference difference data (remaining frames) with other frames. By encoding with a simple compression encoding and outputting as differential encoded data, it is possible to provide an imaging apparatus capable of performing high-speed imaging and normal imaging without requiring a large-capacity recording medium.

  The image pickup apparatus of the present invention can take the following various modes based on the above configuration.

  That is, in the imaging apparatus having the above configuration, the first encoding method can be configured not to refer to the differentially encoded data.

  The second encoding scheme refers to other differentially encoded data, or refers to the main encoded data that is imaged at the high-speed frame rate and encoded according to the first encoding scheme. Can be.

  In addition, the first encoding scheme may be an intra-frame prediction encoding scheme, and the second encoding scheme may be a bidirectional prediction encoding scheme or a forward prediction encoding scheme.

  In addition, a multiplexing unit that multiplexes the main part encoded data and the audio data may be further provided, and the multiplexed main part encoded data may be recorded on the recording medium as a main part file.

  A playback processing unit is further provided, and the playback processing unit refers to the management file recorded on the recording medium, and uses a ratio between the high-speed frame rate and the normal frame rate to transmit one frame of video data in the main file. The number of frames of the difference file to be inserted into the frame is calculated, and one frame is reproduced from the main file and the number of frames to be inserted from the difference file is within the section where the high-speed frame rate is imaged during the reproduction of the video. It is possible to have a configuration in which a high-definition slow video is played back by playing back and playing back one frame from the main file again and playing back at a normal frame rate.

  The playback processing unit sweeps and interpolates audio for one frame of video data on a time axis during playback of a high-definition slow video, and performs video data 1 frame of the main file and video data 1 of the main file. The frame of the video data of the difference file inserted between the frames can be extended to the playback time and the slow playback can be performed.

  The management file may include at least the normal frame rate, the high-speed frame rate, and a frame length from the start of recording to the start of high-speed frame rate imaging.

  Hereinafter, embodiments of an imaging apparatus according to the present invention will be described with reference to the drawings.

(Embodiment)
The imaging apparatus according to the present embodiment can use two imaging methods, imaging at a normal frame rate and imaging at a high frame rate.

  FIG. 1 is a block diagram of an imaging apparatus according to an embodiment of the present invention. The imaging apparatus 1 according to the present embodiment includes an imaging / encoding unit 2, a system control unit 3, a playback unit 4, a command input unit 6, a status display unit 7, a recording memory interface unit (hereinafter referred to as IF). 8, which are connected to the I / O bus 5. Further, a recording memory (recording medium) 9 is detachably connected to the IF unit 8.

  The imaging / encoding unit 2 includes a camera unit 21, a video processing unit 22, a video encoding unit 23, an audio input unit 24, an audio processing unit 25, and a multiplexing unit 26, and images a subject. Then, the compressed and encoded multiplexed data is generated. The camera unit 21 captures an image captured by a CCD, which is an image sensor, as a video signal. The video processing unit 22 performs preprocessing (such as noise removal) on the video signal, and converts the video signal into digital video data. The video data is the difference between the data captured at the normal frame rate or the main data that is the predetermined data captured at the normal frame rate at the time of high-speed frame rate imaging and the data other than the main data at the time of high-speed frame rate imaging. It consists of data.

  The video encoding unit 23 generates main encoded data and differential encoded data (hereinafter collectively referred to as encoded data) obtained by compression encoding the main data and the difference data. The video encoding unit 23 switches between a plurality of types of compression encoding methods. The voice input unit 24 is composed of a microphone or the like, and converts voice into a voice signal that is an electrical signal. The sound processing unit 25 performs preprocessing on the sound signal and converts the sound signal into digital sound data. The multiplexing unit 26 generates multiplexed data in which the encoded data converted by the video encoding unit 23 and the audio data converted by the audio processing unit 25 are multiplexed.

  The system control unit 3 includes an imaging frame rate control unit 31, an encoding method control unit 32, an encoded data classification unit 33, a management file processing unit 34, a reproduction processing unit 35, and a recording processing unit 36. Then, management and control of each unit and the recording memory 9 are performed. The system control unit 3 can also be realized by a microcomputer and a memory. In this case, each unit of the system control unit 3 is realized by operating various programs stored in the memory.

  The imaging frame rate control unit 31 switches the imaging frame rate of the camera unit 21. The encoding method control unit 32 switches the encoding method of the video encoding unit 23. The encoded data classification unit 33 classifies whether the video data input to the video encoding unit 23 is main data or differential data. The management file processing unit 34 performs processing related to a management file (for example, metadata) in which data for managing each file recorded in the recording memory 9 is recorded. The reproduction processing unit 35 performs processing for reproducing the main file and the difference file recorded in the recording memory 9. The recording processing unit 36 generates a main file and a difference file from the multiplexed data output from the multiplexing unit 26 and records them in the recording memory 9 in a predetermined file format.

  The I / O bus 5 connects each part and transmits various data. The command input unit 6 receives an instruction from the user (starting high-speed imaging, displaying video, etc.). The state display unit 7 presents the operating state of the device to the user. The IF unit 8 is an interface for connecting the recording memory 9 to the I / O bus 5. The recording memory 9 is a PC card, for example, and is detachable from the IF unit 8. Since the recording memory 9 is detachable from the IF unit 8, it can be connected to another editing apparatus.

  The reproduction unit 4 includes a demultiplexing unit 41, a video decoding unit 42, a video output unit 43, and an audio decoding unit 44, and reproduces multiplexed data. The demultiplexer 41 separates the multiplexed data (stream) multiplexed by the multiplexer 26 or the multiplexed data recorded in the recording memory 9 into encoded data and audio data. The video decoding unit 42 decodes the encoded data and restores the original video data. The video output unit 43 converts the video data into a video signal and inputs the video signal to the video display unit 10. The video display unit 10 is a monitor provided in the video device 1, a monitor connected to the outside, or the like, and displays a video signal as a video. The audio decoding unit 44 converts the audio data separated by the demultiplexing unit 41 into an audio signal. The audio output unit 11 is a speaker, for example, and outputs an audio signal as audio.

  Next, the normal imaging operation of the imaging device 1 will be described. When the user instructs the start of normal recording through the command input unit 6, the camera unit 21 images the subject and inputs a video signal to the imaging processing unit 22. Next, the video processing unit 22 performs preprocessing such as noise removal and band limitation using a filter on the video signal, generates video data obtained by converting the preprocessed video signal into a digital format, The data is input to the encoding unit 23.

  The video encoding unit 23 converts the video data into, for example, the MPEG-2 system or MPEG-4 AVC / H. Compression encoding is performed based on the H.264 standard. The same effect can be achieved with a compression coding method to which intra-frame predictive coding is applied, but in this embodiment, MPEG-4 AVC / H. An example in which compression coding is performed using the H.264 system (hereinafter abbreviated as H.264) will be described.

  FIG. 2 is a block diagram showing a configuration of the video encoding unit 23. With this configuration, video data input from the video processing unit 22 is H.264. It is encoded in the H.264 format. Although details of the video encoding unit 23 will be described later, an intra-frame prediction processing unit 56, an inter-frame prediction processing 57, and a switch 59 for switching between them are included. The compression encoding method for the video data is appropriately selected by operating the switch 59 based on a command from the encoding method control unit 32. During normal imaging, the switch 59 is connected to the intra-frame prediction processing unit 56.

  The encoded data compressed and encoded by the video encoding unit 23 is supplied to the recording processing unit 36 of the system control unit 3 via the I / O bus 5. The recording processing unit 36 records the multiplexed data in the recording memory 9 as a main file through the IF unit 8.

  Next, the high-speed imaging operation of the imaging device 1 will be described. When the user instructs the start of high-speed imaging from the command input unit 6 during normal imaging, the imaging frame rate control unit 31 increases the imaging frame rate of the imaging device of the camera unit 21 (for example, changes from 30 frames per second to 120 frames per second). )

  FIG. 3 is a conceptual diagram showing the video signal output to the camera unit 21 and the audio signal output from the audio input unit 24. The horizontal axis indicates the imaging time. In the drawing, 1 to 7 and 31 to 33, 41 to 43, 51 to 53 indicate one frame of the video signal, and A1 to A7 indicate an audio signal in units of one frame. In the high-speed imaging section, the imaging time of one frame of video signal is ¼ of the imaging time of one frame in the normal imaging section.

  The camera unit 21 outputs a video signal at the same frame rate as the imaging frame rate. Next, this video signal is subjected to processing similar to that during normal imaging in the video processing unit 22 and input to the video encoding unit 23.

  The encoded data classification unit 33 is a predetermined one of the frames output from the camera unit 21 during a period corresponding to one frame of the normal frame rate with respect to the frame of the video signal input to the video encoding unit 23. Frames are set as main data, and are classified as difference data when the remaining frames. Based on this classification, the encoding method control unit 32 does not refer to the difference data for the main data, for example, the intra-frame encoding method, and the difference data is not limited to the encoding method. For example, compression encoding is performed by a bidirectional predictive encoding method that refers to the main data.

  That is, in FIG. 2, the switch 59 is connected to the intra-frame prediction processing unit 56 for input of the main data, and the inter-frame prediction processing unit 57 is connected for input of difference data. Details of the processes of the encoding method control unit 32 and the encoded data classification unit 33 will be described later.

  The video encoding unit 23 inputs the main program encoded data obtained by encoding the main program data using the intra-frame encoding method and the differential encoded data obtained by encoding the difference data using the bidirectional predictive encoding method. To do. The multiplexing unit 26 supplies multiplexed data obtained by multiplexing audio data and the like to the main program encoded data to the recording processing unit 36 via the I / O bus 5 in the same manner as in normal recording. Note that differentially encoded data is not multiplexed with audio data. The recording processing unit 36 records the main encoded data from the multiplexed data as a main file and the differential encoded data as a difference file in the recording memory 9 via the IF unit 8. A group of files such as a main file and a difference file created by a series of recordings is called a clip. The clip includes at least a data file obtained by compressing and encoding a video, and may include, for example, an audio file and a file that records metadata indicating an imaging location and time.

  In addition, the management file processing unit 34 records the relevance between the main file and the difference file and additional information in the recording memory 9 as a management file. A specific description of this management file will be described later.

  FIG. 4 is a conceptual diagram showing the configuration of the main file and the difference file. In the figure, 1, 2, 3,... Represent one frame of the main encoded data, and 31, 32,. A1, A2,... Indicate audio data.

  Next, the voice input operation of the imaging apparatus 1 will be described. When the user instructs the command input unit 6 to start imaging, the camera unit 21 captures video and the audio input unit 24 starts capturing audio. The voice captured from the voice input unit 24 is input to the voice processing unit 25 as a voice signal. The audio processing unit 25 converts the input audio signal into audio data in a digital format and inputs the audio data to the multiplexing unit 26. As shown in FIG. 4, the multiplexing unit 26 multiplexes the input audio data and the main program encoded data to generate multiplexed data, and supplies the multiplexed data to the recording processing unit 36. The recording processing unit 36 records the main file as a main file in the recording memory 9.

  As described above, the audio captured by the audio input unit 24 is recorded in the recording memory 9 as the main file in parallel with the video capture. This series of recording processing for audio is the same processing in the high-speed imaging section for video.

  In this embodiment, audio compression is not performed. However, if audio is compressed in a form in which the main file is multiplexed and recorded in the recording memory 9 or audio is in a separate file, Similar effects can be obtained.

  Next, the normal reproduction operation of the imaging apparatus 1 will be described. When the user instructs the command input unit 6 to start reproduction, the management file processing unit 34 reads information indicating the relationship between the main file and the difference file from the management file recorded in the recording memory 9. The reproduction processing unit 35 reads the main file designated for reproduction from the information of the management file in the normal imaging section, and inputs the multiplexed data to the demultiplexing unit 41. The demultiplexing unit 41 separates the multiplexed data from the encoded data and the audio data, and inputs the encoded data to the video decoding unit 42 and the audio data to the audio decoding unit 44.

  The video decoding unit 42 The video decoding process conforming to the H.264 standard is performed, and the decoded video data is input to the video output unit 43. The video output unit 43 converts the input video data into a video signal, and the video display unit 10 presents a reproduced image to the user. The voice decoding unit 44 converts the voice data into a voice signal and inputs the voice signal to the voice output unit 11. The audio output unit 11 outputs an audio signal as audio.

  FIG. 5 is a conceptual diagram showing the video signal output from the video output unit 43 during normal playback and the audio signal output from the audio decoding processing unit 44 in frame units using the main file shown in FIG. is there. 1 to 7 and 1A to 7A shown in FIG. 5 correspond to the frame numbers shown in FIG. Since playback is performed using only the main file, even during a high-speed imaged section, playback is performed in the same manner as a normal imaged section, and the operability and searchability of the user such as improved search efficiency during editing are improved.

  Next, a high-definition slow playback operation in a section recorded by high-speed imaging will be described. When the management file processing unit 34 reads the management file at the start of playback, the management file processing unit 34 holds the time (for example, the frame length) between the start time of the first frame of the main file and the start time of the first frame of the difference file. Then, the reproduction processing unit 35 is notified. The playback processing unit 35 reads the difference file when the time (frame length) from the first frame of the file to the current playback frame coincides with the held time (frame length) in the playback of the main file. To start. Further, the reproduction processing unit 35 processes the differentially encoded data of the difference file in the order of the demultiplexing unit 41, the video decoding unit 42, and the video output unit 43, as in the normal reproduction. Details of the reproduction processing of the rational file processing unit 34 and the reproduction processing unit 35 will be described later.

  The audio data input to the audio decoding unit 44 is converted into an analog audio signal by the audio decoding unit 44 and input to the audio output unit 11. The audio output unit 11 outputs an audio signal as audio. In addition, during the high-definition slow playback of the video, the playback processing unit 35 performs an audio slowdown based on the number of frames of the differentially encoded data inserted between 1 frame of the main encoded data in the section recorded by high-speed imaging. Perform playback. That is, the audio data is swept on the time axis, for example, audio data interpolation is performed so that the audio data is also played back at the same speed as the video output.

  FIG. 6 shows a video signal output from the video output unit 43 and an audio decoding unit when high-definition slow playback is performed on the section recorded at high speed in the state where the main file and the difference file shown in FIG. 4 are recorded. FIG. 4 is a conceptual diagram showing audio signals output from 44 in units of frames. 1 to 4, 31 to 33 and 41 and A1 to A4 shown in FIG. 6 correspond to the frame numbers of the encoded data and audio data shown in FIG. 4, respectively. Between the video signals corresponding to the main data (between frames 3 and 4 in the figure), the video signals corresponding to the difference data (frames 31, 32, 33) of 3 frames are inserted. By reproducing at a normal frame rate (30 frames per second), high-definition slow reproduction at 1/4 times speed is achieved in the high-speed imaging section. At this time, the audio decoding unit 44 inputs an audio signal to the audio output unit 11 so that the audio is 1/4 times faster.

  By executing such a series of playback operations, the compressed video data and audio data recorded in the recording memory 9 as two files of the main file and the difference file are read out and played back as one video from the video display unit 10. A sound is output from the sound output unit 11 as an image.

  Hereinafter, the processing of each block will be specifically described.

  FIG. 7 is a table showing examples of recording file names, reference relationships, and picture types of the main encoded data and the differential encoded data. Since reproduction may be performed only with the main part file (main part encoded data), the main part encoded data does not refer to the difference encoded data. On the other hand, when the differentially encoded data is reproduced, the main encoded data is also reproduced, so that the reference relationship is not limited. Any compression method (picture type) may be used as long as it has such a reference relationship. That is, other differentially encoded data may be referred to, or the main encoded data may be referred to. In the present embodiment, a case will be described below where the main encoded data is an I picture and the differential encoded data is a B picture.

  FIG. 8 is a flowchart showing the flow of the encoding method switching process of the encoding method control unit 32. Each time data for one frame is supplied to the video encoding unit 23, a switching process of the encoding method is executed. That is, at the time of high-speed imaging, it is executed at the same cycle as the high-speed imaging frame rate. The encoded data classification unit 33 classifies whether the video data is main data or difference data (step S101). Next, the encoding method control unit 32 determines whether or not the video data is the main encoded data (step S102). When the video data is the main data, the encoding method control unit 32 determines an encoding method based on the restriction of the main data (step S103). In step S102, if the video data is not the main data, the encoding method is determined based on the restriction on the differentially encoded data (step S104).

  The encoding method of the video encoding unit 23 is switched according to the determined encoding method (step S105). That is, in the example of FIG. 7, when the video data is the main encoded data, the video data is converted into a picture (I picture) encoded only by intra-frame encoding, and is differential encoded data. Is converted into a bi-predictive coded picture (B picture). This has the advantage that the I picture is easy to implement because it does not refer to other pictures. Also, in a system that presupposes editing in the production field, etc., only intra-frame coding is edited. This is because it has the advantage of being easy. FIG. 9 shows encoded data when encoding is performed based on the constraints shown in the reference relationship of FIG. FIG. 9A shows encoded data encoded by the encoding method at the time of normal imaging, and FIG. 9B shows encoded data encoded by the encoding method at the time of high-speed imaging ( (Main encoded data and differential encoded data).

  Next, the encoded data classification unit 33 and the encoding method control unit 32 will be described in further detail. It should be noted that compression efficiency is better when more bidirectional prediction coding (B picture) and forward prediction coding (P picture) are employed. However, as long as the constraints shown in the reference relationship of FIG. The same effect can be obtained even if such an encoding method is employed.

  First, a method for classifying video data into main data and difference data by the encoded data classification unit 33 (step S101 in FIG. 8) will be specifically described.

  As shown in FIG. 4, the encoded data classification unit 33 classifies frames corresponding to the normal frame rate from the first frame from which high-speed imaging has started, into main data, and classifies the other frames into difference data. Classification is performed using the following formula.

A = (Cn) mod (Ff / Fn) (Formula 1)
Here, Fn in Equation 1 is the normal frame rate, Ff is the high-speed imaging frame rate, mod is the remainder, and Cn is the number of frames from the high-speed imaging start frame (counting at the high-speed imaging frame rate). . In Formula 1, when A is 1 frame, it is classified as main data, and when A is other than 1, it is classified as difference data.

  If the classification is performed according to the ratio of the normal imaging frame rate and the high-speed imaging frame rate, the same effect can be obtained even if the value of A is classified as 2 or other. When time information is added to the encoded frame and recorded, the time information given to the encoded frame is classified as main data when the normal imaging frame rate is a multiple of Fn, and other In this case, a method of classifying the difference data may be used.

  Next, the encoding method control unit 32 determines the encoding method according to the classification determined by the encoded data classification unit 33. In this embodiment, since the encoding method is determined according to FIG. 7, the video encoding unit 23 encodes the frame classified into the main data into an I picture and the frame classified into the difference data into a B picture. .

  When a more complicated encoding method is controlled, switching may be performed according to the value A.

  Next, the encoding method of the video encoding unit 23 will be described in detail.

  First, when encoding is started by a normal imaging instruction from the command input unit 6, the video data input from the video processing unit 22 is input to the DCT / quantization unit 51 as shown in FIG. 2. The DCT / quantization unit 51 performs discrete cosine transform (DCT) and quantization processing on the video data. The entropy encoding unit 52 performs variable length encoding or arithmetic encoding on the video data output from the DCT / quantization unit 51 and outputs encoded data as a bit stream. At the same time, the inverse quantization / IDCT unit 53 performs inverse quantization processing and inverse DCT (IDCT) on the video data output from the DCT / quantization unit 51, and inputs the converted video data to the filter unit 54 To do.

  The filter unit 54 performs deblocking filtering on the video data, and then stores the video data in the frame memory 55. The intra-frame prediction processing unit 56 applies H.264 to the video data stored in the frame memory 55. Prediction is performed within a frame that is also one of the H.264 features. The predicted information is used in the DCT / quantization unit 51 when DCT / quantization of video data of the next frame.

  On the other hand, the video data input from the video processing unit 22 is also input to the motion vector detection unit 58. The motion vector detection unit 58 detects horizontal and vertical motion vectors from the reference frame stored in the frame memory 55. The inter-frame prediction processing unit 57 performs block division processing, motion compensation processing, and motion vector encoding using variable MC (Motion Compensation). The changeover switch 59 selects whether to apply the intraframe predictive coding method or the interframe predictive coding for each frame, and inputs the output data to the DCT / quantization unit 51. The operation of the changeover switch 59 is controlled by the encoding method control unit 32 as described above.

  Furthermore, the motion vector detection unit 58 first divides one frame of video data to be encoded into rectangular areas of a predetermined size (for example, 16 × 16 pixels, 8 × 8 pixels, etc.), and the frame memory 55 An appropriate block is searched as a predicted value of each block from the reference frame stored therein. Within the search range set in advance in the reference frame, the position of the block is moved little by little to determine whether or not the block at the position is appropriate as the predicted value. As a criterion for determining whether or not it is appropriate, a sum of squares of differences of pixel values in a block, a sum of absolute differences, or the like is used. In this manner, for each block of one frame of video data to be encoded, a shift in the position of the block searched as a predicted value of the block in the reference screen is detected as a motion vector.

  As is conventionally known, H.P. In the H.264 compression method, the concept of MPEG-2 GOP (Group Of Pictures) structure does not exist, but in order to facilitate the implementation, a method that is realized by adopting the same configuration as the MPEG-2 equivalent GOP structure is generally used. Used. For example, when applying the GOP structure in which 1 GOP is 15 frames and the encoded picture is BBIBBPBBPBBPBBP, the I switch and the B / P picture are periodically switched by the changeover switch 59 and stored in the frame memory 55. By controlling the reference frame to be referenced among the stored reference frames, the B picture and the P picture are switched to form 1 GOP.

  That is, when the encoding method (picture type) determined by the encoding method control unit 32 is an I picture, the intraframe predictive encoding is enabled by the changeover switch 59. In the case of a B picture, the changeover switch 59 controls inter-frame prediction encoding and controls to refer to both reference frames before and after the currently encoded frame. In the case of a P picture, the changeover switch 59 enables inter-frame predictive encoding, and controls to refer to a reference frame in only one direction from the currently encoded frame.

  In this embodiment, since only the I picture and the B picture are switched, the coding method of the video coding unit 23 is changed to the coding method determined by the coding method control unit 32 by switching the changeover switch 59. Switch.

  When outputting encoded data, according to the classification determined by the encoded data classification unit 33, a classification flag indicating whether the encoded data is main encoded data or differential encoded data is output at the same time as the encoded data. Distinguish the two classifications. The classification flag is input to the multiplexing unit 26, and the multiplexing unit 26 multiplexes the main encoded data and the audio data input from the audio processing unit 25 according to the classification flag to generate multiplexed data. . The multiplexing unit 26 also outputs a similar classification flag indicating that the multiplexed data belongs to the main file and supplies it to the recording processing unit 36. The recording processing unit 36 refers to the classification flag, controls whether to record the main file or the difference file in the recording memory 9, and writes the main file and the difference file to the recording memory 9.

  When the encoded data needs to comply with the alignment of the packet format, for example, the alignment is performed by inserting, for example, invalid data or invalid packets at the switching point when dividing into two files. Can be observed.

  Next, a method for generating metadata by using the management file processing unit 34 as a management file with the relation between the main file and the difference file will be specifically described. FIG. 10 is a conceptual diagram showing clips recorded in the recording memory 9. The management file processing unit 34 records the recorded metadata of the clips 1, 2, and 3 in the recording memory 9 as a management file.

  FIG. 11 is a diagram showing the contents of metadata assigned to a clip. The content shown in FIG. 11 (a) relates to the main encoded data, and the content shown in FIG. 11 (b) relates to differentially encoded data.

  In FIG. 11A, the Clip ID is a unique ID automatically assigned by the device for each clip. Video File Name is the file name of the main file that records data obtained by compression-coding video. Frame Rate indicates the number of frames per second of an image captured during normal recording. Duration indicates the length of the recorded clip and indicates the frame length in terms of the normal frame rate. NumSubVideo indicates the number of difference files recorded by high-speed imaging. SubVideoInfo is information of the difference file defined in FIG. 11B, and is recorded for the value of NumSubVideo.

  In FIG. 11B, Offset is the number of frames in terms of the normal frame rate from the beginning of the clip to the start of the first frame of the video encoded data recorded in the difference file. Sub Video File Name is the file name of the difference file. Sub FrameRate is the encoding frame rate of video encoded data in the difference file (equal to the high-speed imaging frame rate). Sub Duration indicates the time length (frame length) of the encoded data in the difference file, and indicates the frame length in terms of the encoded frame rate (equal to the high-speed imaging frame rate).

  In the present embodiment, in order to improve the convenience of search, the clip metadata is in a text file format as a separate file from the video encoded data (main file and difference file).

  FIG. 12 is a diagram illustrating a change in the content of metadata from the start to the end of recording. FIG. 12A shows an example of metadata content immediately after the start of clip recording. Clip ID is C0001, Video File Name is C0001. At m2ts, FrameRate is 30 frames per second. Clip ID and Video File Name are determined before the start of clip recording. For example, the Clip ID can be a 4-digit increment counter value with “C” added at the beginning, and the Video File Name can be in a format with an extension added to the Clip ID.

  Here, the extension of the video file is m2ts as an m2ts file format conforming to MPEG-2TS (Transport Stream) multiplexing, but other multiplexing schemes and file formats may be used. Also, Duration is set to 0 because it is not fixed immediately after the start of recording. Further, NumSubVideo is set to 0 until high-speed imaging is performed, and when NumSubVideo is 0 after recording of the clip, it indicates that high-speed imaging is not performed in the clip.

  FIG. 12B shows the content of the metadata immediately after starting high-speed imaging for the first time with a clip. 12A, NumSubVideo is set to 1, and corresponding Sub Video Info (Offset: 1234, SubVideoFile: C0001a.m2ts, SubFrameRate: 120, SubDuration: 0) is added. Also, the Duration item is changed to the clip length (1234) at that time.

  FIG. 12C shows the content of metadata immediately after the completion of high-speed imaging of 142 frames in terms of normal frame rate and 568 frames in terms of high-speed imaging frame. Since the time length (frame length) of the difference file is fixed, the SubDuration item is changed to 568. FIG. 12D shows the content of the metadata after the clip recording is completed. In this example, there are two NumSubVideos, which indicate that two high-speed images have been taken for the clip C0001 (two differential files are present).

  Next, a method for reproducing a file recorded in the recording memory 9 by the camera recorder will be specifically described.

  First, based on the metadata shown in FIG. 12 (d), the processing of the management file processing unit 34 and the reproduction processing unit 35 will be specifically described when the main file and the difference file are reproduced in high-definition slow reproduction.

  When a reproduction start instruction is issued by the user from the command input unit 6, the management file processing unit 34 first reads the management file, acquires the contents of the metadata, and holds the information. In this example, it can be seen that high-speed imaging recording is performed from the 1234th frame from the offset of the first SubVideoInfo.

  Since the high-speed imaging / recording is not performed until the 1233th frame, the reproduction processing unit 35 reproduces only the main-encoded data recorded as the main file. From the 1234th frame, since this applies to the section where high-speed imaging is recorded, the read file is switched and reproduced according to the ratio between the high-speed imaging frame rate and the normal frame rate. That is, assuming that the high-speed imaging frame rate is Ff (120 frames per second), the normal frame rate is Fn (30 frames per second), and the quotient is div, the difference file ((Ff div Fn) -1) between one frame of the main file. (3 in the example of FIG. 12) The read file is repeatedly switched and reproduced during the high-speed imaging section so as to insert the frame.

  Even if the encoded data recorded as the main file and the difference file in the recording memory 9 is divided and recorded in a plurality of files, the encoded data before the divided recording is one continuous encoded data. Both the video decoding units 42 need not be aware that the encoded data is divided.

  Next, when the management file processing unit 34 reads the metadata, it can be seen from the first SubVideoInfo SubDuration that the high-speed imaging section is a section of 142 frames in terms of the normal frame rate and 568 frames in terms of the high-speed frame rate. In the case of high-definition slow reproduction, the reproduction processing unit 35 reproduces the high-speed imaging portion at a normal frame rate (for example, 30 frames per second), so that the high-definition slow reproduction described above is repeated for 568 frames. I do.

  Note that the normal frame rate conversion and the high-speed imaging frame rate conversion frame length coexist in the high-speed imaging section. For example, counters for normal frame rate conversion, high-speed frame rate conversion, and current playback position frame are provided. May be controlled.

  If the same processing as the above-described high-definition slow reproduction is executed for the second SubVideoInfo, all data recorded in the recording memory 9 by the high-speed imaging recording of FIG. 12D can be reproduced. .

  Next, normal speed reproduction will be described. Here, normal speed playback refers to a playback method at normal speed that does not perform high-definition slow playback even in a high-speed imaging section. As described above, the main file and the difference file are recorded in the recording memory 9 of the present embodiment, and the main file includes the main program encoded data and all audio data. Further, if only the main encoded data is taken out, it is equivalent to data obtained by performing video encoding at a frame interval according to a normal frame rate and subjected to compression encoding processing. The playback processing unit 35 reads only the main file and supplies it to the demultiplexing unit 41. Thereafter, the playback processing unit 35 performs the same processing as in the high-definition slow playback, thereby playing back the video at normal speed instead of slow playback. Can do.

  This is because the high-definition slow playback process is not executed and only the main file is played back, so that it is possible to easily play back at the normal speed even in a section where high-speed imaging recording is performed. This shows that switching between high-definition slow playback and normal speed playback methods can be easily realized. This feature is effective for improving the convenience of the user, for example, confirmation of the material at the time of imaging, playback on a video playback device not according to the present embodiment, and search for the material at the time of editing.

  As described above, according to the high-speed imaging device of this embodiment, it is possible to realize a camera recorder with high compression efficiency and high convenience, and the operation cost of the entire production workflow including material shooting and editing is reduced. Can be lowered.

  In the present embodiment, the method for recording metadata as a separate file has been described. However, the same effect can be obtained by embedding the metadata in the header portion of the main file. Further, the method of recording in the text format when recording the metadata has been described, but the same effect can be obtained even if it is described in a markup language such as an XML (extensible Markup Language) format, for example. .

  Further, in the present embodiment, bi-predictive encoding is given as an example as an encoding method that does not refer to a frame of differential encoded data from main encoded data, but is not limited to this encoding, for example, A similar effect can be obtained even if forward predictive coding is used.

  The imaging apparatus according to the present invention can switch between normal imaging and high-speed imaging, and can reduce the recording size per recording time, and can be applied to a video recording / reproducing apparatus or a video editing apparatus using a semiconductor memory or an optical disk. Widely applicable.

The block diagram of the imaging device in the embodiment of the present invention The block diagram which shows the structure of the image | video encoding part of an imaging device same as the above. The conceptual diagram which shows the audio | voice signal output from the video signal output from the camera part and the audio | voice input part in the imaging device same as the above. Conceptual diagram showing the structure of the main file and difference file of the same imaging device Conceptual diagram showing the video signal and the audio signal during normal playback in units of frames in the imaging device Conceptual diagram showing the video signal and audio signal in frame units for the high-definition slow playback during the high-speed imaging recording section in the same imaging device Table showing examples of recording file names, reference relationships, and picture types of the main encoded data and differential encoded data in the imaging device Flowchart showing the flow of the encoding method switching process of the encoding method control unit in the imaging device The conceptual diagram which shows the encoding data of an imaging device same as the above as a frame unit The conceptual diagram which shows the clip recorded on the recording memory in an imaging device same as the above The figure which shows the content of the clip metadata provided with respect to a clip in an imaging device same as the above. The figure which shows the change of the content of metadata in an imaging device same as the above from the start of recording

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Imaging / encoding part 3 System control part 4 Playback part 5 I / O bus 6 Command input part 7 Status display part 8 IF part (recording memory interface part)
9 recording memory 10 video display unit 11 audio output unit 21 camera unit 22 video processing unit 23 video encoding unit 24 audio input unit 25 audio processing unit 26 multiplexing unit 31 imaging frame rate control unit 32 encoding method control unit 33 encoding Data classification unit 34 Management file processing unit 35 Playback processing unit 36 Recording processing unit 41 Demultiplexing unit 42 Video decoding unit 43 Video output unit 44 Audio decoding unit 51 DCT / quantization unit 52 Entropy encoding unit 53 Inverse quantization unit / IDCT unit 54 Filter unit 55 Frame memory 56 Intraframe prediction processing unit 57 Interframe prediction processing unit 58 Motion vector detection unit 59 Switch

Claims (8)

A camera unit capable of imaging by switching a normal frame rate and a high-speed frame rate higher than the normal frame rate;
In a state where the frame rate of the camera unit is the normal frame rate, the video signal output from the camera unit is encoded by the first encoding method and output as main encoded data, and at the high frame rate. In a certain state, a predetermined one frame out of frames output from the camera unit in a period corresponding to one frame of the normal frame rate is encoded by the first encoding method and output as main encoded data. A video encoding unit that encodes the remaining frames using the second encoding method and outputs the encoded data as differential encoded data;
A recording processing unit that records the main encoded data as a main file and records the differential encoded data as a differential file on a recording medium;
A management file processing unit for storing a management file in which the relationship between the main file and the difference file is recorded in the recording medium;
An input unit for receiving an instruction for high-speed imaging or an instruction for normal imaging from the outside;
When the input unit receives an instruction for high-speed imaging, the frame rate of the camera unit is set to the high-speed frame rate. When the input unit receives an instruction for normal imaging, the frame rate of the camera unit is set to the normal frame rate. An imaging device comprising an imaging frame rate control unit.   The imaging apparatus according to claim 1, wherein the first encoding method does not refer to the differentially encoded data.   The second encoding method refers to other differentially encoded data, or refers to main-encoded data that has been imaged at the high-speed frame rate and encoded according to the first encoding method. The imaging device described. The first encoding scheme is an intra-frame predictive encoding scheme,
The imaging apparatus according to claim 3, wherein the second encoding method is bidirectional predictive encoding or forward predictive encoding. A multiplexer that multiplexes the main encoded data and the audio data;
The imaging apparatus according to claim 1, wherein the multiplexed main-encoded data is recorded on the recording medium as a main file. It further includes a playback processing unit,
The reproduction processing unit
Referring to the management file recorded on the recording medium, using the ratio between the high-speed frame rate and the normal frame rate, calculate the number of frames of the difference file to be inserted between one frame of video data in the main file,
During playback of the video, within the section where the high-speed frame rate is captured, one frame is played from the main file, the number of frames inserted from the difference file is played, and one frame from the main file is again played. The imaging apparatus according to claim 1, wherein high-definition slow video is reproduced by repeating reproduction and reproduction at a normal frame rate.   The playback processing unit sweeps and interpolates audio for one frame of video data on a time axis during playback of a high-definition slow video, and performs video data 1 frame of the main file and video data 1 of the main file. The image pickup apparatus according to claim 6, wherein the frame of the video data of the difference file inserted between the frames is extended to the playback time and is subjected to slow playback.   9. The management file according to claim 1, wherein the management file includes at least the normal frame rate, the high-speed frame rate, and a frame length from the recording start time to the start of high-speed frame rate imaging. Imaging device.

Images