JP2008125059A - Device for recording/playing back moving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving image recording device and a reproduction device for achieving the recording/playback of a moving image while reducing data quantity. <P>SOLUTION: A moving image recording/playback device records the moving image of a subject. When recording, the moving image recording/playback device sequentially outputs a plurality of image frames that constitute the moving image at a predetermined frame rate, and of the output image frames, records the image frames as a whole at intervals of a predetermined number of frames and partly records each of the other image frames, to at least one recording medium. When playing back and when reading a partly recorded image frame, the moving image recording/playback device generates a display frame from recorded image frames with the whole recorded and the partly recorded image frame. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to high-speed shooting using a high-speed camera, and more particularly to a technique for improving recording performance and convenience during reproduction.

In recent years, the amount of data of moving images has increased in order to satisfy the demand for high image quality. In particular, when performing high-speed shooting, the amount of data to be recorded increases because of the high frame rate.
Accordingly, in consideration of recording on a recording medium, a technique for reducing the data amount of these moving images has been developed.
One of such technologies is compression coding technology such as MPEG (Motion Picture Experts Group). This is a technique for recording data of one screen constituting a moving image smaller.

Furthermore, a technique for reducing the data amount by reducing the size of the screen to be recorded has been developed.
For example, there is a monitoring camera system that can reduce the amount of image storage when an abnormality is detected (see Patent Document 1).
In this technique, an area (see FIG. 16: area A to area D) smaller than the screen is set in advance in a screen (see FIG. 16: screen 20) in which the monitoring area is captured, and movement is performed in that area. If detected, only that area is recorded. Details will be described later with reference to FIG.

According to this technique, the image data of a portion that has moved can be recorded, and the amount of data can be reduced while serving as a surveillance camera.
JP 2005-266373 A

However, in the case of this technique, although the moving image data to be recorded becomes small, the reproduction screen only updates the recorded area, for example, the display area of the area A, and the object etc. that has deviated from this area. Does not appear on the playback screen.
In addition, because of the nature of the surveillance camera, since the surveillance area, that is, the photographing screen is fixed, there is a problem that it is difficult to apply to data reduction of a normal video camera or the like that captures a moving image without fixing the photographing screen. .

  Therefore, an object of the present invention is to provide a moving image recording apparatus and a reproducing apparatus that can record and reproduce moving images while reducing the amount of data.

  In order to solve the above problems, a moving image recording apparatus of the present invention is a moving image recording apparatus that records a moving image of a subject, and an imaging unit that sequentially outputs image frames constituting the moving image at a predetermined frame rate. Recording means for recording a predetermined number of image frames in the image means outputted by the imaging means on the whole image frame, and other image frames recording a part of the image frame on one or a plurality of recording media; It is characterized by providing.

  The term “subject” as used herein refers to an object photographed by a video camera or the like, and refers to a person, an object, or a scene that is a subject of photographing.

Since the moving image recording apparatus according to the present invention has the above-described configuration, when recording an image frame obtained by capturing a moving image, a part of the frame is recorded as a partial area as the image. Thus, the amount of moving image recording data can be reduced.
By recording a part of the image frame, when recording by compression encoding, the time required for the encoding process can be shortened, and the capacity of the frame memory for storing the input image before the encoding process can be reduced. Can be reduced.

Therefore, even in high-speed shooting in which the frame rate of the input image is faster than that of the encoding process, high-speed frame shooting can be realized for a long time.
In this way, even when a captured image is input at a faster frame rate than the encoding process and the storage process, it is possible to realize both high resolution and memory capacity reduction with an inexpensive apparatus. become.

The part of the image frame recorded by the recording unit may be a part including a specific part of the subject.
As a result, the specific part of the subject is included in all the frames since it is included in the recorded part. That is, the specific part is always recorded.
The moving image recording apparatus further includes an encoding unit that encodes an image frame. When the recording unit records a part of the image frame, the moving image recording apparatus other than the part of the image frame output by the imaging unit. An image frame having a predetermined color in the portion may be encoded and recorded by the encoding means.

As a result, the image frame for recording a part can be a single color image frame other than the part, so that it is possible to encode while recording the specific part as compared with the case of encoding the entire captured image frame. The amount of later data can be reduced.
Further, since all the frames have the same size, there is an advantage that they can be played back by a conventional playback device.

  The moving image recording apparatus further includes a generating unit that generates a display frame for display from the image frame recorded on the recording medium, and the generating unit sequentially reads out the image frame and displays the display frame. When the generated and read image frame is only a part, a display frame may be generated from the image frame in which the whole is recorded and a part of the image frame.

As a result, an area other than the part in the image frame in which a part is recorded can be supplemented by another image frame, and can be reproduced without any omission in the display screen.
For example, when a captured moving image is played back, only a specific part appears to be played back at the frame rate at the time of shooting, but other parts other than the specific part are played back at a lower rate. appear. Specifically, when a moving image shot at 180 fps (frame per second) is played back at 180 fps, it is played back at 1 × speed. In this case, the specific portion is smoother than the other portions. Will appear to be played.

Also, for example, when a moving image shot at 180 fps (frame per second) is played back at 30 fps, it is played back at 1/6 times slow speed. In this case as well, the specific part is the other part. It will appear to be played more smoothly.
That is, when playing back a high-frame-rate image taken at high speed, a high-resolution image is obtained from a partial image cut out while maintaining high resolution and a high-resolution image that is recorded regularly before and after the whole. Therefore, even when high-speed captured images are reproduced, it is possible to reproduce all frames without loss of subject information and with high resolution including information around a specific portion. .

The recording unit records a part of the image frame in association with information indicating the position of the part of the image frame on the recording medium, and the generation unit stores the read image frame as a part. May generate a display frame in which the portion is reflected in the position indicated by the information.
As a result, the position of a part of the image frame can be easily obtained, so that the part and the image frame in which the entire part is recorded can be easily combined.

Further, the generation means may further generate a display frame by sequentially reading out only the image frames in which the whole is recorded.
As a result, only the image frames recorded as a whole can be played back, so that playback at a frame rate lower than a predetermined frame rate can be performed.
The moving image reproduction device of the present invention is a moving image reproduction device for reproducing a captured image recorded on one or a plurality of recording media, and the captured image recorded on the recording medium is a predetermined image. It is composed of a plurality of image frames shot at a frame rate, and the size of the first group of image frames is smaller than the size of the second group of image frames. A reading unit that reads out the image frame that constitutes the image frame; and a generation unit that generates a display frame from the image frame read out by the reading unit. The reading unit sequentially reads out the image frame and generates a display frame in the generation unit. When the image frame is the first group of image frames, the generating means generates a display frame from the second group of image frames and the first group of image frames. And generating a beam.

As a result, even if image frames having a small size are mixed in the recorded image frames, the image frame having a small size can generate a display image with reference to another image frame having a large size. Therefore, it is possible to supplement the peripheral area of the image frame having a small size, and it is possible to reproduce the image without causing omission in the reproduction screen.
The reading means may sequentially read only the second group of image frames and cause the generating means to generate a display frame.

As a result, only large image frames can be reproduced, so that thinning reproduction can be performed at a frame rate lower than a predetermined frame rate that is a frame rate at the time of shooting.
Each of the image frames recorded on the recording medium has time stamp information indicating a display timing, and the first group of image frames and the second group of image frames are different from each other. When the display means generates the display frame, the reading means sequentially reads out the image frames based on the time stamp information, and causes the generation means to generate the display frame. Also good.

  The image frames recorded on the recording medium have time stamp information indicating the display timing, and the first group of image frames and the second group of image frames are recorded separately. The reading means may be recorded on a medium, and the reading means may sequentially read image frames based on the time stamp information and cause the generating means to generate display frames.

As a result, even if a large image frame and a small image frame are recorded separately, the display timing of each image frame can be known, so that it can be reproduced in the correct order.
Further, since only large image frames are recorded in the continuous area, it is possible to facilitate thinning reproduction at a frame rate lower than a predetermined frame rate.

<Embodiment 1>
<Overview>
The present invention pays attention to the fact that when shooting a moving image, there are cases where it is desired to record the entire screen, and cases where it is desired to record while focusing on a specific part of the subject.
That is, the present invention aims to reduce data when it is desired to record a specific part of the subject among these cases.

When it is desired to record a specific part of the subject, only the partial image including the specific part needs to be reproduced at a desired frame rate.
For example, when a thrown ball is shot at high speed, only the rotating ball portion needs to be played back at a high frame rate, and the background need not be played back at a high frame rate. The subject in this case is a scene to be photographed, and the specific part is a ball.

In the moving image recording / playback apparatus according to the present invention, the entire image frame is recorded every predetermined number of image frames shot at a specified frame rate, and the other image frames are partial images including a specific portion. To record.
By recording in this way, the specific portion records image data of a specified frame rate, and at the time of reproduction, the screen of the specific portion can be updated at the specified frame rate. Further, since the entire screen is recorded every predetermined number, the entire screen is also updated.

First, before describing the configuration of a moving image recording / playback apparatus according to the present invention, image frames recorded by the apparatus will be described with reference to FIGS.
Note that the sizes of the frames, partial images, balls, and the like drawn in FIG. 2 and the like do not accurately represent the actual sizes for the sake of easy understanding of the drawing and explanation. Also, the relationship between the frame rate and the ball position does not represent the actual movement of the thrown ball for the sake of easy understanding of the figure and convenience of explanation.

<Image Frames Recorded by Moving Image Recording / Reproducing Device According to the Present Invention>
FIG. 2 is a diagram illustrating an example of an image frame recorded by a normal video recorder or the like.
This is a photograph of a thrown ball at a frame rate of 180 fps, and shows frames 1 to 8. This image frame is recorded.
FIG. 3 is a diagram illustrating an example of an image frame in which a moving image similar to FIG. 2 is recorded by the moving image recording / playback apparatus according to the present invention. In this figure, the captured image frame is indicated by a dotted rectangle, and the recorded partial image is indicated by a solid rectangle.

In the moving image recording / playback apparatus according to the present invention, the entire image frame corresponding to the frame rate of 60 fps is recorded, and the other frame records a partial image including the ball.
Specifically, only the frame 1, the frame 4, and the frame 7 are recorded as an image frame having a normal frame size, and the other image frames are recorded as a portion including a ball as an image frame.

Next, a case where a part of the image frame including the ball is reproduced will be described with reference to FIG.
FIG. 4 is a diagram showing a procedure for reproducing frame 2.
The frame 1 and the frame 4 are combined to generate a background image, and the recording portion of the frame 2 is further combined to reflect the frame 2 on the background image to generate the entire frame 2.

The background image is synthesized by weighting according to the frame timing position. For example, a weighted average is taken for the luminance value of each pixel of the frame. Specifically, in the case of frame 2, the frame 1 is weighted “2”, the frame 4 is “1”, and the image is synthesized.
Frame 2 is generated by fitting the partial image in accordance with the position information of the partial image. In the case of generating the frame 3, the synthesis is performed in the same manner. At this time, the background image for frame 3 may be generated by changing the weighting ratio, or the background image for frame 2 may be used.

In this way, even in the case of moving images taken at high speed, the image quality of the moving image in the region of interest does not deteriorate, and the surrounding images of the region of interest are interpolated from the front and back, minimizing information loss. Therefore, it is possible to reproduce continuous frame images even when reproducing high-speed moving images.
In addition, the size of the partial images of the frame 2 and the frame 3 is set so as to hide the balls of the frames 1 and 4. In this figure, the rectangle 302 indicates the frame 2 and the rectangle 303 indicates the frame 3 (see the frame 301).

The size of the partial image is determined in consideration of the frame rate and the object to be photographed at high speed, for example, the speed of the ball, the moving distance of the ball on the screen, and the like.
In this figure, the background image is obtained by excluding the pixel data of the rectangle 304 which is a common part of the rectangle 302 and the rectangle 303. This is because when the frames 2 and 3 are generated, the same background image can be used.

  The moving picture recording / reproducing apparatus according to the present invention will be described below. This moving image playback apparatus can play back a moving image shot at a frame rate of 180 fps at a frame rate of 180 fps or 60 fps by thinning out the frame. It shall be recorded after being encoded according to the H.264 standard. The frame size is 1920 × 1080 (pixels), and the size of the partial image is 1280 × 720 (pixels), which is approximately ½ size.

<Configuration>
Here, functions of the moving image recording / playback apparatus according to the present invention will be described with reference to FIG.
FIG. 1 is a functional block diagram showing a configuration of a moving image recording / playback apparatus.
First, the moving image recording / playback apparatus 1000 has a function of shooting a moving image in the same way as a normal video recorder, and the data of the shot video signal is recorded on a recording medium 3000 such as a memory card. .

In the present embodiment, the recording medium 3000 is assumed to be a recording medium that can be removed from the moving image recording / reproducing apparatus, but may be a storage area inside the moving image recording / reproducing apparatus.
The moving image recording / reproducing apparatus 1000 includes a camera unit 1110, an image input unit 1100, a subject lock unit 1200, a specific partial position detection unit 1300, a recording frame generation unit 1400, a video compression unit 1500, a frame memory 1600, a recording control unit 1700, and a display. 2110, an image display unit 2100, a display frame generation unit 2200, a playback frame rate storage unit 2300, a video expansion unit 2400, a frame memory 2500, and a playback control unit 2600.

The moving image recording / reproducing apparatus 1000 has two functions: a function of recording a moving image and a function of reproducing a moving image.
<Function to record moving images>
First, a functional unit that realizes a function of recording a moving image will be described.
The camera unit 1110 includes a lens, a CCD (Charge Couple Device), and the like. The CCD has a function of converting the reflected light from the imaging target collected by the lens into an electric signal and outputting the electric signal to the image input unit 1100. At that time, the CCD converts the reflected light into an electrical signal along an arbitrarily set imaging frame rate.

The image input unit 1100 converts the signal received from the camera unit 1110 into a digital signal, and performs various signal processing on the digital signal to generate and output an image frame (hereinafter referred to as “input frame”). .
Next, the subject lock unit 1200 has a function of storing information for detecting a specific portion of the subject that the user wants to record. For example, when it is desired to make the ball a specific part, the information of the image is stored. Specifically, in order to create a template of a specific portion of a subject to be photographed mainly before starting high-speed shooting, the specific portion is recognized in advance, and the edge (contour) information is extracted and used as specific portion template information.

The specific partial template information is stored in a template information storage unit 1210 included in the subject lock unit 1200.
The specific part position detection unit 1300 has a function of detecting the position of the specific part in the input frame output from the screen input unit 1100 after shooting is started.
Specifically, using the specific part template information stored in the template information storage unit 1210, the specific part is detected from the input frame, and the position of the specific part in the input frame is obtained. The obtained position information (hereinafter referred to as “specific partial position information”) is passed to the recording frame generation unit 1400.

Furthermore, the specific part position detection unit 1300 extracts the latest edge information from the detected specific part and stores it in the template information storage unit 1210 as the latest specific part template information. This is because it is used for recognition of a specific portion in the next input frame.
The recording frame generation unit 1400 has a function of generating an image for recording. There are two types of images to be generated: the input frame itself output from the image input unit 1100 (hereinafter referred to as “entire frame”) or a partial area of the input frame (hereinafter referred to as “partial frame”). It is. Which recording frame is generated depends on an instruction from the recording control unit 1700.

The recording frame generation unit 1400 includes a specific part cutout unit 1410. The specific part cutout unit 1410 has a function of cutting out a specific part from an input frame and generating a partial frame.
The partial frame cut out here is an image cut out according to the detected specific partial position information, and is a smaller frame than the input frame including the specific portion. In the present embodiment, the size of the input frame and the whole frame is 1920 × 1080 (pixels), and the size of the partial frame is 1280 × 720 (pixels) of about ½ size.

The recording frame generation unit 1400 outputs the generated whole frame or partial frame to the video compression unit 1500. At this time, information indicating whether the frame is a whole frame or a partial frame, and if it is a partial frame, the partial frame And information indicating the position in the input frame.
For example, it outputs together with a flag indicating whether it is an entire frame or a partial frame and header information including position information of the partial frame.

The video compression unit 1500 has a function of compressing and encoding the recording frame received from the recording frame generation unit 1400 and outputting it to the recording medium 3000. When this compression encoding is performed, header information such as distinction between whole frames and partial frames is also output to the recording medium 3000. Details of the compression encoding will be described later with reference to FIGS.
The frame memory 1600 is a memory used when the video compression unit 1500 performs an encoding process, and has a function of storing pixel data in units of frames for use as a reference image.

The recording control unit 1700 has a function of instructing the recording frame generation unit 1400 whether to generate a whole frame or a partial frame from an input frame.
For example, when the imaging frame rate is 180 fps, an instruction to generate partial frames in two input frames is issued in succession, and an instruction to generate an entire frame in the next one input frame is issued. That is, an instruction to generate an entire frame is issued every two input frames. An entire frame is generated from an input frame corresponding to a frame rate of 60 fps, and a partial frame is generated from another input frame (see FIG. 3).

<Function to play moving images>
Next, a functional unit that realizes a function of reproducing a moving image will be described.
The playback frame rate storage unit 2300 has a function of storing a playback frame rate. In the present embodiment, a moving image shot at 180 fps can be played back at 180 fps or 60 fps. Therefore, the frame rate storage unit 2300 stores information indicating 180 fps or 60 fps.

The playback control unit 2600 refers to the playback frame rate storage unit 2300 and notifies the video decompression unit 2400 of a recording frame to be decoded. In the case of 180 fps, it is notified that both the entire frame and the partial frame are to be decoded, and in the case of 60 fps, it is notified that only the entire frame is to be decoded.
Next, the video decompression unit 2400 has a function of reading and decoding moving image data that has been compression-encoded from the recording medium 3000. Decoding is performed based on the notification from the reproduction control unit 2600, and only the entire frame or the entire frame and the partial frame are decoded.

Furthermore, the video decompression unit 2400 has a function of storing the encoded whole frame and partial frame in the frame memory 2500, and in the case of a partial frame, passing the encoded partial frame to the display frame generation unit 2200.
Also, when the decoded partial frame is transferred to the display frame generation unit 2200, the header information of the partial frame is also transferred.

The frame memory 2500 is a memory used when the video decompression unit 2400 performs a decoding process, and has a function of accumulating pixel data of a partial frame, pixel data to be used as a reference image, and the like. The entire frame accumulated by the video decompression unit 2400 is used as a display frame.
The display frame generation unit 2200 has a function of generating an image frame for display (hereinafter referred to as “display frame”) from the partial frame received from the video decompression unit 2400 and storing it in the frame memory 2500. The size of the display frame is the same as the input frame.

That is, when a partial frame is received, a display frame with a background added to the partial frame is generated and stored in the frame memory 2500. Note that the display frame of the entire frame is stored in the frame memory 2500 by the video decompression unit 2400.
The display frame generation unit 2200 includes a synthesis unit 2210 and a background image generation unit 2220, and generates a display frame from the partial frames using these functional units.

The background image generation unit 2220 generates a background image from high-resolution whole frames before and after the partial frame.
The synthesizing unit 2210 synthesizes the partial frame and the background image of the display frame size. Specifically, the region other than the partial frame is a background image generated by the background image generation unit 2220, and the partial frame region generates a display frame by fitting the partial frame image as it is (see FIG. 4).

The image display unit 2100 reads the display frame from the frame memory 2500, generates a video signal with a playback frame rate, and displays it on the display 2110.
In this functional block diagram, only functional units directly related to the present invention are shown, and functions that a normal camera recorder has, for example, a so-called user interface for setting an imaging frame rate or the like is not illustrated. Shall.

<About compression encoding>
Details of compression encoding of image data recorded by the moving image recording / reproducing apparatus 1000 of the present invention will be described with reference to FIGS.
FIG. 5 is a diagram showing image frames reproduced at 180 fps and 60 fps.
The moving image recording / reproducing apparatus 1000 can reproduce a moving image shot at a frame rate of 180 fps at a frame rate of 180 fps or 60 fps.

Therefore, it is necessary to perform an encoding process so that such a reproduction process can be easily performed.
The upper recording data indicates the recording data of the moving image recorded at the frame rate of 180 fps. Full frame and partial frame are mixed.
When the recorded data is reproduced at a frame rate of 60 fps, only the entire frame is displayed as shown in the middle.

In the case of reproduction at 180 fps, as shown in the lower part, a display frame having the same size as the entire frame is generated from the partial frames and displayed.
FIG. 6 is a diagram illustrating a reference relationship of recording frames.
In this figure, the arrow indicates the frame reference direction. For example, frame 4 refers to frame 1 (see arrow 200).

When playing back at 60 fps with this apparatus, only the entire frame is required as shown in FIG. 5, and therefore the frame to be referred to must be limited to the entire frame. That is, it is necessary that the entire frame cannot refer to the partial frame.
However, the reverse, that is, the partial frame can refer to the entire frame. This is because when decoding a partial frame, the entire frame is always decoded.

For example, the frame 4 cannot refer to the frame 6 (see arrow 202), but the frame 6 can refer to the frame 4 (see arrow 201).
Therefore, when performing the encoding process, the entire frame with the timing of 1/60 seconds and the partial frame with the timing of 1/180 seconds have different motion detection targets, and therefore it is preferable to encode them as separate sequences. That is, motion compensation is performed on the entire frame, and motion compensation is performed on the partial frame including the partial frame.

In addition, although it is possible for a partial frame to refer to another partial frame, in this embodiment, the partial frame is recorded as a picture that is not referred to.
FIG. 7 is a diagram illustrating a structure example of an access unit.
The bit stream of the compressed image is composed of a plurality of NAL (Network Abstraction Layer) units, and in order to access in units of pictures, one segment of several NAL units is used as an access unit.

An example of the access unit 100 that is a bit stream obtained by compressing an image frame of the present invention will be described.
The access unit 100 is composed of a supplemental enhancement information (SEI) 120 that is supplementary additional information, picture data, and the like following the AU delimiter 110 that indicates the head.

The AU delimiter 110 includes reference information 111 indicating whether or not it is a reference picture, and “9” indicating that it is an NAL unit of the AU delimiter.
In the case of a partial frame, the reference information 111 is set to indicate that it is a non-reference picture, and in the case of an entire frame, it is set according to the entire frame.
The SEI 120 includes size information 121, a screen size 122, and a position 123.

The size information 121 is information indicating whether the access unit 100 is the data of the entire frame or the data of the partial frame, and “0” is set for the entire frame, and “1” is set for the partial frame. And
The image size 122 is a frame size. In this embodiment, the size of the partial frame is fixed. In the case of the whole frame, (1920, 1080) is set in (Sx, Sy), and (1280, 720) is set in (Sx, Sy) in the case of a partial frame.

The position 123 is information representing the position of the partial frame in the input frame. In the case of the whole frame, (Px, Py) is set to (0, 0), and in the case of a partial frame, (Px, Py) is set as follows.
In the coordinate system 130, the upper left of the input frame (dotted rectangle) is (0, 0), and the upper left of the partial frame (solid rectangle) is represented by coordinates (Px, Py).

The image size 122 and the position 123 may be meaningful only in the case of a partial frame. This is because the whole frame is always set to (1920, 1080) for (Sx, Sy) and (0, 0) for (Px, Py).
FIG. 8 is a diagram showing a frame memory that stores image frames in the encoding process according to the present invention.

A frame memory 1699 shows an example in which pixel data of an image frame is stored in order when encoding an image frame (see FIG. 2) taken by a normal video camera. The frame memory 1600 is a moving image recording / reproducing of the present application. An example is shown in which pixel data of an image frame is stored when an image frame (see FIG. 3) photographed by the apparatus is encoded.
In the case of this embodiment, the entire frame is 1920 × 1080 (pixels) and the partial frame is 1280 × 720 (pixels) of about ½ size. 5 times.

For example, in the area where frames 1 to 7 are stored in the frame memory 1699, frames 1 to 10 can be stored in the present invention.
FIG. 9 is a diagram showing the time required for the encoding process in the present invention.
The upper part is a diagram showing the encoding processing rate when the entire frame is processed at 180 fps by the encoding processing device having the encoding capability of 60 fps, and the lower part is a mixture of the entire frame and the partial frame, It is a figure which shows the encoding process rate when it is made to process at 180 fps.

In the case of the present embodiment, processing performance equivalent to that achieved about 1.5 times the encoding processing performance can be exhibited.
Therefore, for example, when the shooting frame rate is set to exceed the encoding processing rate, shooting at the shooting frame rate can be performed by designating a specific portion.
<Operation>
Next, the operation of the moving image recording / reproducing apparatus 100 will be described.

Two processes, a process of recording a moving image and a process of reproducing a moving image, will be described.
<Process for recording moving images>
First, processing for recording a moving image will be described with reference to FIG.
FIG. 10 is a flowchart showing a moving image recording process performed by the moving image recording / playback apparatus 100.

The user designates that shooting is to be performed at 180 fps (step S100), and locks a specific portion that is mainly desired to be shot (step S110).
Specifically, for example, a shooting frame rate is selected from a menu or the like, a specific portion is copied in a predetermined rectangle in the viewfinder of the liquid crystal screen, and a lock confirmation button is pressed.
The subject lock unit 1200 recognizes a specific portion in the rectangle, extracts the edge information, and stores it in the template information storage unit 1210 as specific portion template information.

When the subject lock unit 1200 stores the specific part template information in the template information storage unit 1210, the subject lock unit 1200 notifies the user of registration of the specific part by displaying a message or the like indicating that the lock has been completed.
The user starts shooting by pressing the start button or the like (step S120).
The start of shooting is notified to the recording control unit 1700 via a user interface unit (not shown) or the like, and the recording control unit 1700 that has received the shooting start notification issues an instruction to start shooting to each function unit. , Start timing at 180 fps.

When the timing is 60 fps, the recording control unit 1700 notifies the recording frame generation unit 1400 to that effect, and at other 180 fps timings, it notifies that the timing is not 60 fps.
Upon receiving an instruction to start shooting, the camera unit 1110 converts the reflected light of the subject into an electrical signal at the set imaging frame rate of 180 fps, and transmits it to the image input unit 1100.

The image input unit 1100 that has received the electrical signal performs various signal processing to generate an input frame, and transmits the input frame to the specific portion position detection unit 1300 and the recording frame generation unit 1400 (step S125).
The specific part position detection unit 1300 that has received the input frame searches the input frame with the specific part template information stored in the template information storage unit 1210, detects the position of the specific part, and generates a recording frame as the specific part position information. It transmits to the part 1400 (step S130).

At the same time, the specific part position detection unit 1300 transmits the specific part in the searched input frame to the subject lock unit 1200. The subject lock unit 1200 updates the specific part template information in the template information storage unit 1210 based on the received specific part.
The recording frame generation unit 1400 that has received the specific partial position information, when notified from the recording control unit 1700 that the timing is 60 fps (step S140: YES), the entire input frame received from the image input unit 1100 The frame is transmitted to the video compression unit 1500 as a frame.

The video compression unit 1500 that has received the entire frame stores it in the frame memory 1600 (step S170).
On the other hand, the recording frame generation unit 1400 that has received the notification that the timing is not 60 fps from the recording control unit 1700 (step S140: NO), the specific part position received from the specific part position detection unit 1300 is sent to the specific part cutout unit 1410. The partial frame is generated by passing the information and the input frame (step S150).

The recording frame generation unit 1400 transmits the partial frame generated by the specific partial cutout unit 1410 to the video compression unit 1500.
The video compression unit 1500 that has received the partial frame stores it in the frame memory 1600 (step S160).
The video compression unit 1500 storing the recording frame, that is, the entire frame or the partial frame in the frame memory 1600 encodes the recording frame (step S180) and records it on the recording medium 3000 (step S190).

The moving image recording process in steps S125 to S190 is performed until an instruction to end shooting is issued (step S195: NO). If there is an instruction to end shooting (step S195: YES), the process ends.
<Processing to play back moving images>
Next, processing for reproducing a moving image will be described with reference to FIGS.

The moving image recording / reproducing apparatus 1000 can reproduce a moving image captured at a frame rate of 180 fps at 180 fps or by thinning out frames at 60 fps. The process of reproducing at 180 fps and the process of reproducing at 60 fps will be described with reference to FIGS. 11 and 12, respectively.
FIG. 11 is a flowchart showing a moving image reproduction process at a frame rate of 180 fps.

The user designates playback at 180 fps (step S200), and presses a playback button or the like to start playback (step S205).
The start of reproduction is notified to the reproduction control unit 2600 via a user interface unit (not shown) or the like, and the reproduction control unit 2600 that has received the reproduction start notification issues an instruction to start reproduction to each function unit. . It is assumed that the playback frame storage unit 2300 stores the playback frame rate specified by the user by the user interface.

In response to the instruction to start reproduction, the video decompression unit 2400 sequentially reads the data of the recording frames encoded from the recording medium 3000 (step S210).
The video decompression unit 2400 determines whether the read data is for an entire frame or a partial frame (step S220). Specifically, referring to the size information 121 of the access unit 100 (see FIG. 7), it is determined that “0” is the data of the entire frame, and “1” is the data of the partial frame.

When the image size is the whole frame (step S220: whole), a decoding process corresponding to the size of the whole frame is performed (step S230) and stored in the frame memory 2500 (step S270).
On the other hand, when the image size is a partial frame (step S220: portion), the video decompression unit 2400 performs a decoding process according to the size of the partial frame (step S240), and the decoded partial frame is stored in the frame memory 2500. Is temporarily stored and transferred to the display frame generation unit 2200. At this time, the image size 122 and the position 123 (see FIG. 7) of the access unit 100 are also passed.

The display frame generation unit 2200 that has received the decoded partial frame instructs the background image generation unit 2220 to generate a background image of the partial frame.
Receiving the instruction, the background image generation unit 2220 reads the entire frame before and after the partial frame from the frame memory 2500 and generates a background image (step S250).
Next, the display frame generation unit 2200 instructs the combination unit 2210 to generate a display frame by combining the background image generated by the background image generation unit 2200 and the partial frame image.

Upon receiving the instruction, the composition unit 2210 refers to the image size 122 and the position 123 received from the video decompression unit 2400, and synthesizes the partial frame with the background image to generate a display frame (step S260).
The display frame generation unit 2200 stores the display frame generated by the synthesis unit 2210 in the frame memory 2500 (step S270).

The image display unit 2100 displays the display frame stored in the frame memory 2500 at the playback frame rate (step S280).
Until there is an instruction to end reproduction or there is no encoded moving image data, the reproduction process of moving images in step S210 to step S280 is performed (step S290: NO), and the reproduction is ended. When there is an instruction or when the data is finished (step S290: YES), the process is finished.

Next, processing for reproducing at 60 fps will be described.
FIG. 12 is a flowchart showing moving image reproduction processing at a frame rate of 60 fps.
Only differences from the case of reproduction at a frame rate of 180 fps will be described.
The difference is that the user designates playback at 60 fps (step S300) and displays at a frame rate of 60 fps (step S320), and if the frame is a partial frame (step S310: partial ), The partial frame decoding process is not performed, and the display frame is naturally not generated. That is, the partial frame is thinned out. Other processes are the same as those in FIG.

Specifically, the video decompression unit 2400 determines whether the read image data is an entire frame or a partial frame (step S310), and if the image size is an entire frame (step S310: entire), Decoding processing according to the size is performed (step S230) and stored in the frame memory 2500 (step S270).
On the other hand, if the image size is a partial frame (step S310: part), the video decompression unit 2400 reads out the next encoded frame data without performing any processing (step S210).

<Modification 1>
In the first embodiment, the whole frame and the partial frame are mixedly recorded, but in this modification, the whole frame and the partial frame are recorded separately.
FIG. 13 is a conceptual diagram of Modification 1 in which the entire frame and the partial frame are recorded separately.
By recording in this way, there is an advantage that a file in which only an entire frame, which is an image frame having a frame rate of 60 fps, can be reproduced by a normal reproducing apparatus other than the present invention.

In the moving image recording / reproducing apparatus 100 according to the present invention, a method of reproducing at 180 fps from data recorded by dividing the entire frame and the partial frame will be described below.
First, it is assumed that the data of the entire frame of 60 fps has information indicating that auxiliary data exists in the SEI 120 (see FIG. 7) of each access unit 100.
In this modification, picture timing SEI included in the SEI 120 of each access unit 100 is used.

All access units include a picture timing SEI. This picture timing SEI includes a difference value from the decoding time of an IDR (Instantaneous Decoder Refresh) picture and a difference value between the decoding time and the display time. Thus, the display timing from the IDR picture can be obtained.
Note that an IDR picture is image data that serves as a delimiter of a data stream, and image data after the image data is a picture that can be correctly decoded without using previous data.

Therefore, in this modification, it is possible to display in the correct order using the information indicating that auxiliary data of the entire frame exists and the display timing.
Specifically, when reproducing at 180 fps, a frame having a display timing of every 1/180 second is searched from the display timing of the IDR picture from the data of the entire frame of 60 fps and auxiliary data and displayed.

In addition, even when the auxiliary data exists only in a part of the data stream, there is an advantage that high-speed reproduction, slow reproduction, or the like can be performed only on that part using the auxiliary data.
<Modification 2>
In the first embodiment, the partial frame is recorded as an image of 1280 × 720 (pixel) (see FIG. 3), but in the present modification, the partial frame is recorded as an image of 1920 × 1080 (pixel), which is the same as the entire frame. It is.

FIG. 14 is a diagram illustrating an image frame recorded in the second modification.
The hatched portion of the image frame in this figure represents that it is painted with a predetermined color.
The frames 2, 3 and the like recorded as the partial frames in the first embodiment are recorded in the entire frame in which the color other than the portion of the partial frame is one color.
By recording in this way, there are advantages in the following points as compared with the prior art.

First, since there are few changes in an image frame, the amount of data when compression coding is reduced, and the load of the encoding process is also reduced.
Further, even a normal playback device can perform playback.
In the case of this modification, for example, the recording frame generation unit 1400 cuts out a specific portion and generates an entire frame of one color other than that.
<Embodiment 2>
<Overview>
In the first embodiment, a moving image is taken with one camera. In this embodiment, a moving image is taken with two cameras, and the same image data as in the first embodiment is recorded.

<Configuration>
FIG. 15 is a functional block diagram illustrating a configuration of the moving image recording apparatus according to the second embodiment.
The moving image recording apparatus 2000 according to this embodiment includes a camera (4110, 5110), an image input unit (4100, 5100), a specific position detection unit 4200, a recording frame generation unit 4400, a subject lock unit 1200, a recording control unit 1700, a video compression unit. A unit 1500 and a frame memory 1600.

The subject lock unit 1200, the recording control unit 1700, the video compression unit 1500, and the frame memory 1600 have the same functions as those in the first embodiment.
In addition, the camera (4110, 5110), the image input unit (4100, 5100), the specific position detection unit 4200, and the recording frame generation unit 4400 basically include the camera 1110, the image input unit 1100, the specific position of the first embodiment. Although it has the same function as that of the detection unit 1300 and the recording frame generation unit 1400, different points will be described below.

The moving image recording apparatus 2000 includes two cameras, the first system includes a camera 4110 that captures images at high speed and low resolution, and the second system includes a camera 5110 that captures images at low speed and high resolution.
Specifically, the first system camera captures the same 1280 × 720 (pixel) image as the partial frame at a frame rate of 180 fps, and the second system camera uses the same 1920 × 1080 (pixel) frame as the entire frame. Assume that an image is taken at a frame rate of 60 fps.

The first system zooms and records the specific portion, and the second system records the entire movement of the subject.
That is, the first system high-speed and low-resolution camera 4110 captures a part of the image captured by the second system low-speed and high-resolution camera 5110 in detail.
That is, the whole frame described in the first embodiment is shot with the second system, and the partial frame is shot with the first system.

The image input units (4100, 5100) convert the signals received from the respective camera units (4110, 5110) into digital signals, perform various signal processing on the digital signals, and image frames having the respective number of pixels. Is generated and output to the specific partial position detection unit 4200.
Hereinafter, an image frame output from the first system image input unit 4100 is referred to as a “partial frame”, and an image frame output from the second system image input unit 5100 is referred to as an “overall frame”.

The specific partial position detection unit 4200 detects a position where the partial frame image exists in the entire frame image.
The detected position information is sent to the recording frame generation unit 4400 and is fed back to a camera control actuator (not shown) of the camera unit 4110 to change the position of the captured image of the camera. To control.

The partial frame is output to the recording frame generation unit 4400 together with the detected position information.
The recording frame generation unit 4400 outputs either an entire frame or a partial frame to the video compression unit 1500 in accordance with an instruction from the recording control unit 1700.
That is, processing such as extraction of a specific part as described in the first embodiment is not necessary.

Therefore, even an apparatus that does not have a high-resolution and high-speed sensor can easily obtain the same effects as those of the moving image recording / reproducing apparatus of the first embodiment.
Furthermore, by connecting two systems of cameras as in this embodiment, processing for segmenting frames is not necessary, and there is a possibility that sensor sensitivity cannot be obtained or higher-speed shooting is possible. There is an advantage of spreading.

<Supplement>
As described above, the moving image recording / reproducing apparatus according to the present invention has been described based on the embodiment. However, the present invention can be partially modified, and the present invention is not limited to the above-described embodiment. That is,
(1) In the embodiment, the size of the entire frame is set to 1920 × 1080 (pixels) and the size of the partial frame is set to 1280 × 720 (pixels).

In the embodiment, the size of the partial frame is fixed, but may be variable. For example, it changes according to the change of a specific part.
(2) In the embodiment, an example of creating a template based on edge detection is given as a means for recognizing a specific part of a subject.
(3) In the embodiment, the case where the frame rate is 180 fps and 60 fps has been described, but other frame rates may be used.

In the embodiment, the playback frame rate is specified to specify the frame to be decoded and the frame rate to be displayed. However, the frame to be decoded and the frame rate to be displayed may be specified separately. .
More specifically, in the present embodiment, when shooting at a frame rate of 180 fps, all frames are decoded and displayed at 180 fps (see FIG. 11), and moving image frames shot at 180 fps are thinned out. The case of decoding and displaying at 60 fps (see FIG. 12) has been described. However, all the frames shot at 180 fps may be decoded and displayed at 60 fps, that is, slow playback or high speed display at 240 fps may be performed.

In this case, the playback frame rate storage unit 2300 stores a decoding frame rate that specifies a frame to be decoded and a display frame rate that specifies a frame rate to be displayed, and the playback control unit 2600 stores the decoding frame rate. The video expansion unit 2400 is notified, and the image display unit 2100 displays the display frame at the display frame rate.
Further, in the embodiment, the entire frame and the partial frame are regularly recorded, but may be recorded at random. In this case, for example, by recording the entire frame only when the background changes more than a predetermined value, the data amount can be reduced while accommodating the change in the background.
(4) In the embodiment, when the background image of the partial frame is generated, the background image is generated from the entire frames before and after the partial frame, but may be generated only from the past entire frames. In this case, one frame data is read out.
(5) The moving image recording / reproducing apparatus may realize all or some of the components shown in FIG. 1 with a single-chip or multi-chip integrated circuit. For example, the integrated circuit 9000 and the integrated circuit 9100 shown in FIG.
(6) The moving image recording / playback apparatus may be realized by implementing a computer program for all or part of the components shown in FIG. 1, or may be implemented in any other form.

In the case of a computer program, a program written in any recording medium such as a memory card or CD-ROM may be read and executed by a computer, or a program may be downloaded and executed via a network. Also good.
<Details of conventional technology>
Hereinafter, the prior art will be described with reference to FIG.

When the motion detection unit 2 detects a motion in one of the preset regions based on the image captured by the image input unit 1, the motion detection unit 2 sends an alarm corresponding to the region to the area information management unit 3 and the alarm input processing unit 8. And output.
The area information management unit 3 outputs the coordinate position information of the region corresponding to the input alarm to the area detection unit 4 together with the region identification information.

The area detection unit 4 cuts out the region of the coordinate position information from the image input from the image input unit 1 based on the input coordinate position information, and outputs it to the image accumulation selection unit 5 together with the input region identification information. .
The image storage selection unit 5 stores the input image information in the image storage units (6a to 6n) corresponding to the areas.

On the other hand, the alarm input processing unit 8 that has received the alarm collects information on the time and the area corresponding to the alarm from the time management unit 9 and the alarm corresponding area setting unit 10 and stores them in the alarm storage management unit 11.
The alarm information stored in the alarm storage management unit 11 and the images of the image storage units (6a to 6n) selected by the image output selection unit 7 are connected to a monitor terminal connected to a network or the like via the communication unit 12. Send.

With this conventional technology configuration and method, when high-speed shooting or high-speed continuous shooting is performed with a high frame rate (number of images taken per second), only reduced images and clipped images are stored in order to reduce frame memory capacity. In this case, only the cut out video can be reproduced, and the amount of information is lost.
In addition, even if high-speed shooting is performed with a movie or a digital camera, if you want to watch a normal movie (30 fps, 60 fps) or if you want to thin out a photo, you can only see the clipped video was there.

  The high-speed frame image generation apparatus according to the present invention is useful as a device that enables high-speed shooting of inexpensive and high-resolution images in video cameras, digital cameras, and high-speed shooting cameras. In addition, any camera system is useful.

It is a functional block diagram which shows the structure of a moving image recording / reproducing apparatus. It is a figure which shows the example of the image frame which a normal video recorder etc. record. It is a figure which shows the example of the image frame which the moving image recording / reproducing apparatus based on this invention records the moving image similar to FIG. It is a figure which shows the procedure in the case of reproducing | regenerating the flame | frame 2. FIG. It is a figure which shows the image frame reproduced | regenerated at 180 fps and 60 fps. It is a figure which shows the reference relationship of a recording frame. It is a figure which shows the structural example of an access unit. It is a figure which shows the frame memory which stored the image frame by the encoding process in this invention. It is a figure which shows the encoding process time in this invention. 4 is a flowchart showing a moving image recording process performed by the moving image recording / reproducing apparatus 100. It is a flowchart showing a moving image reproduction process with a frame rate of 180 fps. It is a flowchart showing a moving image reproduction process with a frame rate of 60 fps. It is a conceptual diagram of the modification 1 which records separately an entire frame and a partial frame. FIG. 10 is a diagram illustrating an image frame recorded in a second modification. FIG. 6 is a functional block diagram illustrating a configuration of a moving image recording apparatus according to a second embodiment. It is a figure which shows the example of an imaging | photography screen of a prior art. It is a block diagram of the conventional surveillance camera.

Explanation of symbols

1000 Moving Image Recording / Reproducing Device 1100 4100 5100 Image Input Unit 1110 4110 5110 Camera Unit 1200 Subject Locking Unit 1300 4200 Specific Part Position Detection Unit 1400 4400 Recording Frame Generation Unit 1500 Video Compression Unit 1600 2500 Frame Memory 1700 Recording Control Unit 2000 Moving Image Recording Device 2100 Image display unit 2110 Display 2200 Display frame generation unit 2300 Playback frame rate storage unit 2400 Video expansion unit 2600 Playback control unit 3000 Recording medium

Claims (14)

A moving image recording apparatus for recording a moving image of a subject,
Imaging means for sequentially outputting image frames constituting a moving image at a predetermined frame rate;
Of the image frames output by the imaging means, every predetermined number of image frames, the recording means for recording the whole image frame, and the other image frames recording a part of the image frame on one or a plurality of recording media. A moving image recording apparatus comprising: The moving image recording apparatus according to claim 1, wherein the part of the image frame recorded by the recording unit is a part including a specific part of the subject. The moving image recording apparatus further includes encoding means for encoding an image frame,
The recording means, when recording a part of an image frame, encodes and records an image frame in which a part other than the part of the image frame output from the imaging means has a predetermined color is encoded by the encoding means. The moving image recording apparatus according to claim 1. The moving image recording apparatus further includes generating means for generating a display frame for display from an image frame recorded on the recording medium,
The generating means includes
A display frame is generated by sequentially reading out image frames, and when the read image frame is only a part, the display frame is generated from the image frame recorded as a whole and a part of the image frame. The moving image recording apparatus according to claim 1. The recording means records a part of the image frame on the recording medium in association with information indicating the position of the part in the image frame,
The generating means includes
The moving image recording apparatus according to claim 4, wherein when the read image frame is a part, a display frame in which the part is reflected at a position indicated by the information is generated. The generating means further includes:
The moving image recording apparatus according to claim 4, wherein only the image frames recorded as a whole are sequentially read to generate a display frame. A moving image playback device for playing back captured images recorded on one or more recording media,
The captured image recorded on the recording medium is composed of a plurality of image frames captured at a predetermined frame rate, and the size of the first group of image frames is smaller than the size of the second group of image frames,
The video playback device
Reading means for reading out an image frame constituting a captured image from the recording medium;
Generating means for generating a display frame from the image frame read by the reading means,
The reading means sequentially reads image frames, causes the generating means to generate display frames,
When the image frame is a first group of image frames, the generation unit generates a display frame from the second group of image frames and the first group of image frames. The reading means includes
The moving image reproducing apparatus according to claim 7, wherein only the second group of image frames is sequentially read out and the generation unit is configured to generate a display frame. Each of the image frames recorded on the recording medium has time stamp information indicating a display timing, and each of the first group of image frames and the second group of image frames is in a separate continuous area. Recorded,
The reading means includes
8. The moving image reproduction apparatus according to claim 7, wherein when the display frame is generated by the generation unit, the image frame is sequentially read based on the time stamp information, and the generation unit generates the display frame. . Each of the image frames recorded on the recording medium has time stamp information indicating a display timing, and the first group of image frames and the second group of image frames are stored on different recording media. Recorded,
The reading means includes
8. The moving image reproduction apparatus according to claim 7, wherein image frames are sequentially read out based on the time stamp information, and the generation unit is caused to generate a display frame. A moving image recording method used in a moving image recording apparatus for recording a moving image of a subject,
An imaging step of sequentially outputting image frames constituting the moving image at a predetermined frame rate;
Among the image frames output in the imaging step, a recording step of recording a predetermined number of image frames on the entire image frame, and other image frames recording a part of the image frame on one or more recording media. A moving image recording method comprising: An integrated circuit of a moving image recording apparatus for recording a moving image of a subject,
Imaging means for sequentially outputting image frames constituting a moving image at a predetermined frame rate;
Of the image frames output by the imaging means, every predetermined number of image frames, the recording means for recording the whole image frame, and the other image frames recording a part of the image frame on one or a plurality of recording media. An integrated circuit comprising: The integrated circuit of the moving image recording apparatus further includes a generating unit that generates a display frame for display from an image frame recorded on the recording medium,
The generating means includes
A display frame is generated by sequentially reading out image frames, and when the read image frame is only a part, the display frame is generated from the image frame recorded as a whole and a part of the image frame. The integrated circuit according to claim 12. An integrated circuit of a moving image reproduction apparatus for reproducing a captured image recorded on one or a plurality of recording media,
The captured image recorded on the recording medium is composed of a plurality of image frames captured at a predetermined frame rate, and the size of the first group of image frames is smaller than the size of the second group of image frames,
The integrated circuit of the moving image playback device is:
Reading means for reading out an image frame constituting a captured image from the recording medium;
Generating means for generating a display frame from the image frame read by the reading means,
The reading means sequentially reads image frames, causes the generating means to generate display frames,
When the image frame is the first group of image frames, the generation unit generates a display frame from the second group of image frames and the first group of image frames.

Images