JP2009200985A - Video processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem wherein, in the prior arts, since an interpolation frame is generated on the basis of a motion vector generated on the basis of recorded video information, it is difficult to accurately reproduce panning information, tilting information and zooming information in recording, and since acceleration information can not be estimated from the recorded video information, when the interpolation frame is generated using a motion compensation vector, deviation from actual operation may occur. <P>SOLUTION: In a photographing unit, a video signal is encoded, camera information is recorded together with the encoded video information, and the recorded video signal and camera information are input to a reproduction unit. In the reproduction unit, the recorded video information is decoded, and while using camera information such as, e.g., zooming information, panning information, acceleration information or tilting information, N pieces of interpolation frames are generated and output between image frames of the decoded video signal, thereby achieving frame interpolation meeting a status in recording. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

The present invention relates to a video processing apparatus. For example, in a video shooting and playback device that captures video with a video camera and plays back on a monitor, camera information such as zoom information, pan information, acceleration information, or tilt information is recorded during video shooting, and camera information is used during playback In addition, the present invention relates to a system, method, and apparatus that can reproduce at a rate higher than the recorded frame rate by generating and inserting an interpolation frame between image frames of the recorded video.

  As background art of this technical field, for example, there is JP-A-2005-6275 (Patent Document 1). In this publication, as a subject, “a device and method for further improving the accuracy of interpolation frame creation and an interpolation frame creation program are provided, and a method and program suitable for implementation in the device are provided. An interpolation frame generation device is provided. ”As an solving means,“ interpolation frame generation device 201 is an interpolation for interpolating an image frame obtained by decoding encoded image signal 210 subjected to motion compensation encoding. An apparatus for creating a frame, which includes a motion vector deriving unit 203 and an interpolation frame creating unit 204. The motion vector deriving unit 203 obtains a motion compensation vector of a coding block constituting the coded image signal 210. Based on the motion vectors of the image blocks that make up the image frame, the interpolation frame creation unit 204 And means for creating between frames as the motion vector of the image block, using the motion compensation vectors of coded blocks. "Is described as (see Abstract).

  Another background art of this technical field is, for example, Japanese Patent Application Laid-Open No. 5-14776 (Patent Document 2). In this publication, the purpose is to “use camera operation information together with the camera video after the camera shooting is recorded”, and the configuration is “pan angle when shooting with the TV camera 1, Information on the tilt angle, zoom amount, focus amount, and camera position is detected by the camera information detector 2, encoded by the digital encoder 3, superimposed on the vertical blanking of the video signal from the camera 1, and taped by the VTR 5. Is recorded ”(see summary).

  Moreover, as another background art of this technical field, there exists Unexamined-Japanese-Patent No. 2006-262135 (patent document 3), for example. In this publication, a problem is described as “capable of recording and reproducing a moving image using acceleration acting on a device acquired at the time of moving image shooting.” As a solving means, “the camera device 1 is in a moving image shooting mode. Shoots a moving image composed of a plurality of frame images, and acquires the acceleration by the acceleration detection unit 46 when shooting the moving image, and adjusts the number of frames of the moving image by thinning out / interpolating the frame image according to the acquired acceleration. Then, the moving image with the adjusted number of frames is recorded in the memory card 32 or the built-in memory 33 ”(see summary).

Another background art of this technical field is, for example, Japanese Patent Laid-Open No. 10-233996 (Patent Document 4). The gazette provides a video signal playback apparatus and video signal playback method for playing back an image with less sense of incongruity when detecting parallel movement of an image when playing an intermittently recorded video signal. As a solving means, the video signal of the intermittent image from the monitoring video, the video phone, etc. is input to the input signal processing unit 2. The system control unit 8 is connected to the additional information processing unit 4 or the motion. The system control unit 8 detects that the intermittent image has been translated based on the information from the vector detection unit 5. When the system control unit 8 detects that the parallel image has been translated, the system control unit 8 detects whether the intermittent image is detected from the intermittent interval of the intermittent image detected by the additional information processing unit 4. The number of interpolation frames to be inserted is determined to control the interpolation processing unit 6. The interpolation processing unit 6 generates an interpolation frame from the intermittent image, and performs this interpolation frame processing. It is described to synthesize the intermittent image. "(See Abstract).

Japanese Patent Laying-Open No. 2005-6275 (first page, FIG. 1) JP-A-5-14776 (first page, FIG. 2) JP 2006-262135 A (paragraphs 52 to 82) Japanese Patent Laid-Open No. 10-233996 (paragraphs 13 to 21, FIGS. 14 and 16)

  With the technique described in Patent Document 1, since an interpolation frame is generated based on a motion vector generated based on recorded video information, it is difficult to accurately reproduce pan information, tilt information, and zoom information at the time of recording. . Further, the acceleration information cannot be estimated from the recorded video information. Therefore, when an interpolation frame is generated using the motion compensation vector, a deviation from the actual operation occurs.

  Further, in the technique described in Patent Document 2, information on pan angle, tilt angle, zoom amount, focus amount, and camera position at the time of shooting with the television camera 1 is recorded. Absent.

  In the technique described in Patent Document 3, the acceleration is detected by the acceleration detection unit 46 at the time of moving image shooting, and the number of frames of the captured moving image is adjusted by thinning out / interpolating the frame image according to the acquired acceleration. However, when the interpolation frame is generated, the acceleration is not used at points other than the number of frames.

  In the technique described in Patent Document 4, additional information is recorded in addition to the video signal and the audio signal, and the additional information includes camera work information (panning, zoom-up, etc.) (see paragraph 0018). When the image detects parallel movement, the current frame (or the previous frame) is translated to the position of the interpolation frame, and a still image is synthesized there to generate an interpolation frame (FIGS. 14 and 16). , Paragraphs 52 and 58). However, this technology only deals with the translation of the image, and only the current frame (or the previous frame) is translated to the position of the interpolation frame except for the still image part. Can not.

  In view of the above, an object of one aspect of the present invention is to improve the above-described problem and provide a video processing apparatus with high image quality.

  In order to achieve the above object, one aspect of the present invention employs the configurations described in the claims. As an example, for two consecutive image frames included in the video signal, the camera information is used to correct the image to correspond to the temporal position between the two image frames. Motion compensation is used to generate an interpolation frame that interpolates between the two image frames. For example, in the photographing unit, the video signal is encoded, the camera information is recorded together with the encoded video information, the recorded video signal and the camera information are input to the reproducing unit, and the recorded video information is recorded in the reproducing unit. By generating and outputting N interpolated frames using camera information, for example, zoom information or pan information or acceleration information or tilt information, between the image frames of the decoded video signal, it is possible to record at the time of recording. Realize frame interpolation according to the state.

  According to one aspect of the present invention, a high-quality video processing apparatus can be provided. For example, it is possible to provide a system, a method, and a device that realize frame interpolation according to the recording state.

  Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is an example of a system that embodies this embodiment.

  In the present embodiment, the photographing unit 1 encodes a video signal, records camera information together with the encoded video information, inputs the recorded video signal and camera information to the playback unit 2, and the playback unit 2 Decode the recorded video information, and generate and output N interpolation frames between the image frames of the decoded video signal using camera information, for example, zoom information or pan information, acceleration information or tilt information Thus, the present invention relates to a system that realizes frame interpolation in accordance with the recording state.

  FIG. 2 is a detailed example of the reproducing unit 2 in FIG. The reproduction unit 2 decodes the video signal encoded by the imaging unit 1 based on the decoded video signal and the camera information acquired from the imaging unit 1. One or a plurality of interpolation frames C are generated between the image frame A and the image frame B of the received video signal, and the interpolation frame C is inserted and output between the image frame A and the image frame B. It consists of a frame interpolation unit.

  FIG. 3 is a detailed example of the frame interpolation unit 4 in FIG. The frame interpolation unit 4 generates N interpolation frames at positions that are equally spaced in time between the image frame A and the image frame B, and N frames based on the image frame A and camera information. Interpolation frame A [n] (n = 0, 1,..., N−1), and N interpolation frames B [n] (n) based on the image frame B and camera information. n = 0, 1,..., N−1), and N interpolation frames C [n] (n based on the interpolation frame A [n] and the interpolation frame B [n]. = 0, 1,..., N−1), and an interpolation frame generation unit C.

  FIG. 4 is a detailed example of the interpolation frame generation unit C7 in FIG. The interpolation frame generation unit C7 uses the motion compensation vector and a motion detection unit that generates a motion compensation vector of the image blocks constituting the interpolation frame A [n] and the interpolation frame B [n], and uses the motion compensation vector. And an interpolation frame B [n], and a frame interpolation core unit that generates N interpolation frames C [n] at equally spaced positions in time.

  FIG. 5 is a diagram illustrating an example when one interpolation frame is generated between image frames. With respect to the time interval T0 between the image frame A and the image frame B, an interpolation frame is generated at a temporally central position, that is, at a position T0 / 2 from both the image frame A and the image frame B.

  FIG. 6 is a diagram illustrating an example in which N interpolation frames are generated between image frames. N interpolation frames are arranged at equal intervals with respect to the time interval T0 between the image frame A and the image frame B, that is, between the image frame A and the interpolation frame, between the interpolation frames, and between the interpolation frame and the image frame B. The intervals are T0 / (N + 1), respectively.

  FIG. 7 is a diagram showing another example when N interpolation frames are generated between image frames. Period 210, period 211, and period 212 are periods classified based on camera information. For example, the period 212, the period 211, and the period 210 are set in descending order of correlation between consecutive image frames. In this example, the number of interpolation frames to be inserted between image frames is changed according to the correlation. In the period 212, it is determined that the correlation between the image frames is high, and three interpolation frames are inserted. Similarly, in the period 211, one interpolation frame is inserted. In the period 210, it is determined that the correlation between the image frames is low, and no frame interpolation is performed.

  The correlation is determined based on camera information acquired from the imaging unit 1 and motion compensation vector information. For example, if pan information, zoom information, and tilt information are all within a predetermined threshold, it is determined that the correlation between image frames is high. Also, by determining the correlation based on the degree of disturbance of the motion compensation vector for each image block in the image frame, the camera itself is stationary at the time of shooting, but the object being shot is moving violently. Correlation between image frames may also be determined.

  The degree of disturbance of the motion compensation vector may be determined based on the correlation of the motion compensation vector with respect to surrounding image blocks, or may be determined based on the correlation between the average of the entire screen and the motion compensation vector with respect to each image block.

FIG. 8 is an example of a frame interpolation procedure that embodies this embodiment.
The present embodiment relates to a procedure for generating the interpolation frame 33 at the time intermediate position between two image frames that are separated by T0 in time, that is, the image frame 30 and the image frame 31. The time of the image frame 30 is (T) and the time of the image frame 31 is (T + T0). In this case, the interpolation frame insertion position is (T + T0 / 2). The image frame 30 includes a background image 300 and a moving object 310. The image frame 31 is panned from the image frame 30 by the amount indicated by the pan information 40, and the moving object 310 is further moved toward the upper left.

  The intermediate image frame 32 is a diagram illustrating an example of a motion compensation vector of an image block that forms an image frame with respect to the image frame 30 and the image frame 31. A motion compensation vector 44 for the moving object is generated from the image blocks 50 and 51 constituting the moving object. Also, a motion compensation vector 42 for the background image is generated from the image blocks 52 and 53 constituting the background image. Similarly, a motion compensation vector 43 for the background image is generated from the image blocks 54 and 55 constituting the background image.

  An interpolation frame 33 is generated based on the image frame 30 and the image frame 31 at an intermediate position of the motion compensation vector.

  As shown in the example of the intermediate image frame 32, a motion compensation vector is generated for each image block. As an example of motion compensation vector generation, there is a method of detecting a highly similar image block by a matching search of an image block within a predetermined range in an image frame and using it as a motion compensation vector. In this example, the motion compensation vectors 42 and 43 are generated as parameters equivalent to the pan information 40. However, since the motion compensation vectors 42 and 43 are generated from the reproduced image frame, they do not completely match the actual pan motion. In addition, the probability of erroneous detection of motion compensation vectors is increased. When an interpolation frame is generated based on an incorrect motion compensation vector, an image block is interpolated at an incorrect position, resulting in a decrease in interpolation frame quality.

  From the pan information 40, pan information 41a of the time image frame 30 and the interpolation frame 33 and pan information 41b of the time image frame 31 and the interpolation frame 33 are generated. When the pan information 40 is (+ X, + Y), the pan information 41a is (-X / 2, -Y / 2) and the pan information 41b is (+ X / 2, + Y / 2).

The corrected image frame 30a is a diagram obtained by correcting the image frame 30 using the pan information 40a.
The corrected image frame 31a is a diagram obtained by correcting the image frame 31 using the pan information 40b.
The corrected image frames 30a and 31a are interpolation frames generated from the image frames 30 and 31, respectively, based on the pan information.

  The intermediate image frame 32a generated from the corrected image frames 30a and 31a is a diagram illustrating an example of a motion compensation vector of an image block constituting the corrected image frame with respect to the corrected image frame 30a and the corrected image frame 31a. A motion compensation vector 42a for the moving object is generated from the image blocks 56 and 57 constituting the moving object. The motion compensation vector 42a is a motion compensation vector obtained by removing the pan information 40 from the motion compensation vector 44. In addition, for the image blocks constituting the background image, the panning information 40 is removed, so that the motion compensation vector is almost zero.

The actual pan information 40 is used to generate a motion compensation vector after returning it to a correction frame, thereby reducing the probability of motion compensation vector false detection and interpolating the image block at the correct position as a result. It is possible to improve the interpolation frame quality.

FIG. 9 is a diagram illustrating an example in which linear interpolation is used when generating one interpolation frame between two image frames 400 and 401. The image frames 400 and 401 are composed of the background image 101 and pan from the lower right to the upper left. The pan information of the image frame 400 and the interpolation frame 402, and the interpolation frame 402 and the image frame 401 are assumed to be equal, and the interpolation frame 402 is generated. In addition, when camera information is recorded at a rate twice the speed of an image frame and used during playback, further interpolation is performed by directly using pan information corresponding to an intermediate position frame between two image frames instead of linear interpolation. The quality of the frame can be improved.

10, 11, and 12 are diagrams illustrating an example of generating an interpolation frame when pan information, tilt information, and zoom information are added as camera information, respectively. The interpolation frame generation procedure is as shown in FIG.

FIG. 10 is a diagram illustrating an example in which nonlinear interpolation is used when one interpolation frame is generated between two image frames 400 and 401. The image frames 400 and 401 are composed of the background image 101 and pan from the lower right to the upper left. If the ratio of the interval A between the image frame 400 and the interpolation frame 402 and the interval B between the interpolation frame 402 and the image frame 401 is 1: 3 according to the acceleration information, the pan information is also 1: 3 according to the ratio. It becomes the ratio of. If the pan information of the image frames 400 and 401 is (+ X, + Y), the pan information of the image frame 400 and the interpolation frame 402 is (+ 3X / 4, + 3X / 4). The pan information is (+ X / 4, + Y / 4). In addition, when camera information is recorded at a rate twice the speed of an image frame and used at the time of playback, further interpolation is performed by directly using pan information corresponding to a predetermined position frame of two image frames instead of nonlinear interpolation. The quality of the frame can be improved.

FIG. 11 is a diagram illustrating an example in which one interpolation frame 408 is generated between two image frames 406 and 407 when tilt information is added as camera information. The image frames 400 and 401 are composed of the background image 101 and are inclined clockwise by an angle θ. An interpolation frame using a corrected image frame generated by rotating the image frame 406 clockwise by an angle θ / 2 and a corrected image frame generated by rotating the image frame 407 clockwise by an angle −θ / 2. 408 is generated.

FIG. 12 shows that when zoom information, that is, zoom center coordinates (Xz, Yz) and zoom magnification Z are added as camera information, one interpolation frame 411 is generated between two image frames 409 and 410. It is the figure which showed an example. The image frames 409 and 410 are composed of a background image 412 and are multiplied by Z around the coordinates (Xz, Yz). A corrected image frame generated by multiplying the image frame 409 around the coordinates (Xz, Yz) by √Z, and a correction generated by multiplying the image frame 410 by 1 / √Z around the coordinates (Xz, Yz) An interpolation frame 411 is generated using the image frame.


FIG. 13 is an example of performing search range correction when generating motion compensation vectors from two image frames 409 and 410 when pan information is added as camera information. By making the position shifted by the pan information from the normally determined search range area as the search range, more effective search can be performed, and erroneous motion compensation vector generation determined only from similarity is improved. be able to. Thereby, the quality of the interpolation frame can be improved.

FIG. 14 is another example of a system that embodies this embodiment. The photographing unit 1 has means for confirming the presence or absence of the frame interpolation function of the connected playback unit 2, and when there is a frame interpolation function, supplies camera information together with the encoded video signal, and the frame interpolation function In the case where there is no image data, a switching means for supplying an encoded video signal and not supplying camera information is provided.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment.

In addition, each of the above-described configurations may be configured such that some or all of them are configured by hardware, or are implemented by executing a program by a processor. Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

It is an example of the system which actualizes this example It is an example of the reproduction | regeneration part 2 in the frame interpolation system shown in FIG. It is an example of the frame interpolation part 4 in the reproduction | regeneration part shown in FIG. It is an example of the interpolation frame production | generation part C7 in the frame interpolation part 4 in FIG. It is an example when one interpolation frame is inserted between image frames It is an example when N interpolation frames are inserted between image frames. It is an example when the number of interpolation frames generated between image frames is changed by camera information It is an example of the interpolation frame production | generation procedure which actualizes a present Example It is another example of the interpolation frame production | generation procedure which actualizes a present Example It is another example of the interpolation frame production | generation procedure which actualizes a present Example It is another example of the interpolation frame production | generation procedure which actualizes a present Example It is another example of the interpolation frame production | generation procedure which actualizes a present Example It is another example of the interpolation frame production | generation procedure which actualizes a present Example It is another example of the system which actualizes this example

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up part 2 Playback part 100 Input image | video 101 Image | video information and camera information 102 Image frame containing an interpolation frame

Claims (14)

An imaging unit that captures an image to obtain a video signal, encodes the video signal, records the encoded video signal, and records camera information related to the imaging;
A reproduction unit that decodes the encoded video signal, generates an interpolated frame between image frames of the decoded video signal, and outputs the interpolated frame;
With
The reproducing unit corrects the image frames before and after the interpolation frame is interpolated using the camera information so as to correspond to the temporal position of the interpolation frame, and uses the corrected image frame to perform the interpolation frame. Generating,
A video processing apparatus characterized by the above. In claim 1,
The playback unit decodes the encoded video signal, and a sequence of the decoded video signal based on the decoded video signal and the camera information. A frame interpolation unit that generates one or a plurality of interpolation frames C between the image frame A and the image frame B, inserts the interpolation frame C between the image frame A and the image frame B, and outputs the interpolated frame C. A video processing apparatus comprising: In claim 2,
The frame interpolation unit
Generate N interpolated frames at regular intervals in time between consecutive image frames A and B,
Using the image frame A and camera information, an interpolation frame generation unit A that generates N correction frames A [n] (n = 0, 1,..., N−1), and an image frame B and camera information are used. An interpolation frame generation unit B that generates N correction frames B [n] (n = 0, 1,..., N−1),
An interpolation frame generation unit C that generates N interpolation frames C [n] (n = 0, 1,..., N−1) using the correction frame A [n] and the correction frame B [n]. A video processing apparatus characterized by comprising: In any of claims 1 to 4,
Changing the number of interpolated frames according to the camera information;
A video processing apparatus characterized by the above. In claim 3,
The camera information includes pan information, tilt information, or zoom information,
The interpolation frame generation unit A corrects the image frame A using linear interpolation using pan information or tilt information or zoom information included in the camera information,
A video processing apparatus characterized by the above. In claim 3,
The camera information includes acceleration information,
The interpolation frame generation unit A corrects the image frame A using nonlinear interpolation using acceleration information included in the camera information;
A video processing apparatus characterized by the above. In any one of Claims 1 thru | or 6.
The video processing apparatus, wherein the camera information is recorded at a rate higher than a frame rate. In claim 3,
The interpolation frame generator C
A motion detector that detects a region moving between the correction frame A [n] and the correction frame B [n] and generates a motion compensation vector;
A frame interpolation core unit that generates N interpolation frames C [n] at equal time intervals between the correction frame A [n] and the correction frame B [n] using the motion compensation vector; A video processing apparatus comprising: In claim 8,
The frame interpolation core unit generates an interpolation frame C [n] using linear interpolation;
A video processing apparatus characterized by the above. In claim 8,
The video processing apparatus, wherein the frame interpolation core unit generates an interpolation frame C [n] using nonlinear interpolation using acceleration information. In any of claims 8 to 10,
The motion detector generates a motion compensation vector of image blocks constituting the correction frame A [n] and the correction frame B [n];
A video processing apparatus characterized by the above. In any of claims 8 to 11,
The video processing apparatus, wherein the motion detection unit corrects a search range in which motion compensation vector detection is performed using pan information and tilt information. In any of claims 1 to 12,
The imaging unit has means for confirming the presence or absence of a frame interpolation function of the connected playback unit;
When there is a frame interpolation function, camera information is supplied together with an encoded video signal, and when there is no frame interpolation function, an encoded video signal is supplied, and switching means that does not supply camera information is provided. ,
A video processing apparatus characterized by the above. A reading means for reading the video signal and camera information from a recording medium on which the video signal and camera information are recorded;
Frame correction means for correcting two consecutive image frames included in the read video signal so as to correspond to a temporal position between the two consecutive image frames using the read camera information. When,
Frame interpolation means for generating an interpolated frame for interpolating between the two consecutive image frames using motion compensation for the corrected correction frame;
Output means for combining and outputting the read video signal and the generated interpolation frame;
A video processing apparatus comprising:

Images