JP2017050731A - Moving picture frame interpolation device, moving picture frame interpolation method, and moving picture frame interpolation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a moving picture frame interpolation device capable of performing the interpolation between moving picture frames in higher quality.SOLUTION: A moving picture frame interpolation device creates a lost frame by an interpolation in a color moving picture represented by brightness information of an imaging object and a depth moving picture represented by depth information of an imaging object. The frames of the color moving picture and the depth moving picture are synchronized. The moving picture frame interpolation device includes an interpolation means that, when a frame of any one moving picture of two moving pictures is lost, performs the interpolation of a lost frame on the basis of information of the frame of the other moving picture.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a moving image frame interpolation apparatus, a moving image frame interpolation method, and a moving image frame interpolation program for performing interpolation between frames of moving images.

  Video (moving image) data shot by a camera or the like is transmitted and stored through a physical network such as wired or wireless. In the transmission of this video data, the data bandwidth is limited. Therefore, when the bandwidth for transmitting video data is insufficient, temporal resolution of video is temporarily reduced. Specifically, there is a process of intentionally discarding a part of a video frame acquired at a certain frame rate so as to keep the bandwidth within the limit.

  In particular, when acquiring scene depth information in addition to scene color data as video data, the required bandwidth increases, so there is a need to intentionally discard part of the frame. Rise. Note that the depth information is information that expresses a depth distance between an object existing in the real space and a device used for shooting a scene. Generally, color data requires more bits than depth information, so when discarding a part of a frame, discard both color data and depth information instead of discarding both color data and depth information at a certain time. As a result, processing that satisfies the bandwidth constraint may be performed. In addition, 16 bits or 32 bits may be adopted as the bit depth per pixel of the depth information image. In this case, the depth information may require more bits than the color data. Instead, processing that satisfies the bandwidth constraint may be performed by discarding only the depth information.

  However, if some frames are discarded to satisfy the bandwidth limitation, artifacts such as motion jaggies occur, and the subjective quality of the transmitted video is significantly degraded. In such cases, there is a technique called frame interpolation or frame rate upconversion to reduce artifacts and improve subjective quality.

  Frame interpolation / frame rate up-conversion is processing performed for the purpose of executing frame-speed display for repairing frame loss in a transmission path or expressing smooth motion in video encoding / display. . As a typical method of frame interpolation / frame rate up-conversion, there is a method for realizing frame interpolation by morphing or warping between frames using motion information of a subject or background in a moving image detected by block matching or the like. (See, for example, Non-Patent Document 1).

Takashi Yamamoto, Miki Haseyama, “Study on Frame Interpolation of Moving Images: Approach Using Morphing”, Technical Report of IEICE, vol. 106, no. 112, pp. 1-5, 2006.

  By using the above-described frame interpolation / frame rate up-conversion, it is possible to interpolate a frame discarded to satisfy the restriction due to bandwidth limitation. Thereby, it is possible to prevent the subjective quality of the video from being deteriorated due to the discard of some frames.

  However, in the method as described above, the subject of the target moving image or the background motion has periodicity in the image or is a constant velocity linear motion, the moving object and the object that is not The processing is based on the assumption that some constraints are satisfied, such as clearly distinguishing. Therefore, there is a problem that correct frame interpolation cannot be realized when an irregular motion not corresponding to the above is included in the operation determined from the image.

  The present invention has been made in view of such circumstances, and provides a moving image frame interpolation apparatus, a moving image frame interpolation method, and a moving image frame interpolation program that enable interpolation between moving image frames with higher quality. For the purpose.

  One aspect of the present invention is a moving image frame interpolating apparatus that generates a missing frame by interpolation between a color moving image expressed by luminance information of a shooting target and a depth moving image expressed by depth information of the shooting target. Thus, when the color moving image and the depth moving image are frame-synchronized, and one of the two moving images is missing a frame, it is lost based on the information of the other moving image. It is a moving image frame interpolation apparatus provided with an interpolation means for performing frame interpolation.

  One aspect of the present invention is to provide depth information of the depth moving image to each pixel of the color moving image to form a three-dimensional point group, so that the frame of the color moving image and the depth moving image 3D point grouping means for generating 3D point cloud data that is integrated with the frame in correspondence with each frame, and detecting an optical flow based on the 3D point cloud data, so that frames between the frames in the time direction can be detected. Inter-frame association means for associating each pixel, and the interpolation means interpolates the missing frame based on the association result of each pixel between the frames in the time direction.

  One aspect of the present invention is the moving image frame interpolating apparatus, in which the interpolation unit corresponds to the three-dimensional points corresponding to the frames before and after the missing frame when the depth moving image frame is missing. With reference to the group data and the frame of the color moving image at the same time as the missing frame, the missing frame of the depth moving image is generated by interpolation to obtain the pixel value of the missing frame.

  One aspect of the present invention is the moving image frame interpolating device, in which the interpolation unit corresponds to the three-dimensional points corresponding to the frames before and after the missing frame when the frame of the color moving image is missing. With reference to the group data and the frame of the depth moving image at the same time as the missing frame, the missing frame of the color moving image is generated by interpolation to obtain the pixel value of the missing frame.

  One aspect of the present invention is the moving image frame interpolating apparatus, wherein either the color moving image or the depth moving image in which frames at the same time of both the color moving image and the depth moving image are missing is provided. And further comprising a frame interpolating unit for interpolating the missing frame, wherein the interpolating unit corresponds to each of the frames before and after the missing frame when the moving image interpolated by the frame interpolating unit is a color moving image. Referring to the three-dimensional point cloud data and the frame of the color moving image obtained by the interpolation at the same time as the missing frame, missing by interpolation to obtain the pixel value of the frame of the missing depth moving image When the moving image generated by generating the frame and interpolated by the frame interpolating means is a depth moving image, the missing frame is detected. Referring to the 3D point cloud data corresponding to each of the frames before and after the frame and the frame of the depth moving image obtained by the interpolation at the same time as the missing frame, the frame of the missing color moving image The missing frame is generated by interpolation for obtaining pixel values.

  One aspect of the present invention is the moving image frame interpolating apparatus, wherein the interpolating unit smoothes temporal or spatial changes in the movement of the imaging target in the color moving image or the depth moving image. Interpolate as follows.

  One embodiment of the present invention is performed by a moving image frame interpolation apparatus that generates a missing frame by interpolation between a color moving image expressed by luminance information of a shooting target and a depth moving image expressed by depth information of the shooting target. A moving image frame interpolation method, wherein the color moving image and the depth moving image are in frame synchronization, and if one of the two moving images lacks a frame, the other moving image This is a moving picture frame interpolation method including an interpolation step of interpolating a missing frame based on frame information.

  One aspect of the present invention is a moving image frame interpolation program for causing a computer to function as the moving image frame interpolation device.

  According to the present invention, using a color moving image and a depth moving image composed of synchronized frames, when a frame of one moving image is missing, interpolation is performed using information of the other moving image. Therefore, it is possible to obtain an effect that high-quality moving image frame interpolation becomes possible.

It is a block diagram which shows the structure of the moving image frame interpolation apparatus by one Embodiment of this invention. 3 is a flowchart showing an operation of the moving image frame interpolation apparatus 1 shown in FIG. 1. It is explanatory drawing which shows the process (process which assigns a serial number and performs time synchronization) of step S2 shown in FIG. It is explanatory drawing which shows the process (process which assigns a serial number and performs time synchronization) of step S2 shown in FIG. It is explanatory drawing which shows the process (process which assigns a serial number and performs time synchronization) of step S2 shown in FIG. It is explanatory drawing which shows the process which integrates frame data as a three-dimensional point group. It is explanatory drawing which shows the process which matches each pixel between the frames of a time direction. It is explanatory drawing which shows the process which performs the interpolation to a color moving image using depth information.

  A moving picture frame interpolation apparatus according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral 1 denotes a moving image frame interpolating device. Reference numeral 11 denotes a color moving image acquisition unit that acquires a color moving image from the outside. Reference numeral 12 denotes a depth moving image acquisition unit that acquires a depth moving image from the outside. Reference numeral 13 gives a serial number to each frame to the color moving image output from the color moving image acquisition unit 11 and the depth moving image output from the depth moving image acquisition unit 12, and 2 It is a frame synchronization unit that synchronizes the time of two moving images. Reference numeral 14 denotes information (hereinafter referred to as frame data) of each frame of two moving images (color moving image and depth moving image) output from the frame synchronization unit 13, that is, each frame if it is a color moving image. This is a three-dimensional point grouping unit that integrates the depth information of each frame as a three-dimensional point group in the case of color information and depth moving images. Reference numeral 15 denotes an inter-frame association unit that performs inter-frame association of a three-dimensional point group that is output from the three-dimensional point grouping unit 14. Reference numeral 16 denotes a frame of a color moving image using depth information based on the depth moving image, which is an output of the inter-frame associating unit 15 and which is input with a moving image associated with each pixel between frames in the time direction. This is a moving image interpolating unit that performs interpolation processing of frames of depth moving images using color moving images. Reference numeral 17 denotes an interpolated moving image output unit that outputs the frame of the moving image interpolated by the moving image interpolating unit 16 to the outside.

  Next, the operation of the moving picture frame interpolation apparatus 1 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the moving image frame interpolation apparatus 1 shown in FIG. First, each of the color moving image acquisition unit 11 and the depth moving image acquisition unit 12 acquires a color moving image and a depth moving image from the outside (step S1).

  The color moving image and the depth moving image can be acquired by any method as long as both moving images target the same scene. For example, data stored in a storage device may be used, or may be acquired using a device such as a camera equipped with a sensor capable of acquiring depth. When a device such as a camera is used, a color moving image and a depth moving image may be acquired by a single device, or a color moving image and a depth moving image may be acquired by separate devices. In the color moving image and the depth moving image acquired here, the frame acquisition time of each frame is also given.

  Next, the frame synchronization unit 13 assigns serial numbers of each frame and performs time synchronization (step S2). The frame synchronization unit 13 performs time synchronization and serial number assignment on the color moving image and the depth moving image acquired by the color moving image acquisition unit 11 and the depth moving image acquisition unit 12. The processing operation of step S2 will be described with reference to FIG. 3, FIG. 4, and FIG. 3 to 5 are explanatory diagrams showing the processing of step S2 shown in FIG. 2 (processing for assigning serial numbers and time synchronization). If both moving images are synchronized in time in advance, the first frame is obtained for each of the acquired color moving images and depth moving images without the time lag and the alignment between frames in the following procedure. Consecutive frame numbers are assigned in order.

  The frame synchronization unit 13 performs the process of this step on the color image and the depth moving image acquired in step S1, thereby correcting the correct time series order of each frame in each moving image and the correspondence between the frames of both moving images. Information can be acquired. In addition, the interval between frames of both moving images is also corrected to a uniform value according to the shooting speed of the device. The specific procedure is shown below.

Of the acquired color moving image and depth moving image as a whole, the moving image with the first frame acquisition time being the moving image Y is the moving image Y and the other moving image is the moving image N. In addition, when the acquisition time of the first frame is the same, either may be used as the moving image Y. In the moving image Y, the frame number 0 of the frame existing at the earliest position in time is given (see FIG. 3). Here, already the maximum frame number of the frame of the moving image Y imparted with the frame number n, the frame acquisition time and T _n. Here, assuming that the number of frames acquired per second of the device is f [fps], A is defined by equation (1).
A = 1 / f + ε (1)
Here, ε is arbitrarily given a minute value in consideration of a delay of a transmission line or the like.

When a frame exists within the time [T _n, T _n + A] of the moving image Y, a frame number n + 1 is assigned to the frame. If there is no frame, an empty frame is arranged and frame number n + 1 is assigned. Since this empty frame does not have a frame acquisition time, a frame acquisition time T _{n + 1} is temporarily assigned. T _{n + 1} is expressed by the following equation (2). The above processing is performed up to the end of the moving image Y.
T _{n + 1} = T _n + 1 / f (2)

Thereafter, the same processing as described above is performed on the moving image N (see FIG. 4). A frame is searched at time [T ₀ −1 / f, T ₀ + 1 / f] of the moving image N, and when a frame exists, frame number 0 is assigned to the frame. If there is no frame, an empty frame is arranged, and frame number 0 and T ₀ ′ (= T ₀ ) are assigned as the frame acquisition time. Here, it is assumed that the largest frame number among the frames of the moving image N to which a frame number has already been assigned is m and the frame acquisition time is T _m ′.

When a frame exists within the time [T _m ′, T _m ′ + A] of the moving image N, a frame number m + 1 is assigned to the frame. If there is no frame, an empty frame is arranged and a frame number m + 1 is assigned. Since this empty frame does not have a frame acquisition time, a frame acquisition time T _{m + 1} ′ is temporarily assigned. T _{m + 1} ′ is expressed by the following equation (3). The above processing is performed up to the end of the moving image N.
T _{m + 1} '= T _m ' + 1 / f (3)

The maximum frame number assigned to each frame of the moving image Y and the moving image N is the same, and this is defined as L (see FIG. 5). The following equations (4) to (6) are applied to all frames, and the frame acquisition times of the moving image Y and the moving image N are updated.
T ₀ = 0 (4)
T _n = T _n−1 + 1 / f (1 ≦ n ≦ L) (5)
T _m ′ = T _m (0 ≦ m ≦ L) (6)

  Next, the three-dimensional point grouping unit 14 is time-synchronized, which is an output of the frame synchronization unit 13, and receives the color moving image and the depth moving image to which serial numbers are assigned, and inputs each of the moving image. Frame information (hereinafter referred to as frame data), that is, color information of each frame in the case of a color moving image, and depth information of each frame in the case of a depth moving image are integrated as a three-dimensional point group (step S3). Here, a process of integrating both frame data as a three-dimensional point group will be described with reference to FIG. FIG. 6 is an explanatory diagram showing processing for integrating frame data as a three-dimensional point group. Hereinafter, one frame of a color moving image is referred to as a color image, and one frame of a depth moving image is referred to as a depth image.

  Each pixel of the color image has color information to be photographed. Each pixel of the depth image has depth distance information (depth information) to be imaged. By integrating depth information and color information of pixels with the same pixel position for color images and depth images having the same frame number, each pixel of the depth image has color information in a three-dimensional space. A point is generated, and a three-dimensional point group with color information corresponding to each frame number is generated. Note that a three-dimensional point group without color information (hereinafter referred to as a three-dimensional point group without color information) is generated for a frame number with an empty color image.

  This process is performed on the assumption that the color image and the depth image have the same spatial resolution. However, if the spatial resolutions of the two images are different, the same processing can be performed by up-sampling an image with a low resolution in advance or down-sampling an image with a high resolution so as to have the same spatial resolution. Any filter may be used for up-sampling and down-sampling.

  Next, the inter-frame association unit 15 receives the 3D point group output from the 3D point grouping unit 14 as input, and performs inter-frame association of the 3D point group (step S4). The inter-frame association unit 15 associates each pixel between frames in the time direction for the three-dimensional point group with color information acquired in step S3. Here, processing for associating each pixel between frames in the time direction will be described with reference to FIG. FIG. 7 is an explanatory diagram illustrating a process of associating each pixel between frames in the time direction.

  The association between the frames of the three-dimensional point group with color information is performed by detecting the optical flow using the color information and depth information of each pixel (see FIG. 7). This association processing is performed on all frames having a 3D point cloud with color information. However, a frame having a 3D point cloud with color information is compared with a frame S having 3D point cloud with color information. Of these, the frame number is larger than the frame number of the frame S and the difference is the smallest.

In the conventional optical flow for moving images (see Reference 1), only the motion vector projected on the image plane can be expressed. However, by using the depth information as in the processing of this step, the motion vector in three dimensions Can be detected, and a more accurate optical flow can be detected.
Reference 1: “BD Lucas and T. Kanade,“ An Iterative Image Registration Technique with an Application to Stereo Vision, ”in Proc. Of Int. Joint Conf. On Artificial Intelligence, pp.674-679, Aug.

  Next, the moving image interpolating unit 16 receives the moving image in which each pixel is associated between the frames in the time direction, which is an output of the inter-frame associating unit 15, and uses the depth information to generate a color moving image. Interpolation processing for each frame is performed (step S5). Then, the interpolated moving image obtained by the interpolation is output to the outside (step S6). Here, processing for performing interpolation on a color moving image using depth information will be described with reference to FIG. FIG. 8 is an explanatory diagram illustrating processing for performing interpolation on a color moving image using depth information.

  This processing is performed on a frame having a three-dimensional point group without color information. In the following description, the frame to be processed is the frame T, the frame number is larger than the frame number of the frame T, and the three-dimensional point group with color information having the smallest difference between the frame numbers of the frame and the frame T is represented. A frame having a three-dimensional point group with color information whose frame number is smaller than the frame number of the frame T and whose frame number difference between the frame and the frame T is the smallest is a frame P.

  First, the estimated three-dimensional point group in the frame T and the correspondence relationship between the estimated three-dimensional point group and the three-dimensional point groups of the frames F and P are obtained. That is, as a result of the step S4 (inter-frame association), the correspondence relationship between the three-dimensional point group of the frame F and the three-dimensional point group of the frame P is obtained. Assuming that it has moved to the corresponding three-dimensional point, a three-dimensional point group in the frame T existing between them is estimated, and a correspondence relationship of the three-dimensional points is given.

  A vector representing a moving direction and a moving distance when a three-dimensional point of frame P moves to a corresponding three-dimensional point of frame F is defined as a motion vector, and a set of motion vectors at a plurality of points. Is defined as a motion vector field. At this time, the motion of each three-dimensional point indicated by the motion vector field from the frame P to the frame F may be regarded as a constant velocity linear motion or a constant acceleration motion. If the result of step S4 indicates that the frame P is further correlated with the past frame, the velocity vector and acceleration vector of each three-dimensional point in the frame P are calculated and taken into account. Thus, the movement of a three-dimensional point between the frame P and the frame F may be assumed.

  On the other hand, as a result of step S4, when the correspondence relationship with the future frame is given to the frame F, the velocity vector and the acceleration vector in the frame F are calculated, and the frames P and You may assume the motion of the three-dimensional point between the frames F.

  Next, the moving image interpolation unit 16 gives color information to the estimated three-dimensional point group. This process is performed by determining the color information of the estimated three-dimensional point using the color information of the three-dimensional point with color information of the frame P and the frame F associated with each estimated three-dimensional point. Any method may be used. For example, one of the color information of the three-dimensional points with color information of the associated frames P and F may be used as the color information of the estimated three-dimensional points. Absent. In addition, the color information including the average value of the color information of the three-dimensional points with color information of the associated frames P and F and the color information according to the time intervals of the frames T, P, and F are weighted. Color information generated by appending and adding may be given.

  When color information is given to the estimated three-dimensional point group, color information is given to the three-dimensional point group without color information by associating the three-dimensional point group without color information of the frame T with the estimated three-dimensional point group. Any process may be used for associating the three-dimensional point group without color information with the estimated three-dimensional point group. For example, the association may be obtained so that the sum of the difference in distance between the associated three-dimensional points is minimized. As another method, the estimated three-dimensional point may be associated with each other so that the difference in distance is minimized for each three-dimensional point having no color information. In this case, one estimated three-dimensional point may correspond to a plurality of color information-free three-dimensional points. When the correspondence between the color information-free 3D point group and the estimated 3D point group is obtained, the color information of the estimated 3D point is set as the color information of the color information-free 3D point based on the correspondence.

  When color information is given to the three-dimensional point group of the frame T, the color image of the frame T is obtained by projecting the three-dimensional point group onto the image plane, and the frame interpolation of the color moving image is realized.

  In the above description, one estimated three-dimensional point is associated with one three-dimensional point without color information, but a plurality of estimated three-dimensional points may be associated with one three-dimensional point without color information. In that case, an average value or median value of color information of a plurality of estimated three-dimensional points associated with each other can be given as color information of the three-dimensional point without color information. Further, weighting may be performed in consideration of a distance between three-dimensional points instead of a simple average value or median value.

  In the above description, after obtaining the color information of the estimated three-dimensional point, the color information of the three-dimensional point without color information is obtained. However, the color information of the estimated three-dimensional point is not obtained, but based on the correspondence relationship. Thus, the color information of the three-dimensional point without color information may be obtained directly from the three-dimensional point with color information of the frame P and the frame F. In that case, any processing may be performed. For example, the velocity vector and the acceleration vector in the frame P and the frame F are calculated, and the three-dimensional point with color information in the frame T is assumed by considering the motion of the three-dimensional point between the frame P and the frame F. You may ask for. For example, when the difference between the acceleration vectors in the frame P and the frame F is equal to or smaller than a certain value, it is considered that the three-dimensional point is performing an equal acceleration motion between the frame P and the frame F, and the frame T You may obtain | require the three-dimensional point with color information in.

  Further, the three-dimensional point with color information in the frame T may be obtained in consideration of the natural change of the temporal motion vector and the natural change of the spatial motion vector. In that case, any processing may be performed.

  Here, the natural change of the temporal or spatial motion vector represents the degree to which the motion of the object indicated by the change of the motion vector can occur in the real space. For example, the naturalness of changes in temporal or spatial motion vectors can be represented by the smoothness of temporal or spatial changes in motion vectors. This is based on the fact that the motion of an object does not change rapidly in real space, and that points on the same object follow the same motion. In order to consider the natural change of the spatial motion vector, the 3D point with color information in the frame T obtained by assuming the motion of the 3D point between the frame P and the frame F, and the frame The 3D point with color information in the frame T may be obtained by adding as a restriction that the movement vectors of the corresponding points do not intersect with the 3D point with color information in P or the frame F.

  Further, in order to consider the natural change of the motion vector over time, the three-dimensional point with color information and the frame P in the frame T obtained by assuming the motion of the three-dimensional point between the frame P and the frame F. 3D with color information in the frame T obtained by assuming the magnitude of the movement vector between the corresponding points between the 3D points with color information in the image and the motion of the 3D points between the frames P and F. By adding a condition that the difference between the magnitude of the movement vector between the corresponding points between the point and the three-dimensional point with color information in the frame F does not become a certain value or more, the color information in the frame T is added. A three-dimensional point may be obtained.

  Next, details of the interpolation processing to the depth moving image using the color information will be described below. The processing here is performed on a frame in which only color information exists and does not have depth information. In the following description, the frame to be processed is the frame T, the frame number is larger than the frame number of the frame T, and the difference between the frame numbers of the frame and the frame T is the smallest 3D point group with color information. A frame is a frame F, and a frame having a three-dimensional point group with color information in which the frame number is smaller than the frame number of the frame T and the difference between the frame numbers of the frame and the frame T is the smallest.

  First, each pixel in the frame T is associated with each pixel in the frame F and the frame P using an optical flow or the like. Any processing may be used for associating each pixel. Thereby, the movement vector of each pixel between the frame P and the frame T on the image plane and the movement vector of each pixel between the frame T and the frame F on the image plane are calculated.

  Since this movement vector has only a component parallel to the image plane, a component in a direction perpendicular to the image plane is derived. In this case, any processing may be used. For example, assuming that a certain region has the same motion vector field, the pixel area occupied by the region in the frame P and the frame T is obtained, and from the difference between the pixel area of the region in the frame P and the pixel area of the region in the frame T The component in the direction perpendicular to the image plane of the motion vector field in the region between the frame P and the frame T may be derived. Further, the component in the direction perpendicular to the image plane of the motion vector field in the region between the frame P and the frame T may be derived from the difference in the degree of blur in the region.

  When depth information is given to the three-dimensional point group of the frame T, the depth image of the frame T is obtained by projecting the three-dimensional point group onto the image plane, and frame interpolation of the depth moving image is realized.

  In addition, when neither a color image nor a depth image exists in the frame to be interpolated, after interpolating one of the images using a known frame interpolation technique, the above-described color information using the depth information is converted. Interpolation processing for depth moving images using color information. At this time, the interpolation using the conventional frame interpolation technique may be performed on either the color image or the depth image.

  Finally, the interpolation moving image output unit 17 outputs the moving image interpolated by the moving image interpolation unit 16 as a still image or a moving image file for each frame. Either of these output methods may be used.

  As described above, it is possible to perform interpolation between moving image frames with higher quality by using depth information in the scene for video with missing color data (color image) of a frame at a certain time. It becomes. Further, by using a color image in a scene for a video in which a depth image of a frame at a certain time is missing, it is possible to perform interpolation between moving image frames with higher quality.

  According to this configuration, the motion vector field between frames can be obtained by using the color information of each pixel that can be acquired from the color data and the information in the depth direction of the depth information in addition to the information in the two-dimensional direction on the image plane. Can be estimated with higher accuracy, so that high-quality moving image frame interpolation is possible.

  You may make it implement | achieve all or one part of the moving image frame interpolation apparatus 1 in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  It can also be applied to applications where it is essential to interpolate between frames using color and depth images.

  DESCRIPTION OF SYMBOLS 1 ... Moving image frame interpolation apparatus, 11 ... Color moving image acquisition part, 12 ... Depth moving image acquisition part, 13 ... Frame synchronization part, 14 ... Three-dimensional point grouping part, 15 ... Inter-frame association unit, 16 ... moving image interpolation unit, 17 ... interpolated moving image output unit

Claims (8)

A moving image frame interpolation device that generates a missing frame by interpolation between a color moving image expressed by luminance information of a shooting target and a depth moving image expressed by depth information of the shooting target,
When the color moving image and the depth moving image are in frame synchronization, and the frame of one of the two moving images is missing, the frame of the missing frame is determined based on the information of the frame of the other moving image. A moving picture frame interpolating apparatus provided with an interpolation means for performing interpolation. Three-dimensional integration of the color moving image frame and the depth moving image frame by giving depth information of the depth moving image to each pixel of the color moving image to form a three-dimensional point cloud Three-dimensional point grouping means for generating point cloud data corresponding to each frame;
An inter-frame association unit that associates each pixel between frames in the time direction by detecting an optical flow based on the three-dimensional point cloud data;
The moving image frame interpolating apparatus according to claim 1, wherein the interpolating unit interpolates the missing frame based on a result of associating each pixel between frames in the time direction. The interpolation means includes
When a frame of the depth moving image is missing, refer to the three-dimensional point cloud data corresponding to each frame before and after the missing frame and the frame of the color moving image at the same time as the missing frame. The moving image frame interpolating apparatus according to claim 2, wherein a frame of the depth moving image is generated by interpolation to obtain a pixel value of the missing frame. The interpolation means includes
When the frame of the color moving image is missing, refer to the three-dimensional point cloud data corresponding to the frames before and after the missing frame and the frame of the depth moving image at the same time as the missing frame. The moving image frame interpolating apparatus according to claim 2, wherein a frame of the color moving image is generated by interpolation to obtain a pixel value of the missing frame. Frame interpolation means for interpolating the missing frame in either the color moving image or the depth moving image in which the same time frame of both the color moving image and the depth moving image is missing,
The interpolation means includes
When the moving image interpolated by the frame interpolating means is a color moving image, the three-dimensional point cloud data corresponding to the frames before and after the missing frame and the interpolation at the same time as the missing frame are obtained. Generating the missing frame by interpolation to obtain a pixel value of the missing depth moving image frame with reference to the color moving image frame;
When the moving image interpolated by the frame interpolating means is a depth moving image, the three-dimensional point cloud data corresponding to the frames before and after the missing frame and the interpolation at the same time as the missing frame are obtained. The moving image frame interpolating apparatus according to claim 2, wherein the missing frame is generated by interpolation to obtain a pixel value of the missing color moving image frame with reference to the depth moving image frame.   The said interpolation means performs interpolation so that the temporal or spatial change of the motion of the said imaging | photography object in the said color moving image or the said depth moving image may become smooth. Video frame interpolator. This is a moving image frame interpolation method performed by a moving image frame interpolation apparatus that generates a missing frame by interpolation between a color moving image expressed by luminance information of a shooting target and a depth moving image expressed by depth information of the shooting target. And
When the color moving image and the depth moving image are in frame synchronization, and the frame of one of the two moving images is missing, the frame of the missing frame is determined based on the information of the frame of the other moving image. A moving image frame interpolation method including an interpolation step for performing interpolation.   A moving picture frame interpolation program for causing a computer to function as the moving picture frame interpolation apparatus according to any one of claims 1 to 6.

Images