JP6534964B2 - Video editing method, video editing apparatus and video editing program - Google Patents

Description

  The present invention relates to a video editing method, a video editing apparatus, and a video editing program.

  It is known that an object (subject) can be tracked and estimated in each frame of a plurality of cameras to generate a good free viewpoint video (see, for example, Non-Patent Document 1). In addition, it is known that, when occlusion occurs in each frame of a plurality of cameras, the position can be predicted by interpolation in time space to generate a good free viewpoint video. (For example, refer to nonpatent literature 2). There is also known a system in which a part of the video is scaled up and down so that the viewer can select a part to be viewed (see, for example, Non-Patent Document 3).

  By the way, a normal video is one in which time-varying images are continuously reproduced. On the other hand, it is performed to generate a reproduced video by continuously making images of which the viewpoint is changed between frames. Such an image is called a wraparound image. In the wraparound video, images of cameras taken from multiple viewpoints can be used to generate a video that looks as if one camera wraps around an object.

"Free-viewpoint video generation of football by extraction and tracking of player area", Hiroki Sango et al, Journal of the Institute of Image Information and Television Engineers Vol. 68, No. 3, p 125-134, 2014 "Free-viewpoint image generation based on object separation and inter-camera / inter-frame prediction" Kentaro Yamada et al., Transactions of the Institute of Electronics, Information and Communication Engineers D, Vol. J 97-D, No. 2, pp. 303-315, 2014.02.01 “Interactive and panoramic video distribution technology that allows you to freely choose what you want to see,” Business Communication, Vol. 48 No. 4 Page. 41-43 (2011.04.01)

  However, in general object position estimation in Non-Patent Document 1 and Non-Patent Document 2, if the scene / pattern is not similar to the past, learning will not work well and position estimation accuracy will be low. As a result, fluctuations due to the position error occur largely. In particular, in the case of a violent motion or occlusion or body pose which is a scene / pattern outside the assumed condition, the learning does not work well and the position estimation accuracy becomes low, resulting in a large fluctuation. When the fluctuation occurs, the continuity of the wraparound image is lost and the image becomes difficult to view. Here, the continuity of the wraparound video means that switching of the wraparound video with the viewpoint changed is performed smoothly.

  In order to solve the above-mentioned problems, the present invention provides a video editing method and a video editing apparatus capable of generating a video having continuity of a wraparound video favorably even in a situation where fluctuation occurs in a position estimation value. And providing a video editing program.

  One aspect of the present invention is an estimation step of estimating a screen position and an image size by object tracking from each of a plurality of input images, and calculating a fluctuation of the estimated value of the screen position and the estimated value of the image size by a statistical method Performing a fluctuation calculation step, determining a screen position and an image size based on the fluctuation of the estimated value, a first viewpoint video at a viewpoint change start time, a close viewpoint video at a viewpoint change end time, and the first viewpoint According to another aspect of the present invention, there is provided a video editing method including a video generation step of generating a wraparound video based on a viewpoint change video between the video and the final viewpoint video, and the determined screen position and image size.

  One aspect of the present invention is the video editing method, wherein in the fluctuation calculation step, a filter operation is performed on the estimated value, and a statistical operation is performed based on a difference between the estimated value and the estimated value subjected to the filter operation. The fluctuation of the estimated value is calculated by the method.

  One aspect of the present invention is the video editing method, wherein in the determination step, when the fluctuation of the estimated value is small, an estimated position estimated by object tracking is determined as the screen position, and is determined in advance. The standard size is determined as the image size.

  One embodiment of the present invention is the video editing method, wherein, in the determination step, when the fluctuation of the estimated value is large, the estimated position subjected to the filter operation is determined as the screen position to alleviate the screen fluctuation. The image size is determined based on the image size.

  According to one aspect of the present invention, there is provided an estimation unit for estimating a screen position and an image size by object tracking from each of a plurality of input images, and calculating a fluctuation of the estimated value of the screen position and the estimated value of the image size by a statistical method. A fluctuation calculation unit, a determination unit that determines a screen position and an image size based on the fluctuation of the estimated value, a first viewpoint video at a viewpoint change start time, a close viewpoint video at a viewpoint change end time, and the first viewpoint The video editing apparatus includes a video generation unit that generates a wraparound video based on a viewpoint change video between the video and the final viewpoint video, and the determined screen position and image size.

  One aspect of the present invention is a video editing program for causing a computer to execute the video editing method.

  According to the present invention, it is possible to generate an image having continuity of a wraparound image even in a situation in which fluctuation of an estimated value of an object position occurs, so that an easily viewable screen can be generated.

BRIEF DESCRIPTION OF THE DRAWINGS It is explanatory drawing which shows the outline | summary of the video editing apparatus based on the 1st Embodiment of this invention. FIG. 1 is a block diagram showing a configuration of a video editing apparatus according to a first embodiment of the present invention. It is explanatory drawing of a process of the object tracking part in the video editing apparatus concerning the 1st Embodiment of this invention. It is a graph which shows the change of the estimated value of the estimated position of the horizontal direction, the estimated position of the orthogonal | vertical direction, and an image size. It is a flowchart for demonstrating the process of the tracking statistics part in the video editing apparatus concerning the 1st Embodiment of this invention. It is explanatory drawing of the information sent to a edit process judgment part from a tracking statistics part in the video editing apparatus concerning the 1st Embodiment of this invention. It is explanatory drawing of the drawing size of each viewpoint in the video editing apparatus concerning the 1st Embodiment of this invention. It is a flowchart for demonstrating the process of the edit process judgment part in the video editing apparatus concerning the 1st Embodiment of this invention. FIG. 7 is an explanatory diagram of information sent from the editing process determination unit to the video generation unit in the video editing device according to the first embodiment of the present invention. It is a flowchart for demonstrating the process of the imaging | video production | generation part in the video editing apparatus which concerns on the 1st Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, an outline of a video editing apparatus 1 according to an embodiment of the present invention will be described.

  FIG. 1 is an explanatory view showing an outline of a video editing apparatus 1 according to an embodiment of the present invention. As shown in FIG. 1A, an image of each viewpoint is input to the image editing apparatus 1 from N (N is a natural number) cameras. The input images 1 to N input to the video editing apparatus 1 are images captured by cameras “1” to “N” of different viewpoints. As shown in FIG. 1B, the input images 1 to N are shot at wide angles. The image editing apparatus 1 estimates the horizontal position and the vertical position of the input images 1 to N and the image size (the length of the diagonal of the screen) by object (subject) tracking. Then, the video editing apparatus 1 performs a filter operation on the estimated position in the horizontal direction and the estimated position in the vertical direction of the input images 1 to N and the estimated value of the image size, and the horizontal direction of the input images 1 to N The estimated position and the estimated position in the vertical direction, and the estimated value of the image size, the fluctuation, and the coefficient for the editing determination are obtained. Note that the coefficient for editing determination is set based on the fluctuation of the estimated value. Then, the video editing apparatus 1 performs editing determination based on the coefficient for editing determination, and determines the horizontal position, the vertical position, and the value of the image size. Then, the video editing apparatus 1 has an input composed of a first viewpoint video (video at the start of viewpoint change), a last viewpoint video (video at the end of viewpoint change), and a viewpoint change video between the first viewpoint video and the last viewpoint video. A wraparound image is generated based on the images 1 to N, the determined position in the horizontal direction and the determined position in the vertical direction, and the determined values of the image size. The video editing apparatus 1 generates a tracking video of the first viewpoint, a tracking wraparound video, and a tracking video of the last viewpoint as shown in FIG. 1 (C). Then, the video editing device 1 combines these and displays the wraparound video of the object of interest on the display device 2.

  Next, the video editing apparatus 1 shown in FIG. 1 will be described in detail. FIG. 2 is a block diagram showing the configuration of the video editing apparatus 1 shown in FIG. As shown in FIG. 2, the video editing apparatus 1 shown in FIG. 1 includes an IF (Interface) unit 101, a video storage unit 102, an object tracking unit 103, a tracking statistics unit 104, and an editing processing determination unit 105. The image generation unit 106 and the IF unit 107 are provided.

  The IF unit 101 inputs the input images 1 to N of a plurality of cameras “N” (N is a natural number) in time synchronization, and sends the input images 1 to N to the video storage unit 102. In addition to the camera input video, the input video may be an N viewpoint file input video.

  After temporarily storing the input images 1 to N input from the IF unit 101, the video storage unit 102 sends the input images 1 to N to the object tracking unit 103. Also, the video storage unit 102 sends, to the video generation unit 106, the input images 1 to N of the viewpoint-changed video between the first viewpoint video, the last viewpoint video, the first viewpoint video, and the last viewpoint video.

  The object tracking unit 103 performs object tracking to estimate the horizontal position and the vertical position of the input images 1 to N and the image size. FIG. 3 is an explanatory diagram of object tracking performed by the object tracking unit 103. As shown in FIG. 3, the image of each viewpoint is shot at a wide angle, and the area indicated by A1 is a wide angle shooting area. The standard size (default value) is the area indicated by the area A2. An area indicated by A3 is a tracking screen area. As shown in FIG. 3, the object tracking unit 103 moves the screen position in the horizontal and vertical directions so that the object B1 is positioned at the center of the screen while keeping the aspect ratio of the screen constant. Then, the object tracking unit 103 sends the estimated values of the estimated position in the horizontal direction and the estimated position in the vertical direction and the image size to the tracking statistics unit 104. Here, when making the display screen wide-angle, the aspect ratio of the screen does not change.

  In FIG. 2, the tracking statistics unit 104 is a value based on the fluctuation of the estimated value by the statistical method from the estimated position in the horizontal direction and the estimated position in the vertical direction estimated by the object tracking unit 103 and the estimated value of the image size. Calculate The tracking statistics unit 104 first performs a filter operation on the estimated position in the horizontal direction and the estimated position in the vertical direction estimated by the object tracking unit 103 and the estimated value of the image size. Then, the tracking statistics unit 104 calculates the estimated position in the horizontal direction and the estimated position in the vertical direction estimated by the object tracking unit 103, the estimated value of the image size, the estimated position in the horizontal direction subjected to the filter operation, and the vertical direction obtained by the filter operation. The fluctuation of the estimated value is calculated from the estimated position of the image and the estimated value of the filtered image size. Then, the tracking statistics unit 104 calculates the instantaneous inter-camera distance at the viewpoint change time. Subsequently, the tracking statistics unit 104 calculates a coefficient for editing determination from the fluctuation of the estimated value, the instantaneous inter-camera distance at the viewpoint change time, and the image size at the viewpoint change time. Then, the tracking statistics unit 104 calculates the estimated position in the horizontal direction and the estimated position in the vertical direction, the estimated value of the image size, the estimated position in the horizontal direction subjected to the filter operation, the estimated position in the vertical direction subjected to the filter operation, and the filter operation. The estimated value of the image size, the fluctuation, and the coefficient for the editing determination are sent to the editing processing determination unit 105.

  The editing process determination unit 105 determines the horizontal position and the vertical position, and the value of the image size, based on the coefficient for the editing determination obtained by the tracking statistics unit 104. When the coefficient for editing determination is smaller than the threshold value, the editing process determining unit 105 determines the estimated value in the horizontal direction and the estimated position in the vertical direction estimated by the object tracking unit 103 as the position in the horizontal direction and the position in the vertical direction. Decide as. Further, in this case, the editing process determination unit 105 determines the image size as the standard size (default value). On the other hand, when the coefficient for editing determination is larger than the threshold value, the editing processing determination unit 105 obtains an image size that reduces the screen fluctuation. Then, the editing process determination unit 105 determines the estimated position in the horizontal direction subjected to the filter operation and the estimated position obtained in the vertical direction subjected to the filter operation as the position in the horizontal direction and the position in the vertical direction. Also, the image size is determined using the image size that reduces the screen fluctuation. Then, the editing process determination unit 105 sends the determined position in the horizontal direction, the determined position in the vertical direction, and the determined value of the image size to the image generation unit 106.

  The video generation unit 106 receives input images 1 to N of a first viewpoint video, a last viewpoint video, and a video of viewpoint change time (a viewpoint changed video between the first viewpoint video and the last viewpoint video) input from the video storage unit 102 Based on the determined position in the horizontal direction and the determined position in the vertical direction input from the editing process determination unit 105 and the determined value of the image size, the tracking video of the first viewpoint, the tracking wraparound video, and the tracking video of the last viewpoint Generate Then, the generated video is combined and sent to the IF unit 107 as one video.

  The IF unit 107 outputs a video and a file in response to the input of the wraparound video. This generates a tracking wraparound video.

  Next, processing in the tracking statistics unit 104 in the video editing device 1 shown in FIG. 2 will be described in detail. The tracking statistics unit 104 receives, from the object tracking unit 103, an estimated position in the horizontal direction and an estimated position in the vertical direction, and an estimated value of the image size. The estimated position in the horizontal direction, the estimated position in the vertical direction, and the estimated value of the image size include fluctuations that fluctuate with respect to time.

  FIG. 4 shows changes in the estimated position in the horizontal direction, the estimated position in the vertical direction, and the estimated value of the image size. 4A to 4C, the horizontal axis indicates time, and the vertical axis indicates an estimated value in pixel units. FIG. 4A shows the fluctuation of the estimated position x (t) in the horizontal direction with respect to time t. FIG. 4B shows the fluctuation of the estimated position y (t) in the vertical direction with respect to time t. FIG. 4C shows the variation of the estimated value S (t) of the image size with respect to time t. From the graphs shown in FIGS. 4A to 4C, the values of the estimated position x (t) in the horizontal direction, the estimated position y (t) in the vertical direction, and the estimated value S (t) of the image size It can be seen that fluctuations that fluctuate with time are included.

  The tracking statistics unit 104 shows equations (1) to (3) for the estimated position x (t) in the horizontal direction and the estimated position y (t) in the vertical direction, and the estimated value S (t) of the image size. Perform a filter operation like this: fx (t) is the estimated position in the horizontal direction after the filter operation, fy (t) is the estimated position in the vertical direction after the filter operation, and fS (t) is the estimated position of the image size after the filter operation. Here, t is a time value, h is a filter variable, and k is a frame number. At each viewpoint, fx (t), fy (t) and fS (t) are calculated.

  A filter operation as shown in equations (1) to (3) is performed on the estimated position x (t) in the horizontal direction, the estimated position y (t) in the vertical direction, and the estimated value S (t) of the image size. . As shown in FIG. 4A to FIG. 4C, the estimated values are smoothed and the fluctuation component is removed at the filtered values fx (t), fy (t) and fS (t). .

Next, the tracking statistics unit 104 estimates the estimated position x (t) in the horizontal direction and the estimated position y (t) in the vertical direction, and the estimated value s (t) of the image size and the estimated position fx in the horizontal direction that has been filtered. The fluctuation of the estimated value is calculated from (t) and the estimated position y (t) in the vertical direction subjected to the filter operation and the estimated value fS (t) of the image size subjected to the filter operation. The estimated position in the horizontal direction, the estimated position in the vertical direction, and the fluctuation of the estimated value of the image size can be calculated as shown in equations (4) to (6). In each viewpoint, σ ² X, σ ² Y, and σ ² S are calculated.

The variance σ ² of the fluctuation of the estimated value is obtained from these total averages as shown in equation (7). Calculated from σ ² X, σ ² Y, σ ² S of each viewpoint.

  Next, the tracking statistics unit 104 calculates the instantaneous inter-camera distance at the viewpoint change time. Here, the value of the estimated position in the horizontal direction of the camera n is n_x (t), the value of the estimated position in the vertical direction is n_y (t), and the estimated value of the image size of the camera n is n_S (t). Also, let the value of the estimated position in the horizontal direction filtered by camera n be n_fx (t) and the value of the estimated position in the vertical direction filtered be n_fy (t), and the estimated value of the image size calculated by the filter of camera n It is assumed that n_fS (t).

  The instantaneous inter-camera distance in the horizontal direction at the viewpoint change time t1 at the camera n can be obtained as follows.

  Similarly, the vertical instantaneous inter-camera distance at the viewpoint change time t1 can be obtained as follows.

  Similarly, the instantaneous inter-camera distance of the image size at the viewpoint change time t1 can be obtained as follows.

  Therefore, the instantaneous inter-camera distance at the viewpoint change time of the camera n can be obtained as shown in equation (8).

  The tracking statistics unit 104 obtains the maximum value among the cameras (n = N-1) from the camera (n = 1) as the instantaneous inter-camera distance dc.

  Next, the tracking statistics unit 104 calculates the fluctuation .sigma. Of the estimated value obtained by the equation (7), the instantaneous inter-camera distance dc at the viewpoint change time determined by the equation (9), and the filter at the viewpoint change time From the calculated estimated value fS (t1) of the image size, a coefficient for editing determination is calculated as follows. Here, α is a coefficient.

  Next, the operation of the tracking statistics unit 104 in the video editing apparatus 1 shown in FIG. 2 will be described with reference to a flowchart. FIG. 5 is a flowchart showing the operation of the tracking statistics unit 104.

  The tracking statistics unit 104 inputs the estimated position in the horizontal direction, the estimated position in the vertical direction, and the estimated image size of each viewpoint from the object tracking unit 103 (step S101).

  The tracking statistics unit 104 performs the filter operation shown in the equations (1) to (3) on the estimated position in the horizontal direction, the estimated position in the vertical direction, and the estimated value of the estimated image size for each viewpoint (step S102) .

  The tracking statistics unit 104 calculates the fluctuation of the estimated value by the statistical method by the calculation shown in the equations (4) to (7) (step S103).

  The tracking statistics unit 104 calculates the instantaneous inter-camera distance at the viewpoint change time by the calculation shown in the equations (8) to (9) (step S104).

  The tracking statistics unit 104 calculates a coefficient for editing determination by the calculation shown in equation (10) (step S105).

  The tracking statistics unit 104 estimates the estimated position in the horizontal direction and the estimated position in the vertical direction, the estimated value of the image size, the estimated position in the horizontal direction filtered and the estimated position in the vertical direction filtered, and the image size filtered The estimated value of R, the fluctuation, and the coefficient for editing determination are sent to the editing processing determination unit 105 (step S106).

  FIG. 6 is a diagram showing information sent from the tracking statistics unit 104 to the editing process determination unit 105. As shown in FIG. 6A, from the tracking statistics unit 104 to the editing process determination unit 105, the estimated position x (t) in the horizontal direction, the estimated position y (t) in the vertical direction, and the estimated value S of the image size (T) will be sent. The estimated position x (t) in the horizontal direction, the estimated position y (t) in the vertical direction, and the estimated value S (t) of the image size are estimated by the object tracking unit 103. Further, as shown in FIG. 6B, from the tracking statistics unit 104 to the editing process judging unit 105, the estimated position fx (t) in the horizontal direction subjected to the filter operation and the estimated position fy (t) in the vertical direction subjected to the filter operation. , And a filtered image size estimate fS (t). The estimated position fx (t) in the horizontal direction subjected to the filter operation, the estimated position fy (t) in the vertical direction subjected to the filter operation, and the estimated value fS (t) of the image size subjected to the filter operation can be expressed by equations (1) to (3) Calculated as shown. Further, as shown in FIG. 6C, the tracking statistic unit 104 sends the fluctuation σ and the coefficient bn for the editing judgment to the editing process judging unit 105. The fluctuation σ is calculated as shown in equations (4) to (7). The coefficient bn for editing determination is calculated as shown in equation (10).

  Next, the process in the editing process determination unit 105 in the video editing apparatus 1 shown in FIG. 2 will be described in detail. Information as shown in FIGS. 6A to 6C is input to the editing process determination unit 105. Then, the editing process determination unit 105 determines whether the coefficient bn for editing determination is larger than the coefficient bn. When the coefficient bn for editing determination is smaller than the threshold value, the editing processing determination unit 105 determines the values of the estimated positions x (t) and y (t) in the horizontal and vertical directions to the determined position in the horizontal direction and the vertical direction. Let it be the decision position. Further, the editing process determination unit 105 sets the standard size (see the area A2 shown in FIG. 3) as the determined image size.

  On the other hand, when the coefficient bn for editing determination is larger than the threshold value, the editing processing determination unit 105 obtains an image size that reduces screen fluctuation. The image size for reducing the screen fluctuation can be obtained by calculating the minimum size Snew which satisfies the following condition.

  Then, the editing process determination unit 105 sets the estimated position fx (t) in the horizontal direction subjected to the filter operation and the estimated position fy (t) in the vertical direction subjected to the filter operation as the determined position in the horizontal direction and the determined position in the vertical direction. Further, the image size S of each viewpoint at the time of changing the viewpoint of n is determined as follows.

  FIG. 7 is an explanatory diagram of the image size of each viewpoint. In FIG. 7, the horizontal axis is the camera number, and the vertical axis is the image size. From the graph shown in FIG. 7, it is shown that the reduction ratio changes for the viewpoints of camera numbers “1” to “N”.

  Next, the operation of the editing process determination unit 105 in the video editing apparatus 1 shown in FIG. 2 will be described with reference to a flowchart. FIG. 8 is a flowchart showing the operation of the editing process determination unit 105.

  The editing processing determination unit 105 performs, from the tracking statistics unit 104, the estimated horizontal position x (t) and the estimated vertical position y (t), and the estimated value S (t) of the image size in the filtered horizontal direction. The estimated position fx (t) of the image, the estimated position y (t) in the vertical direction subjected to the filter operation, and the estimated value fS (t) for the filtered image size, the fluctuation .sigma., And the coefficient bn for editing judgment (Step S201).

  The editing process determination unit 105 determines whether the coefficient bn for editing determination is equal to or less than the threshold (bn <threshold) (step S202).

  When the coefficient for editing determination is equal to or less than the threshold (step S202: Yes), the editing process determining unit 105 advances the process to step S203, and when the coefficient for editing determination is not less than the threshold (No at step S202) The process then proceeds to step S204.

  The editing process determination unit 105 determines the horizontal and vertical positions and the image size on the assumption that the fluctuation is good, and advances the process to step S206 (step S203). In this case, the editing process determination unit 105 determines the estimated position in the horizontal direction and the estimated position in the vertical direction (FIG. 6A) as the positions in the horizontal direction and the vertical direction at each viewpoint. The image size of each viewpoint is determined as a standard size (see area A2 in FIG. 3).

  The editing process determination unit 105 obtains the image size Snew for reducing the screen fluctuation so as to satisfy the equation (11), and advances the process to step S205 (step S204).

  The editing process determination unit 105 determines the image size and the position, assuming that the fluctuation is not good, and advances the process to step S206 (step S205). In this case, the positions in the horizontal and vertical directions at each viewpoint are the estimated position in the horizontal direction subjected to the filter operation and the estimated position in the vertical direction subjected to the filter operation (FIG. 6B). The image size of each viewpoint is calculated by the calculations shown in equations (12) and (13).

  The editing process determination unit 105 sends the determined position in the horizontal direction and the determined position in the vertical direction determined in step S203 or step S205 and the determined value of the image size to the image generation unit 106 (step S206).

  FIG. 9 is a diagram showing information sent from the editing process determination unit 105 to the video generation unit 106. As shown in FIG. 9A, the editing processing determination unit 105 to the video generation unit 106 use the start time and the determination position in the horizontal direction and the determination position in the vertical direction as the viewpoint change time as the information of the first viewpoint, And the determined position of the picture size is sent. Further, as shown in FIG. 9B, the editing processing determination unit 105 to the video generation unit 106 determine the determination position in the horizontal direction and the determination position in the vertical direction at the viewpoint change time and the end time as the information on the closing viewpoint. And the determined position of the picture size is sent. Further, as shown in FIG. 9C, the editing processing determination unit 105 to the video generation unit 106 determine the determination position in the horizontal direction and the determination position in the vertical direction at the first viewpoint and the last viewpoint as the information of the viewpoint change time. And the determined position of the picture size is sent.

  Next, the operation of the video generation unit 106 in the video editing apparatus 1 shown in FIG. 2 will be described in detail. The video generation unit 106 receives, from the video storage unit 102, input images 1 to N of the first viewpoint video, the last viewpoint video, and the viewpoint change time. The image generation unit 106 also receives, from the editing process determination unit 105, the determined position in the horizontal direction and the determined position in the vertical direction at each time, and the determined value of the image size. The video generation unit 106 receives input images 1 to N, which are a first viewpoint video, a last viewpoint video, and a video of viewpoint change time (viewpoint modified video between the first viewpoint video and the last viewpoint video), and the determination position and vertical direction in the horizontal direction. Based on the determined position of the direction and the determined value of the image size, the tracking video of the first viewpoint, the tracking wraparound video, and the tracking video of the last viewpoint are generated and combined to form an IF part as one video. Send to 107.

  Note that the method described in Non-Patent Document 3 describes generation of each viewpoint tracking image at viewpoint change time, generation of initial viewpoint tracking video up to the time of viewpoint change, and final viewpoint tracking video generation from the time of viewpoint change. Can be used.

  FIG. 10 is a flowchart showing the operation of the video generation unit 106 in the video editing device 1 shown in FIG.

  The video generation unit 106 receives the initial viewpoint video from the video storage unit 102, and advances the process to step S302 (step S301).

  The image generation unit 106 determines the determination position in the horizontal direction at the start time, the determination position in the vertical direction, the determined image size, and the determination in the horizontal direction at the viewpoint change time The position, the determined position in the vertical direction, and the determined value of the image size are input, and the process proceeds to step S303 (step S302).

  The image generation unit 106 generates a first viewpoint tracking image up to the time of viewpoint change, and advances the process to step S310 (step S303).

  The video generation unit 106 receives the input images 1 to N of the viewpoint change time from the video storage unit 102, and advances the process to step S305 (step S304).

  The image generation unit 106 receives, from the editing process determination unit 105, the determination position in the horizontal direction at the first viewpoint, the determination position in the vertical direction, the determination position of the image size, and the determination of the image size as information of the viewpoint change time. The determination position, the determination position in the vertical direction, and the determination value of the image size are input, and the process proceeds to step S306 (step S305).

  The image generation unit 106 combines the viewpoint tracking images to generate an image, generates an expression image that wraps around the tracking target, and advances the process to step S310 (step S306).

  The video generation unit 106 receives the end-view video from the video storage unit 102, and advances the process to step S308 (step S307).

  The image generation unit 106 determines the determination position in the horizontal direction at the viewpoint change time, the determination position in the vertical direction, the determined image size, and the determination in the horizontal direction at the end time The position, the determined position in the vertical direction, and the determined image size are input, and the process proceeds to step S309 (step S308).

  The image generation unit 106 generates an end-view tracking image from the viewpoint change time, and advances the process to step S310 (step S309).

  The video generation unit 106 uses the first viewpoint tracking video up to the viewpoint change time generated in step S303, the wraparound video of the viewpoint change time generated in step S306, and the viewpoint change time generated in step S309. The end point tracking images of the above are combined, and the process proceeds to step S311 as one image (step S310).

  The video generation unit 106 sends the generated video to the IF 107, and ends the processing (step S311). The video from the IF 107 may be displayed on the display device or may be output as a file.

  As described above, in the video editing apparatus 1 according to the embodiment of the present invention, when the estimated position in the horizontal direction and the estimated position in the vertical direction of the input images 1 to N have large fluctuations in estimated values of the image size, The viewpoint change image is generated based on the estimated position subjected to the filter operation and the tracking size for reducing the screen fluctuation. As a result, it is possible to generate an easy-to-see screen even when generating an image with wraparound continuity in a situation where fluctuations in estimated values occur largely.

  In this example, as described in the reference document, errors and vibrations of an object tracking estimation algorithm using a general image are described. The same method and program can be obtained by adjusting the threshold value and the coefficient bn for editing determination also for an infrared depth camera with higher accuracy of position estimation, a camera using position estimation with a laser in combination, and the like.

Note that a program for realizing all or a part of functions of the video editing apparatus 1 is recorded in a computer readable recording medium, and the program recorded in the recording medium is read into a computer system and executed. The processing of each part may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices.
The "computer system" also includes a homepage providing environment (or display environment) if the WWW system is used.
The term "computer-readable recording medium" refers to a storage medium such as a flexible disk, a magneto-optical disk, a ROM, a portable medium such as a ROM or a CD-ROM, or a hard disk built in a computer system. Furthermore, “computer-readable recording medium” dynamically holds a program for a short time, like a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, the volatile memory in the computer system which is the server or the client in that case, and the one that holds the program for a certain period of time is also included. The program may be for realizing a part of the functions described above, or may be realized in combination with the program already recorded in the computer system.

  Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like within the scope of the present invention are also included.

1: Video editing apparatus, 101: IF unit, 102: Video storage unit, 103: Object tracking unit, 104: Tracking statistics unit, 105: Editing processing determination unit, 106: Video generation unit, 107: IF unit

Claims (6)

Estimating the screen position and the image size from the plurality of input images by object tracking;
A fluctuation calculation step of calculating fluctuation of the estimated value of the screen position and the estimated value of the image size by a statistical method;
A determining step of determining a screen position and an image size based on the fluctuation of the estimated value;
A wraparound based on the first viewpoint video at the start of viewpoint change, the last viewpoint video at the time of viewpoint change end, the viewpoint change video between the first viewpoint video and the last viewpoint video, and the determined screen position and image size And a video generating step of generating a video. In the fluctuation calculation step,
Perform a filter operation on the estimated value,
The video editing method according to claim 1, wherein the fluctuation of the estimated value is calculated by a statistical method based on the difference between the estimated value and the estimated value subjected to the filter operation. In the determination step,
The video editing according to claim 1 or 2, wherein when the fluctuation of the estimated value is small, the estimated position estimated by object tracking is determined as the screen position, and a predetermined standard size is determined as the image size. Method. In the determination step,
4. The video editing method according to claim 3, wherein when the fluctuation of the estimated value is large, the filtered estimated position is determined as the screen position, and the image size is determined based on the image size for reducing the screen fluctuation. . An estimation unit that estimates the screen position and the image size from the plurality of input images by object tracking;
A fluctuation calculation unit that calculates fluctuation of the estimated value of the screen position and the estimated value of the image size by a statistical method;
A determination unit that determines a screen position and an image size based on the fluctuation of the estimated value;
A wraparound based on the first viewpoint video at the start of viewpoint change, the last viewpoint video at the time of viewpoint change end, the viewpoint change video between the first viewpoint video and the last viewpoint video, and the determined screen position and image size And a video generation unit that generates a video.   A video editing program for causing a computer to execute the video editing method according to any one of claims 1 to 4.

Images