JP4971813B2 - Video generation apparatus and video generation program - Google Patents

Description

  The present invention relates to a video generation device and a video generation program, and more particularly, to a video generation device and a video generation program for presenting a subject moving with time as a video with high accuracy.

  In recent years, in movies, commercials, etc., by using images taken synchronously with multiple cameras arranged so as to surround the subject, switching and presenting the image of the moment when the subject jumped in the air according to the camera arrangement, Techniques for displaying a video as if one camera was moving while moving around a subject floating in the air are actively used.

  Such video display technology is referred to as multi-view video display technology, which is applied to, for example, sports broadcasts and the like, and is known as a technology that conveys the flow of a game and the movement of a player using multi-view video (for example, Patent Document 1, (See Non-Patent Document 1, etc.).

On the other hand, by extracting the subject area that moves for each video frame from the video captured by one camera and combining and displaying it on the background image, the movement of the subject over time can be expressed as a single video. Multi-motion video display technology that can be used is also used, and is known as a method for easily explaining the movement of a player in sports broadcasting, for example (see, for example, Patent Document 2, Non-Patent Documents 2, 3, etc.). .
JP 2006-115298 A Kenichi Isa and four others, “Latest sports broadcast technology, the world's first! Utilizing EyeVison ™ (eye vision) in professional baseball broadcasts”, Broadcast Technology, Kenrokukan Publishing, November 2001, p. 96-p. 105. Japanese Unexamined Patent Publication No. 2006-5542 Lee et al., “Gymnastic player form / trajectory display system”, Journal of the Institute of Image Information and Television Engineers, Vol. 51 No. 11 p. 1881-p. 1887 (1997). Kato et al., “Super Multi Motion”, Journal of the Institute of Image Information and Television Engineers, Vol. 57 No. 8 p. 936-p. 938 (2003).

  By the way, the above-described conventional method is a subject display technology using one of a multi-view video display technology and a multi-motion video display technology. However, in order to present a moving subject with time as a video with high accuracy, it is preferable to present a multi-view multi-motion video in which the movement of the subject along the passage of time can be seen from a plurality of viewpoints. There has been no precedent for such video presentation technology.

  Furthermore, in order to generate a multi-motion video without a sense of incongruity, it is necessary to determine the order in which the subject area image extracted for each video frame is combined with a predetermined background image in consideration of the positional relationship between the camera to be photographed and the subject. There are no proposed methods for automatically controlling these.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a video generation apparatus and a video generation program for presenting a subject that moves with time as a video with high accuracy.

  In order to solve the above problems, the present invention employs means for solving the problems having the following characteristics.

  According to the first aspect of the present invention, in a video generation device that generates a predetermined video based on a preset condition from videos of a moving subject photographed by a plurality of imaging means, the content of the video to be output is set. Output video setting means; subject area extraction means for extracting a subject area of a video selected from videos taken by the plurality of imaging means based on setting conditions obtained by the output video setting means; and subject area extraction Image synthesis means for synthesizing the subject area extracted by the means with a predetermined background image, and a video selected from the video captured by the plurality of imaging means based on a setting condition obtained by the output video setting means. Obtained by the multi-view video generation means for generating the multi-view video by switching in the above order, the image synthesis means and the multi-view video generation means. And having an output image generation means for generating an output image from the video.

  According to the first aspect of the present invention, it is possible to present a subject that moves with time as a video with high accuracy.

  The invention described in claim 2 is characterized in that the image synthesizing unit synthesizes a plurality of subject region images extracted by the subject region extracting unit with the background image in a predetermined order.

  According to the second aspect of the present invention, since the images are synthesized according to a predetermined order, a highly accurate synthesized image can be efficiently generated.

  According to a third aspect of the present invention, the image synthesizing unit synthesizes the background image in an order based on distance information from the imaging unit to a subject area obtained by camera parameters in the plurality of imaging units. Features.

  According to the third aspect of the present invention, since the distance to each imaging means and the subject can be obtained with high accuracy, for example, when the subjects overlap on the same image, with high accuracy in the order based on the distance information. A composite image can be generated. Further, the order of synthesis can be automated, and the operability of video generation can be improved.

  The invention described in claim 4 is characterized in that the imaging means obtains only the video from the imaging means for capturing the video used for the output video set by the output video setting means.

  According to the invention described in claim 4, it is possible to reduce the recording capacity without accumulating useless video. In addition, only necessary images can be acquired efficiently.

  The invention described in claim 5 is characterized in that the image synthesizing means generates images having different numbers of subjects to be synthesized with the same image, based on setting conditions obtained by the output video setting means. .

  According to the fifth aspect of the present invention, various multi-view multi-motion images can be generated. This makes it possible to present a novel video that more easily expresses the movement of a subject such as a sports player. Therefore, it is possible to provide a more accurate output video.

  According to a sixth aspect of the present invention, there is provided a video generation program for causing a computer to execute a video generation process for generating a predetermined video based on a preset condition from videos of a moving subject photographed by a plurality of imaging means. , An output video setting step for setting the content of the video to be output, and a subject area of the video selected from the video captured by the plurality of imaging means based on the setting conditions obtained by the output video setting step Based on the setting conditions obtained by the subject region extraction step, the image composition step of combining the subject region extracted by the subject region extraction step with a predetermined background image, and the output video setting step, by the plurality of imaging means A multi-viewpoint video is generated by switching a selected video from a captured video in a predetermined order. And the point image generating step, to perform the output image generation step of generating an output image from said image synthesizing step and video obtained by the multi-view image generation step to the computer.

  According to the sixth aspect of the present invention, it is possible to present a subject that moves with time as a video with high accuracy. In addition, video generation can be easily realized by installing an execution program in a computer.

  According to the present invention, it is possible to present a subject that moves with time as a video with high accuracy.

<Outline of the present invention>
In the present invention, by combining multi-view video display and multi-motion video display, control is performed based on a preset video format to be generated, and the movement of the subject over time is viewed from a plurality of viewpoints. Multi-view multi-motion video that can be presented. This makes it possible to present a novel video that more easily expresses the movement of a subject such as a sports player.

  Further, in the present invention, the distance information from each camera to the moving subject is calculated using images obtained by photographing the subject with a plurality of cameras and the camera parameters of the plurality of cameras, and the order based on the distance information is calculated. Thus, the subject area images of a plurality of video frames extracted for each camera are combined with a predetermined background image. Thereby, for example, when subjects overlap on the same image, a composite image can be generated with high accuracy in the order based on the distance information. Further, the order of synthesis can be automated, and the operability of video generation can be improved.

<Embodiment>
Next, a preferred embodiment of the video generation apparatus and video generation program according to the present invention having the above-described features will be described with reference to the drawings.

<Video generation device: functional configuration example>
First, a functional configuration example of a video generation device according to the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a configuration example of a video generation apparatus according to the present invention.

1 includes an imaging unit 11, an input unit 12, an output unit 13, a storage unit 14, an output video setting unit 15, a subject area extraction unit 16, an image synthesis unit 17, A viewpoint video generation unit 18, an output video generation unit 19, a transmission / reception unit 20, and a control unit 21 are configured.

  The imaging unit 11 captures a subject that performs some operation and acquires an image. The imaging unit 11 stores the acquired video in the storage unit 14. Here, it is assumed that the imaging unit 11 is fixed at a predetermined position using, for example, a plurality of high-definition cameras, commercially available digital video cameras, or the like. When shooting with a plurality of cameras, the movement of the subject is shot from different positions.

  Further, the imaging unit 11 acquires time information at the time of photographing, camera parameters such as the photographing position, photographing direction, and focus adjustment of the camera at the time of photographing.

  Note that the imaging unit 11 may acquire only the video from the imaging unit that captures the video used for the output video based on the setting condition from the output video setting unit 15. Thereby, it is possible to reduce the recording capacity without accumulating useless video. In addition, only necessary images can be acquired efficiently.

  The input means 12 is an instruction to acquire an image of a subject that performs some operation obtained from the imaging means 11, an instruction to read or write predetermined data from the storage means 14, a setting of what kind of video is output, It accepts inputs such as instructions for extracting a subject area from a video, synthesizing a video, and generating an output video. The input unit 12 includes a pointing device such as a keyboard and a mouse, a voice input interface such as a microphone, and the like.

  Further, the output unit 13 includes an executed output video setting result, a subject region extraction result, an image synthesis result, a multi-view video generation result obtained from each component based on the instruction content input by the input unit 12 and the instruction content, Various data such as output video and subject video taken by the image pickup means 11 are displayed and / or output as audio. Note that the output means 13 includes a display, a speaker, and the like.

  The accumulating unit 14 includes a video obtained by the imaging unit 11, an output video setting result obtained by the output video setting unit 15, a subject area extraction result obtained by the subject region extracting unit 16, a composite image obtained by the image synthesizing unit 17, Various data such as multi-view video data obtained by the multi-view video generation means 18 is stored.

  The output video setting unit 15 sets what video is output based on the video obtained by the imaging unit 11. Specifically, based on time information, camera parameters, etc., at which time point (frame) the composite image or multi-viewpoint video is used, the number of subjects to be composited every predetermined time, the setting of the background image to be composited, etc. Do. Thereby, a multi-view multi-motion video can be realized using a plurality of cameras. In other words, since a subject can be seen from various viewpoints using a plurality of cameras, it is possible to present exercise information that is visually easy to understand in fields such as sports relay and exercise analysis.

  The subject area extraction means 16 generates a subject area image obtained by extracting a moving subject area from a plurality of camera images photographed from a plurality of cameras. Note that an extraction method in the subject area extraction unit 16 will be described later.

  The image composition means 17 uses the subject area extracted by the subject area extraction means 16 and a predetermined background image for each camera, and converts the video obtained for each camera into a background image of a predetermined video frame. A multi-motion image is generated by combining subject area images of a plurality of video frames. As for which video frame is to be synthesized as a background image, the above-described output video setting means 15 generates a necessary composite image based on the set output setting condition of the final multi-view multi-motion video. To do. A specific example of the composition processing in the image composition means 17 will be described later. The image synthesized by the image synthesizing unit 17 is temporarily accumulated in the accumulating unit 14.

  The multi-view video generation unit 18 generates a multi-view multi-motion video while switching the multi-motion video obtained from the image synthesis unit 17 in a predetermined order based on the setting condition from the output video setting unit 15. The multi-view video generation unit 18 can also generate a multi-view video while switching the synchronized video obtained from the plurality of imaging units 11 in a predetermined order. A specific example of multi-view multi-motion video generation in the multi-view video generation means 18 will be described later. Also, the video generated by the multi-view video generation unit 18 is temporarily stored in the storage unit 14.

  The output video generation means 19 outputs the image synthesized by the image synthesis means 17 based on the setting information in the output video setting means 15 and the multi-view multi-motion video and multi-view video obtained by the multi-view video generation means 18 over time. Combine the information as a reference to generate the output video. A specific example of video generation in the output video generation means 19 will be described later.

  The transmission / reception means 20 acquires a moving subject image or the like from another external device or the like via a communication network, or outputs video (multi-view multi-motion video or the like) obtained by the present invention or each of the video generation devices 10 described above. It is a communication interface for transmitting various data acquired by configuration etc. to an external terminal.

  The control unit 21 controls the entire functional configuration of the video generation apparatus 10. Specifically, the control unit 21 extracts a subject area from the acquired video, performs image synthesis based on input information from the user input by the input unit 12, and generates a multi-view video. , Generation of output video, generation and display of a result screen in each process, and accumulation of various data generated in each process described above.

<Example of acquiring multi-viewpoint video>
Next, an example of acquiring a multi-view video by the above-described imaging unit 11 will be described with reference to the drawings.

  FIG. 2 is a diagram illustrating an example for acquiring a multi-view video. FIG. 3 is a diagram showing an example of a video frame obtained corresponding to FIG. As shown in FIG. 2, in the present embodiment, in order to photograph a moving subject 30, a plurality of cameras 31-1 to 31-8 are installed at predetermined positions by tripods or the like.

  Here, images obtained by photographing the subject 30 moving with the passage of time with a plurality of cameras 31-1 to 31-8 are referred to as Im1 to Im8, respectively. In addition, the images captured by the cameras 31-1 to 31-8 each have a predetermined number of frames every second. That is, as shown in FIG. 3, a plurality of video frames are generated by each of the plurality of cameras 31-1 to 31-8 shown in FIG.

  As information acquired from each camera 31-1 to 31-8, camera parameters such as camera direction and angle of view are acquired in addition to video information including video frames and time information corresponding to video. can do.

<Output video setting means 15>
Next, setting conditions in the output video setting means 15 will be described. Specifically, the output video setting means 15 performs setting such as when to output a composite video or a multi-view video from the video frame shown in FIG.

  Specifically, the output video setting unit 15 outputs a composite image based on which position in which time zone is used as a reference (background image) based on time information corresponding to video obtained from the imaging unit 11 such as a camera in advance. In addition, settings are made such as in which time zone and in which order the multi-viewpoint video to be switched is output. Further, the output video setting means 15 may perform settings such as selecting a frame to be output, changing the number of subjects to be combined with the passage of time, and changing the camera that acquires the subject to be combined with the passage of time. it can. Thereby, a user etc. can realize complicated video editing.

<Subject area extraction means 16>
Next, the extraction contents in the subject area extraction means 16 will be described. Here, subject region images (SilIm1 to SilIm8) are generated by extracting moving subject regions present in the videos Im1 to Im8 obtained from the plurality of cameras 31-1 to 31-8.

  For example, among a plurality of video frames taken by all cameras or some cameras, video frames preset by the output video setting means 15 (for example, video frames 1, 3, 5, 7, 9, 11, To output subject area images of a plurality of video frames corresponding to each camera. Here, FIG. 4 is a diagram illustrating an example of the video signal of the subject area image. The subject area image shown in FIG. 4, for example, is obtained by assigning “1” to the subject area 32 that moves with reference to each pixel, and assigning “0” to the other background area 33, and serves as a key signal. It is. That is, the subject region extraction unit 16 extracts a subject region by a video signal or the like in which a key signal as shown in FIG. 4 is superimposed on the original video signal.

  In addition, as a method of extracting the moving subject area described above, for example, a background difference method for extracting only a moving area of a video from a difference between a background image and a target video, or a video of a neighboring video frame and a target video. There is an inter-frame difference method or the like that extracts a moving region from the difference between the two.

  In the background subtraction method described above, an image with no moving object (subject) is captured as a background image by a fixed camera, the background image is subtracted from the image containing the moving object, and the area of the moving object is a value other than “0”. The difference image with is acquired. Next, threshold processing is performed on the pixel value of the difference image to obtain a binary image. Moreover, since the obtained binary image includes small holes and small connected components, these are removed to obtain the region of the moving object. Then, using this binary image, a moving object image is obtained by extracting pixels located in the region of the target object.

  The inter-frame difference method described above acquires a moving object region using, for example, three images taken at different times when a moving object is captured. That is, it is assumed that the background does not change in the three images A, B, and C, the difference images AB and BC between A and B and B and C are created, and threshold processing is performed to obtain a binary image. Next, a logical product process (AND process) of the binary images AB and BC is performed, and a common area of AB and BC is extracted, thereby acquiring a moving object area in the image B.

  As another method for extracting the subject area, there is a method in which the user manually extracts the subject area by software such as Adobe PhotoShop.

<Image composition means 17>
Next, the contents of image composition in the image composition means 17 will be described. The image synthesizing unit 17 generates a synthesized image corresponding to the finally output video set by the output video setting unit 15, and selects a corresponding subject from the subject extracted by the subject region extracting unit 16. Select and compose using the selected image.

  That is, here, the subject area image (SilIm1 to SilIm8) output from the subject area extraction means 16 and the predetermined background image (BgIm1 to BgIm8) selected for each corresponding camera are used for each camera. Multi-motion images (MmIm1 to MmIm8) are generated by synthesizing subject region images (SilIm1 to SilIm8) of a plurality of video frames with a predetermined background image (BgIm1 to BgIm8).

  For example, the video frame used as the background image is a 13-frame image of each camera, and the subject area image of the video frame (for example, 1, 3, 5, 7, 9, 11) extracted by the subject area extraction means 16 is provided there. Synthesize. Then, multi-motion images (MmIm1 to MmIm8) in which images of subject areas of a plurality of video frames are combined and displayed for each camera are generated.

  Here, FIG. 5 is a diagram illustrating an example of a composite image. FIG. 5 shows an example of a multi-motion composite image 40 composed of images obtained by compositing subjects. Of the video frames taken by one camera installed at a predetermined position, the pre-set conditions are used. Subjects 41 to 41-5 obtained from the five selected video frames are displayed on the background image 42.

  Any one of the subjects 41 to 1 to 41-5 is a subject included in the background image 42. In other words, in the example of FIG. 5, a video frame including the subject 41-5 is used as the background image 42, and the subjects 41 to 41-4 extracted by the subject region extraction unit 16 are combined with the background image 42.

  As a technique for synthesizing the subject area image with the predetermined background image, there is a key synthesis technique for synthesizing based on the key signal of the subject area image described above. Here, the key composition method is, for example, when two images are combined to create one image, and when it is desired to specify which part is to be taken from which image, An image (mask image) designated by “1” and “0” is created for each pixel, and image composition is performed based on the image. This can be regarded as a kind of calculation between images.

  A chroma key is often used as a mask image generation method. In this process, each pixel is cut off based on its color information.

<Image composition: first specific example>
Here, a first specific example of image composition in the image composition means 17 will be described. The image synthesizing unit 17 includes subject area images (SilIm1 to SilIm8) output from the subject area extracting unit 16 and predetermined background images (BgIm1 to BgIm8) for each camera, Based on the order in which the subject area image is combined with a predetermined background image (Order 1 to Order 8) for each camera set in advance, the subject area images of a plurality of video frames are added to the predetermined background image (BgIm1 to BgIm8) for each camera. A multi-motion image (MmIm1 to MmIm8) is generated by synthesizing (SilIm1 to SilIm8).

  For example, when a predetermined background image is a 13-frame video of a predetermined camera and the subject area images of the video frames 1, 3, 5, 7, 9, and 11 extracted by the subject area extraction unit 16 are combined therewith, In the case of generating the multi-motion video 31-1 (MmIm1), it is assumed that the order of synthesis by the camera 31-1 (Order1) is in the order of video frames 11 → 9 → 7 → 5 → 3 → 1.

  At this time, a synthesized video is generated by synthesizing the subject area image of the video frame 11 of the camera 31-1 with the background image (13 video) of the camera 31-1 based on the key signal. Further, the subject area image of the video frame 9 of the camera 31-1 is synthesized with this synthesized video based on the key signal.

  Similarly, the subject area images are synthesized in the order of the video frames 7, 5, 3, and 1 to generate a multi-motion image (MmIm1) of the camera 31-1. The same processing is performed for other cameras than the camera 31-1, and multi-motion images (MmIm1 to MmIm8) corresponding to the number of cameras are generated.

<Image composition: second specific example>
Next, a second specific example of image composition in the image composition means 17 will be described. In the second specific example, the image composition means 17 is provided with distance information calculation means. Specifically, the image compositing means 17 uses each camera 31-1 to 31-8 to capture each moving camera from images (Im1 to Im8) and camera parameters (Calib1 to Calib8) of the plurality of cameras. To obtain information on the distance from a moving subject area of a plurality of video frames. Also, the distance information to the moving subject area of a plurality of video frames is compared for each camera, and the order (Order1 to Order8) for combining the subject area images (SilIm1 to SilIm8) with a predetermined background image is determined for each camera. Then, image synthesis is performed based on the result.

  Here, in order to obtain the above-described distance information, for example, the position of the subject is represented by a certain three-dimensional coordinate, and the distance is measured from the three-dimensional coordinate. For example, when calculating the distance information to the moving subject of the video frames 1, 3, 5, 7, 9, and 11 for each camera, for example, between the videos of the video frames 1 of the cameras 31-1 and 31-8. Coordinates (u1, v1) and (u8, v8) in the video of the points of the common subject area shown (gaze points such as the center of the head and the center of the abdomen) are given. Then, the three-dimensional coordinates of the gazing point are calculated using the coordinate values and the camera parameters Calib1 and Calib8 of the cameras 31-1 and 31-8. In addition, the above-mentioned method is, for example, Ninohiro Hiyama and two others, “Prototype of multi-view high-definition video generation system-Use of relay program in all-Japan gymnastics championship”, The Institute of Electronics, Information and Communication Engineers, IEICE Technical 2006-12, P. 43-45. Etc. are described.

  Specifically, the three-dimensional coordinates are obtained by defining the following coordinate conversion formulas (1) to (3) for each camera.

Here, ω represents the image distance, Am represents the internal parameter matrix of the camera m, a represents the aspect ratio of the image, Fm represents the focal length of the camera m, γ represents the skew, and (Cx, Cy ) Represents the intersection of the camera optical axis and the image plane, Rm represents the rotation matrix of the camera m, and Tm represents the translation vector of the camera m.

  In other words, the object is to generate a multi-view video centered on a gazing point selected by the user from a subject that is a target area captured by a plurality of cameras. By seamlessly projecting all camera images into panned and tilted images so that the selected gaze point is at the center, and continuously presenting them, seamlessly focusing on the gaze point Multi-view video can be generated.

  Therefore, camera parameters of each camera necessary for the projective transformation process are obtained by camera calibration. Here, the world coordinates of the gazing point selected by the user are obtained using the above-described coordinate transformation equations (1) to (3), and a straight line connecting the three-dimensional coordinate of the center and the optical center of each camera passes through the center of the image. Perform a projective transformation on

When determining the point of gaze, first, world coordinates of the point of gaze are calculated in order to perform projective transformation so that the point of gaze selected by the user is the center of the image. First, one camera image is selected from a plurality of cameras, and a gazing point (u _k , v _k ) is given. Next, one camera is selected from the other cameras, and image coordinates (u ₁ , v ₁ ) of the same gazing point are given. From the image coordinates of the gazing point of these two cameras and the coordinate conversion equations (1) to (3), the three-dimensional coordinates (X ₁ , Y ₁ , Z ₁ ) of the gazing point are expressed by the following equation (4): Can be obtained by the least square method.

Here, p _ij ^m represents an element in the i-th row and j-th column of the transformation matrix Pm of the coordinate transformation formula (1).

  Next, the distance from each camera to the moving subject area of the video frame 1 can be calculated by calculating the calculated three-dimensional coordinates of the point of the subject area and the distance to each camera.

  Here, the image synthesizing unit 17 obtains distance values from the cameras 31-1 to 31-8 to the moving subject area of the video frame 1 as dist_c1_f1 to dist_c8_f1. Similarly, the distance information to the moving subject area of the video frames 3, 5, 7, 9, and 11 in each camera 31-1 to 31-8, dist_c1_f3 to dist_c8_f3, dist_c1_f5 to dist_c8_f5, dist_c1_f7 to dist_c8_f7, dist_c1_f9_dist_c11_f9_d9_d9 ~ Dist_c8_f11 is obtained.

  Next, the size of distance information for each video frame (1, 3, 5, 7, 9, 11) is compared for each camera, and the order (Order 1 to Order 8) in which the subject area images are synthesized and displayed in order of increasing distance is determined. To do. For example, regarding the camera 31-1, when the distance information for each video frame has a magnitude relationship of “dist_c1_f11> dist_c1_f9> dist_c1_f7> dist_c1_f5> dist_c1_f3> dist_c1_f1,” a subject area to be combined with a predetermined background image of the camera 31-1 The order of images (Order 1) is video frames 11 → 9 → 7 → 5 → 3 → 1.

  That is, the camera images closer to the camera image far from the camera used as the background are sequentially combined. As a result, it is possible to present a subject that moves with time as a video with high accuracy.

<Image composition: Third specific example>
Next, a third specific example of image composition in the image composition means 17 will be described. The image synthesizing unit 17 includes a subject region image (SilIm1 to SilIm8) output from the subject region extracting unit 16, a predetermined background image (BgIm1 to BgIm8) for each camera, and the distance shown in the second specific example described above. By using the order (Order 1 to Order 8) for combining the subject area image with the predetermined background image for each camera determined by the distance information obtained by the information calculation means, the predetermined background image (BgIm1 to BgIm8) is set for each camera. A multi-motion image (MmIm1 to MmIm8) is generated by combining subject area images of a plurality of video frames, and the generated video is output.

  For example, when a predetermined background image is a 13-frame video of each camera and the subject area images of the video frames 1, 3, 5, 7, 9, and 11 extracted by the subject area extraction unit 16 are combined therewith, In the case of generating a multi-motion image (MmIm1) at 31-1, it is assumed that the order of synthesis by the camera 31-1 (Order1) is in the order of video frames 11 → 9 → 7 → 5 → 3 → 1. At this time, a synthesized video is generated by synthesizing the subject area image of the video frame 11 of the camera 31-1 with the background image (13-frame video) of the camera 31-1 based on the key signal.

  Next, the subject area image of the video frame 9 of the camera 1 is synthesized with this synthesized video based on the key signal. Similarly, the subject area images are synthesized in the order of the video frames 7, 5, 3, and 1, a multi-motion video (MmIm1) of the camera 31-1 is generated, and the generated image is output.

  In addition, the image composition unit 17 performs the same processing on other cameras other than the camera 31-1 as necessary based on the setting conditions obtained by the output video setting unit 15, and multi-motion video for the number of cameras. (MmIm1 to MmIm8) are generated, and the generated video is output.

<Multi-view video generation means 18>
Based on the setting conditions in the output video setting unit 15, the multi-view video generation unit 18 generates video that has been captured synchronously by a predetermined camera selected from the plurality of cameras 31-1 to 31-8. A multi-view video is generated by sequentially displaying the images at different times.

  Note that the multi-view video generation unit 18 is based on the setting conditions obtained by the output video setting unit 15, for example, multi-motion images (MmIm 1 to MmIm 8) corresponding to the number of cameras output from the image synthesis unit 17 and selected predetermined ones. For example, the multi-view multi-motion video (OutIm) is generated by switching the images from the multi-motion image or the uncombined video obtained by the plurality of imaging units 11 in the order of camera arrangement, for example.

  For example, when cameras are installed from left to right in the order of the camera 31-1 to the camera 31-8, the multi-motion multi-motion video (OutIm) is switched by continuously switching the multi-motion video in the following frame order. Generate. In this case, the multi-viewpoint video is generated based on the setting conditions such as the switching order and switching time interval set by the output video setting means 15.

  That is, the multi-view video generation unit 18 can output multi-view multi-motion videos (or multi-view videos) of MmIm1 to MmIm8 in, for example, 1 to 8 frames.

<Output video generation means 19>
Next, the content of the output video generated by the output video generation means 19 will be described. The output video generation unit 19 sets the composite image obtained by the image synthesis unit 17, the multi-view multi-motion video obtained by the multi-view video generation unit 18, and the output video setting conditions set by the output video setting unit 15. Based on this, an output video is generated at a predetermined timing.

<Output Video: First Example>
Here, a specific embodiment of the output video in the output video generation means 19 will be described with reference to the drawings. FIG. 6 is a diagram for explaining a first example of output video generation. The output video shown in FIG. 6 is output at predetermined time intervals in the order of the frames 50-1 to 50-8 (FIGS. 6A to 6H). In addition, for each of the frames 50-1 to 50-8, among the plurality of cameras, background images 51-1 to 51-8 from preset cameras are selected.

  Each of the background images 51-1 to 51-8 includes a subject 52-1 obtained from five selected frames among a plurality of video frames photographed by a predetermined camera by the above-described image synthesizing unit 17. ~ 52-5 have been synthesized. Of the five frames, one is also used as a background image. In addition, the subject of the composite image in the frames 50-1 to 50-8 is selected from the five frames selected from the video frames taken by the predetermined cameras selected in the respective frames 50-1 to 50-8. get.

  In this way, according to the first embodiment, as shown in FIG. 6, a highly accurate multi-view multi-motion image can be realized and displayed by the output means 13 such as a display.

<Output Video: Second Example>
Next, a second example of output video generation in this embodiment will be described with reference to the drawings. FIG. 7 is a diagram for explaining a second embodiment of output video generation. The output video shown in FIG. 7 is output in the order of the frames 60-1 to 60-10 (FIGS. 7A to 7J). For the sake of explanation, the frame shown in FIG. 7 uses a frame obtained by selecting and extracting a predetermined frame from an actual output video frame. 13 can be displayed. In each of the frames 60-1 to 60-10, the background image 61 is selected from a preset camera among a plurality of cameras, and the subject image is combined with the background image.

  Here, in FIG. 7A, two subjects respectively obtained from two frames selected by setting among a plurality of video frames shot by one camera preset by the output video setting means 15. Only 62-1a and 62-1b are synthesized and displayed. Further, in FIG. 7B, compared with FIG. 7A, among the plurality of video frames respectively shot with one preset camera, the frame used in FIG. A total of four subjects 62-2a to 62-2d are displayed including subjects obtained from two different frames.

  7C to 7H are obtained from five frames selected by setting among a plurality of video frames respectively shot by six cameras set in advance by the output video setting means 15. , 62-8a to 62-8e are combined, and each of them is generated by the multi-view video generation unit 18 based on the time interval preset by the output video setting unit 15. The generated multi-view video is output. Note that during the reproduction of the images shown in FIGS. 7C to 7H, the actual shooting time itself is stopped (the subject itself is not moving). Thereby, as shown in FIG. 7, it is possible to generate an image that wraps around the subject from different angles, and this image can be displayed on the output means 13 such as a display.

  Also, in FIG. 7 (i), subjects 62-9a to 62-9a obtained from three frames selected by setting among a plurality of video frames photographed by one camera preset by the output video setting means 15 are used. 62-9c is displayed.

  Further, in FIG. 7 (j), only a frame image (image without synthesis) taken with a predetermined camera is output after a lapse of time. Note that such an image is obtained by temporarily storing various images generated in advance by the image synthesizing unit 17 and the multi-viewpoint video generating unit 18 on the storage unit 14 based on the output conditions set in advance by the output video setting unit 15. The output video generation means 19 generates an output video using these images.

  In this way, by using the image set by the output video setting means 15, a combined image or a multi-view video is combined to obtain a subject trajectory and a subject motion flow with high accuracy and output to a display or the like. It can be displayed by means 13. Thereby, a dynamic image with a stereoscopic effect can be generated by switching and displaying the 12 camera images.

<Output Video: Third Example>
Next, a third example of output video generation according to the present embodiment will be described with reference to the drawings. As a third example, an output video is generated by combining the first and second examples described above.

  For example, referring to FIG. 6 described above, in FIG. 6A, an image in which only the subject 52-1e is displayed is displayed, and in FIG. 6B, the subject 52 obtained from two frames selected by setting. -2d and 52-2e are displayed.

  6C to 6G, only the subject obtained from the two frames selected by setting is displayed as in FIG. 6B. Specifically, in FIG. 6C, an image displaying only the subjects 52-3d and 52-3e is displayed, and in FIG. 6D, an image displaying only the subjects 52-4c and 52-4d is displayed. 6 (e) displays an image displaying only the subjects 52-5b and 52-5c, and FIG. 6 (f) displays an image displaying only the subjects 52-6b and 52-6c. In g), an image in which only the subjects 52-7a and 52-7b are displayed is displayed.

  Further, in FIG. 6H, an image displaying only the subject 52-8a is displayed by the output means 13 such as a display. As a result, it is possible to display an image that moves around the subject from different angles while the subject moves, so that it is possible to display a composite image as if time has elapsed in a multi-viewpoint video. Thereby, a more accurate output image can be provided. The above-described video generation apparatus 10 can present a subject that moves with time as a video with high accuracy.

<Execution program>
Here, the video generation device 10 described above can perform video generation in the present invention using the dedicated device configuration described above, but generates an execution program for causing a computer to execute the processing in each configuration, For example, the video generation according to the present invention can be realized by installing the program in a general-purpose personal computer, server, or the like.

<Hardware configuration>
Here, an example of a hardware configuration of a computer capable of executing the video generation processing according to the present invention will be described with reference to the drawings. FIG. 8 is a diagram illustrating an example of a hardware configuration capable of realizing the video generation processing according to the present invention.

  8 includes an input device 71, an output device 72, a drive device 73, an auxiliary storage device 74, a memory device 75, a CPU (Central Processing Unit) 76 for performing various controls, and a network connection device. 77, which are connected to each other by a system bus B.

  The input device 71 includes a keyboard and a pointing device such as a mouse operated by a user, a voice input device, and the like, and inputs various operation signals and voice signals such as a program execution instruction from the user. The output device 72 has a display, a speaker, and the like that display various windows and data necessary for operating the computer main body for performing the processing according to the present invention. Display or audio output is possible.

  Here, in the present invention, the execution program installed in the computer main body is provided by a recording medium 78 such as a CD-ROM or a DVD. The recording medium 78 on which the program is recorded can be set in the drive device 73, and the execution program included in the recording medium 78 is installed from the recording medium 78 to the auxiliary storage device 74 via the drive device 73.

  Further, the drive device 73 can record the execution program according to the present invention in the recording medium 78. Thus, the recording medium 78 can be used for easy installation on a plurality of other computers, and video generation processing can be easily realized.

  The auxiliary storage device 74 is a storage means such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, etc., and perform input / output as necessary. Further, the auxiliary storage device 74 is used as a storage unit that stores the setting conditions of the motion output video, the subject region extraction result, the composite image, the multi-view video, the output video, the camera parameter, the generated display screen, and the like. You can also.

  The memory device 75 stores an execution program or the like read from the auxiliary storage device 74 by the CPU 76. The memory device 75 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  Based on a control program such as an OS (Operating System) and an execution program read from the auxiliary storage device 74 and stored in the memory device 75, the CPU 76 inputs various calculations and data with each hardware component. Each process in the video generation can be realized by controlling processing of the entire computer such as output. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 74 and stored.

  The network connection device 77 obtains an execution program from another terminal connected to the communication network or executes the program by connecting to a communication network such as a telephone line or a LAN (Local Area Network) cable. The execution result obtained in this way or the execution program in the present invention can be provided to other terminals or the like.

  With the hardware configuration as described above, a video generation process can be realized at a low cost without requiring a special device configuration. Also, by installing the program, it is possible to easily realize the video generation process. Next, a processing procedure using the execution program in the present invention will be described using a flowchart.

<Video generation processing>
FIG. 9 is a flowchart showing an example of a video generation processing procedure in the present invention. In FIG. 9, first, an image obtained by photographing a moving subject using a camera or the like as an imaging unit is acquired (S01). Note that in the processing of S01, images taken from a plurality of cameras having different installation positions are used. In the process of S01, not only the data (frame) of the video (image) itself, but also the shooting time, camera parameters of each camera, and the like are acquired.

  Next, what kind of video is finally output is set for the output video (S02). Specifically, in which time frame, using the frame and background image of which camera, the subject extracted from which camera is synthesized, and the multi-view video is generated from a plurality of cameras. Various conditions are set for generating one video by connecting the video in a predetermined time zone. These setting conditions can be stored, for example, in the storage unit 14 or the like, and the settings can be arbitrarily changed.

  Next, a subject area is extracted from the video captured by the camera according to the conditions set in the process of S02 (S03), and the extracted subject and the output video set in the process of S02 are extracted. A background image (including a subject) obtained from a predetermined camera corresponding to the setting condition is synthesized to generate a synthesized image (S04).

  Further, a multi-view video corresponding to the output video condition set in the process of S02 is generated (S05). Next, using the composite image obtained by the process of S04 and the multi-view video obtained by the process of S05, an output video is generated based on the setting condition of the output video in S02 (S06), and the generated video Is output to the screen (S07). The generated video may be stored in the storage unit 14 or the like, for example.

  Here, in the processing procedure described above, the processing in S01 and the processing in S02 may be reversed. That is, it is possible to acquire only a video corresponding to a preset output video setting condition from a predetermined camera. Thereby, it is possible to reduce the recording capacity without accumulating useless video. In addition, only necessary images can be acquired efficiently. Further, the composite image generation in the process of S04 and the multi-view video generation in the process of S05 may be performed in the reverse order. By the video generation process described above, a subject that moves with time can be presented with high accuracy as a video.

  As described above, according to the present invention, it is possible to present a subject that moves with time as a video with high accuracy. In addition, according to the present invention, for example, movement along the passage of time of a sports player or the like can be seen from various viewpoints, so that it is possible to present exercise information that is visually easy to understand in sports relaying, exercise analysis, and the like. it can.

  Although the preferred embodiment of the present invention has been described in detail above, the present invention is not limited to the specific embodiment, and various modifications are possible within the scope of the gist of the present invention described in the claims. Can be changed.

It is a figure which shows the example of 1 structure of the video production | generation apparatus in this invention. It is a figure which shows an example for acquiring a multiview video. It is a figure which shows an example of the video frame obtained corresponding to FIG. It is a figure which shows an example of the video signal of a to-be-photographed area | region image. It is a figure which shows an example of a synthesized image. It is a figure for demonstrating the 1st Example of output image generation. It is a figure for demonstrating the 2nd Example of output image generation. It is a figure which shows an example of the hardware constitutions which can implement | achieve the image | video production | generation process in this invention. It is a flowchart which shows an example of the image | video production | generation process procedure in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image | video production | generation apparatus 11 Imaging means 12 Input means 13 Output means 14 Accumulation means 15 Output image setting means 16 Subject area extraction means 17 Image composition means 18 Multi-view image generation means 19 Output image generation means 20 Transmission / reception means 21 Control means 30, 41 , 52, 62 Subject 31 Camera 32 Subject region 33 Background region 40 Composite image 42, 51, 61 Background image 50, 60 Frame 71 Input device 72 Output device 73 Drive device 74 Auxiliary storage device 75 Memory device 76 CPU
77 Network connection device 78 Recording medium

Claims (6)

In a video generation device that generates a predetermined video based on a preset condition from videos of a moving subject captured by a plurality of imaging means,
Output video setting means for setting the content of the video to be output;
Subject area extraction means for extracting a subject area of a video selected from videos taken by the plurality of imaging means based on setting conditions obtained by the output video setting means;
Image synthesis means for synthesizing the subject area extracted by the subject area extraction means with a predetermined background image;
A multi-view video generation unit that generates a multi-view video by switching a video selected from videos captured by the plurality of imaging units in a predetermined order based on a setting condition obtained by the output video setting unit;
An image generation apparatus comprising: an output image generation unit configured to generate an output image from images obtained by the image synthesis unit and the multi-viewpoint image generation unit. The image composition means includes
2. The video generation apparatus according to claim 1, wherein a plurality of subject area images extracted by the subject area extraction unit are combined with the background image in a predetermined order. The image composition means includes
3. The video generation apparatus according to claim 2, wherein the background image is synthesized in an order based on distance information from the imaging unit to a subject area obtained by camera parameters in the plurality of imaging units. The imaging means includes
4. The video generation apparatus according to claim 1, wherein only a video from an imaging unit that captures a video used for an output video set by the output video setting unit is acquired. 5. The image composition means includes
5. The video generation apparatus according to claim 1, wherein images having different numbers of subjects to be combined with the same image are generated based on setting conditions obtained by the output video setting unit. 6. . In a video generation program for causing a computer to execute a video generation process for generating a predetermined video based on a preset condition from videos of a moving subject imaged by a plurality of imaging means,
An output video setting step for setting the content of the video to be output;
A subject area extracting step for extracting a subject area of a video selected from videos taken by the plurality of imaging means based on the setting condition obtained by the output video setting step;
An image synthesis step of synthesizing the subject area extracted by the subject area extraction step with a predetermined background image;
A multi-view video generation step of generating a multi-view video by switching a video selected from videos taken by the plurality of imaging means in a predetermined order based on the setting condition obtained by the output video setting step;
A video generation program for causing a computer to execute an output video generation step of generating an output video from the video obtained by the image synthesis step and the multi-view video generation step.