JP7241628B2 - MOVIE SYNTHESIS DEVICE, MOVIE SYNTHESIS METHOD, AND MOVIE SYNTHESIS PROGRAM - Google Patents

Description

The present invention relates to technology for generating augmented reality moving images.

2. Description of the Related Art In recent years, video distribution services that allow individuals to distribute videos via networks have become widespread. 2. Description of the Related Art In moving image distribution services, there is known an application capable of distributing a video by making a computer graphics (CG) character appear in the video instead of the user. Also known is a technique of performing face tracking on a self-portrait of a user's face with a smartphone and reflecting the facial expression of the user in a CG character.

JP 2017-188787 A

By synthesizing a CG character with a self-portrait moving image using face tracking technology, it is possible to easily synthesize a CG character reflecting the user's own facial expression with a photographed image in real time.

However, the photographed live-action video is the scenery behind the user. When synthesizing a CG character that reflects the user's own facial expression against the background of the scenery that spreads out before the user's eyes, the user has to use a selfie stick to take a picture of the scenery that the user wants to shoot, including the user himself. rice field.

SUMMARY OF THE INVENTION The present invention has been made in view of the above, and it is an object of the present invention to generate a moving image by synthesizing expressive computer graphics characters more easily.

A video synthesizer according to the present invention is a video synthesizer that generates an augmented reality video by synthesizing an avatar with a live-action video, and obtains a live-action video that serves as a background of the augmented reality video and an image of an operator who operates the avatar. a video acquisition unit, a control unit that controls the avatar based on the operator's image, and the avatar at an arbitrary initial position in a coordinate system corresponding to the real space of the live-action video based on the operator's operation. and a synthesizing unit that generates an augmented reality video by synthesizing the avatar with the live-action video. The synthetic position of the avatar in the augmented reality moving image is changed according to the position in the coordinate system corresponding to the space .

A video synthesis method according to the present invention is a video synthesis method for generating an augmented reality video by synthesizing an avatar with a live-action video. controlling an avatar based on the operator's image; and moving the avatar to an arbitrary initial position in a coordinate system corresponding to the real space of the live-action moving image based on the operator's operation. and generating an augmented reality video by synthesizing the avatar with the live-action video. The synthetic position of the avatar in the augmented reality video is changed according to the position in the coordinate system corresponding to the real space.

A video synthesis program according to the present invention is a video synthesis program for generating an augmented reality video by synthesizing an avatar with a live-action video, and acquires a live-action video as a background of the augmented reality video and an image of an operator who operates the avatar. a process of controlling the avatar based on the operator's image; and a process of arranging the avatar at an arbitrary initial position in a coordinate system corresponding to the real space of the live-action moving image based on the operator's operation. and a process of generating an augmented reality video by synthesizing the avatar with the live-action video. The synthetic position of the avatar in the augmented reality video is changed according to the position in the coordinate system corresponding to the real space .

According to the present invention, it is possible to more easily generate a moving image in which expressive computer graphics characters are synthesized.

1 is an overall configuration diagram showing the overall configuration of a moving image delivery system including a moving image synthesizing device of this embodiment; FIG. FIG. 4 is a diagram for explaining how a distributor distributes an AR video; FIG. FIG. 10 is a diagram showing an example of an AR moving image when the shooting direction is panned to the right; 1 is a functional block diagram showing a configuration example of a moving image synthesizing device according to an embodiment; FIG. 10 is a flow chart showing the flow of avatar initial placement processing. It is a figure which shows the example which superimposed-displayed the floor part detected on the image|photographed image|video. It is a figure which shows the example which stood the avatar on the floor part of FIG. FIG. 8 is a diagram showing an example in which the photographing direction is tilted upward from the state of FIG. 7; FIG. 10 is a flowchart showing the flow of AR video generation processing; FIG. 10 is a flow chart showing the flow of control processing for the facial expression and posture of an avatar. It is a figure which shows the example which displayed the operation button on the touch panel. FIG. 10 is a diagram showing an example of displaying an avatar facing backward while the broadcaster is moving;

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

With reference to FIG. 1, the overall configuration of a moving picture distribution system including the moving picture synthesizing device of this embodiment will be described. This moving picture distribution system is composed of a moving picture synthesizing device 1 and a moving picture distribution server 3 .

The video synthesizer 1 synthesizes a three-dimensional computer graphics character (avatar) with a live-action video captured by the video synthesizer 1 to generate an augmented reality video (hereinafter referred to as an "AR video"). .

The moving picture distribution server 3 receives the AR moving picture from the moving picture synthesizing device 1 and distributes the AR moving picture to the viewer terminal 9 . The video distribution server 3 may distribute the received AR video in real time (so-called live broadcasting), or store the AR video and distribute the AR video in response to a request from the viewer terminal 9. .

The video distribution system may include an avatar management server 5 and a comment management server 7.

The avatar management server 5 manages three-dimensional data of avatars. The video synthesizing device 1 may combine avatar parts provided by the avatar management server 5 to generate an avatar for itself.

The comment management server 7 receives comments on the AR video from the viewer terminal 9 and distributes the comments to the video synthesizer 1 and other viewer terminals 9 .

The video synthesizer 1, the video distribution server 3, the avatar management server 5, the comment management server 7, and the viewer terminal 9 are communicably connected via a network.

AR moving images generated by the moving image synthesizer 1 will be described with reference to FIGS. 2 and 3. FIG.

The video synthesizing device 1 includes an out-camera and an in-camera whose shooting directions are opposite to each other, a microphone, a touch panel, and various sensors (for example, an acceleration sensor, a gyro sensor, etc.) for detecting its own position. A mobile terminal such as a smartphone and a tablet equipped with an out-camera and an in-camera can be used as the video synthesizer 1 .

As shown in FIG. 2, a distributor 200 has a moving image synthesizing device 1, captures a scene viewed by the distributor 200 with an out-camera, and captures the distributor 200 himself/herself with an in-camera. The moving image synthesizing device 1 generates an AR moving image by synthesizing an avatar 100 with a live-action image taken by an out-camera. The moving image synthesizing device 1 reflects the facial expression of the distributor 200 captured by the in-camera on the facial expression of the avatar 100. - 特許庁For example, when the distributor 200 is speaking, the video synthesizer 1 performs face tracking on the face of the distributor 200 captured by the in-camera, and moves the mouth of the avatar 100 to match the mouth of the distributor 200 . The moving image synthesizer 1 may reflect the movement of the head of the distributor 200 on the avatar 100 , or may reflect the gesture of the distributor 200 on the avatar 100 . As a result, the distributor 200 can control the avatar 100 while photographing the scenery that spreads out in front of him.

The video synthesizer 1 fixes the avatar 100 to a coordinate system corresponding to the real space, and synthesizes the avatar 100 with the video. As shown in FIG. 3, when the shooting direction of the out-camera is panned to the right, the avatar 100 also moves leftward in the image, like an object existing in the real space.

[Configuration of video synthesizer]
A configuration example of the video synthesizing device 1 will be described with reference to FIG. The moving image synthesizing device 1 shown in FIG. A communication control unit 20 and a storage unit 21 are provided. Each unit included in the moving image synthesizer 1 may be configured by a computer including an arithmetic processing unit, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in the storage device provided in the motion picture synthesizing apparatus 1, and can be recorded on recording media such as magnetic disks, optical disks, semiconductor memories, etc., or can be provided through a network. For example, an application may be installed on a smart phone and the smart phone may function as the video synthesizer 1 .

The space measurement unit 11 measures three-dimensional space information of the real space in which the avatar is placed, sets a real space coordinate system corresponding to the real space, and determines an area where the avatar can be placed (hereinafter referred to as "avatar placeable area"). ) is detected. For example, the out-camera 17 can capture a moving image of the real space, and the three-dimensional spatial information of the real space of the shooting location can be measured by markerless AR technology using a monocular camera. The space measurement unit 11 detects, from the three-dimensional space information obtained by the measurement, a flat portion such as a floor as an avatar placeable area. A location detected by the space measurement unit 11 as an avatar placement possible area may be inclined with respect to the ground or may be uneven as long as it is a place where it is not unnatural to place an avatar.

If the video synthesizer 1 is equipped with a depth camera or a stereo camera, the space measurement unit 11 may obtain three-dimensional space information from the measurement results of the depth camera or stereo images. The space measurement unit 11 stores the measured three-dimensional space information in the storage unit 21 .

The initial placement unit 12 determines the initial position of the avatar so that the avatar exists within the avatar placeable area detected by the space measurement unit 11 . For example, an image in which a figure indicating the avatar placement possible area (for example, a frame indicating the floor area, etc.) is superimposed on the image of the real space captured by the out-camera 17 is displayed, and the distributor is asked to specify the initial position of the avatar. prompt. When the distributor taps the avatar placeable area, the initial placement unit 12 calculates the coordinates of the tapped position in the real space coordinate system and determines it as the initial position of the avatar. The initial placement unit 12 may place the avatar at any position in the avatar placeable area. When the distributor shakes the motion picture synthesizer 1, the position of the avatar may be changed at random.

The avatar control unit 13 performs face tracking on the face of the distributor photographed by the in-camera 16, and reflects the facial expression of the distributor on the facial expression of the avatar. The avatar control unit 13 may control the avatar according to the operation input by the distributor through a menu or the like. The avatar control section 13 may control the posture and position of the avatar based on the movement of the video synthesizer 1 . For example, when the broadcaster is moving forward while photographing the scenery, the avatar is turned forward and the avatar is made to walk.

The synthesizing unit 14 arranges the avatar in the real space coordinate system, synthesizes the avatar with the video captured by the out-camera 17, and generates an AR video. The AR moving image is displayed on the display unit 19 and transmitted from the communication control unit 20 to the moving image distribution server 3 . The synthesizing unit 14 may store the AR moving images in the storage unit 21 . Data necessary for rendering the avatar is received from the avatar management server 5 and stored in the storage unit 21 . Avatar data stored in the storage unit 21 in advance may be used.

The position detection unit 15 detects the position and orientation of the video synthesizing device 1 itself (also the position and orientation of the out-camera 17) in the real space coordinate system. The synthesizer 14 renders the avatar based on the position and orientation detected by the position detector 15 .

The in-camera 16 photographs the distributor (who is also the operator of the moving picture synthesizer 1).

The out-camera 17 shoots a landscape and a subject to be transmitted as an AR video.

The input unit 18 receives an operation from the touch panel provided in the moving image synthesizer 1 .

The display unit 19 displays an AR moving image obtained by synthesizing an avatar with a live-action image taken by the out-camera 17 on the touch panel. Various buttons for operating the avatar may be displayed.

The communication control unit 20 transmits the AR video to the video distribution server 3 .

[Initial placement of avatars]
An example of initial placement processing of avatars will be described with reference to FIG.

The processing shown in FIG. 5 is performed when the distributor determines the position of the avatar in the real space coordinate system before generating the AR video.

The distributor activates the video synthesizer 1, takes a video of the place where the avatar is placed with the out-camera 17, and acquires the three-dimensional space information of the place where the avatar is placed (step S11). Specifically, the application is activated to operate the smartphone as the video synthesizer 1 , and the out-camera 17 shoots a flat place where the avatar is to be placed. The distributor shoots the arrangement place while slightly moving the moving picture synthesizing device 1. - 特許庁Three-dimensional space information is acquired from the movement of the moving image synthesizer 1 and the movement of feature points that can be detected from the captured moving image, and an avatar placement possible area that permits the placement of an avatar is detected. Here, a flat "floor" is detected as an avatar placeable area.

When the space measurement unit 11 acquires the three-dimensional space information and detects the floor, the display unit 19 superimposes and displays a figure indicating the floor area on the image captured by the out-camera 17 (step S12). For example, as shown in FIG. 6, a frame 110 indicating the area of the floor is superimposed on the image captured by the out-camera 17 and displayed.

When the distributor taps inside the frame 110, as shown in FIG. 7, the avatar control unit 13 arranges the avatar 100 at the tapped position (step S13). Let the coordinates of the tapped position in the real space coordinate system be the coordinates of the avatar's standing position. When another place within the frame 110 is tapped again, the newly tapped position is set as the standing position of the avatar. When the coordinates of the avatar in the real space coordinate system are determined, the synthesizing unit 14 superimposes the avatar on the actual image taken by the out-camera 17 and displays it. After that, the avatar is displayed superimposed on the live-action video as if the avatar exists in the real space. For example, when the photographing direction of the out-camera 17 is tilted upward from the state shown in FIG. 7, the upper body of the avatar 100 is displayed while the standing position of the avatar 100 remains fixed in the real space, as shown in FIG. Even when the photographing direction of the out-camera 17 is panned left and right, the standing position of the avatar 100 remains fixed in the real space, and the avatar 100 is superimposed on the photographed image.

The standing position of the avatar 100 may be adjusted by dragging the avatar 100 while the avatar 100 is being displayed. For example, as shown in FIG. 8, when the avatar 100 is displayed and the broadcaster taps the avatar 100 and moves his or her finger left or right along the screen, the standing position of the avatar 100 is moved left or right. When the distributor moves his finger up and down, the standing position of the avatar 100 is moved in the depth direction or the front direction. The movement of the avatar 100 is stopped so as not to exceed the range recognized as the floor when the avatar 100 is moved back and forth and to the left and right.

The initial placement unit 12 may randomly determine the standing position of the avatar when the distributor shakes the video synthesizing device 1 . When the distributor tilts the video synthesizer 1 , the initial placement unit 12 may move the standing position of the avatar according to the tilt of the video synthesizer 1 . For example, when the distributor tilts the video synthesizer 1 to the right, the avatar 100 is moved to the right, and when the video synthesizer 1 is tilted forward, the avatar 100 is moved forward.

The standing position of the avatar may be adjusted based on the image of the distributor captured by the in-camera 16 . For example, when the broadcaster turns to the right, the avatar's standing position is moved to the right. When the broadcaster looks down, the avatar's standing position is moved forward.

When the distributor determines the position of the avatar, the initial placement unit 12 determines the size and orientation of the avatar according to the distributor's operation (step S14). For example, when the distributor flicks the touch panel up and down, the size of the avatar is enlarged or reduced. When the broadcaster flicks the touch panel left or right, the direction of the avatar is rotated. When the broadcaster taps the touch panel with two fingers, the size and orientation of the avatar are reset to the initial size and orientation.

Objects such as tatami mats, carpets, and stages may be placed on the floor where the avatar stands. If a tall platform is placed as a foot object, the avatar's standing position is raised by the height of the platform.

The avatar may be allowed to move freely on the floor within a predetermined range centering on the initial position of the avatar. For example, when the silence of the broadcaster continues for a while, the avatar may be controlled to wander within a predetermined range.

[Generation of AR video]
When the distributor finishes the initial placement of the avatar, the video synthesizer 1 starts photographing the distributor with the in-camera 16, and starts generating an AR video.

An example of an AR video generation process will be described with reference to FIG. 9 .

At the same time when the out-camera 17 takes a picture of the scenery (step S21), the in-camera 16 takes a picture of the distributor (step S22). A microphone picks up the sound that accompanies the video.

The position detection unit 15 detects the position and orientation of the moving image synthesizer 1 (step S23).

The avatar control unit 13 controls the expression and posture of the avatar based on the image of the distributor captured by the in-camera 16 (step S24). Details of the processing of the avatar control unit 13 will be described later.

The synthesizing unit 14 synthesizes the avatar with the live-action video captured by the out-camera 17 to generate an AR video (step S25).

[Avatar control]
With reference to FIG. 10, an example of control processing of an avatar's facial expression and posture by the avatar control unit 13 will be described.

The avatar control unit 13 determines whether or not the distributor is moving (step S31). Whether or not the distributor is moving can be determined based on the movement of the moving picture synthesizing device 1 detected by the position detection unit 15 .

If the distributor is moving (YES in step S31), it is determined whether the avatar is in a position-fixed state (step S32). An avatar in a position-fixed state is a state in which the position of the avatar in the real space coordinate system is not moved. Even if the broadcaster moves, the avatar's standing position does not move. When the position of the avatar is not fixed, a predetermined distance is maintained between the video synthesizing device 1 and the avatar in the real space coordinate system, and the position of the avatar is moved according to the movement of the video synthesizing device 1 . When the distributor, that is, the motion picture synthesizer 1 moves, the avatar control unit 13 moves the position of the avatar according to the movement of the distributor. The movement of the avatar will be described later.

Cancellation of the fixed position state of the avatar can be changed by operating the fixed position button 130 displayed on the touch panel shown in FIG. When the position fixing button 130 is operated while the avatar is in the fixed position state, the position fixing of the avatar is released. When the position fixing button 130 is operated when the position of the avatar is not fixed, the position of the avatar is fixed.

If the distributor is not moving (NO in step S31) or if the position of the avatar is fixed (YES in step S32), the avatar control unit 13 determines whether or not the facial expression button has been operated (step S33). In this embodiment, as shown in FIG. 11, expression buttons 120A, 120B, and 120C are displayed on the touch panel. Expression buttons 120A, 120B, and 120C correspond to laughter, cry, and anger, respectively.

When the facial expression buttons 120A, 120B, 120C are operated (YES in step S33), the avatar control unit 13 changes the facial expressions of the avatars to the facial expressions corresponding to the operated facial expression buttons 120A, 120B, 120C (step S34). . By preparing an exaggerated facial expression animation and making the avatar animate according to the operation of the facial expression button, it is possible to express emotions more clearly. In addition to changing the facial expression of the avatar, you may also make the avatar make gestures. For example, when the crying facial expression button is operated, the avatar is made to make a gesture of wiping tears with a hand.

Buttons for controlling the posture of the avatar (including the orientation of the avatar) may be displayed on the touch panel. For example, a button for turning the avatar, a button for turning the avatar's face or upper body to the right or left, etc. may be displayed on the touch panel. When these buttons are operated, the avatar control unit 13 changes the posture of the avatar to the posture corresponding to the operated button.

You may be able to control the avatar's expression and pose from a menu. For example, swipe the edge of the touch panel to bring up a menu bar, and select items corresponding to facial expressions and postures you want your avatar to have.

The hand sign captured by the in-camera 16 may be used to control the avatar. The avatar control unit 13 associates hand signs with avatar control details (such as facial expressions and postures), and when a specific hand sign is detected from a moving image taken by the in-camera 16, the avatar is displayed in accordance with the detected hand sign. to control. For example, when a clenched fist is photographed with the in-camera 16, the avatar is made to express anger.

The avatar may be controlled by characters or graphics captured by the in-camera 16 . When the avatar control unit 13 detects specific characters or graphics from the moving image captured by the in-camera 16, the avatar control unit 13 controls the avatar according to the detected characters or the like. For example, "smile" is written on a piece of paper, and the paper is photographed by the in-camera 16 to make the avatar smile.

The avatar may be controlled by the movement of the motion picture synthesizer 1 . The avatar control unit 13 associates the motion of the motion picture synthesizer 1 with the control details of the avatar, and controls the avatar according to the motion of the motion picture synthesizer 1 detected by the position detection unit 15 . For example, when the motion picture synthesizer 1 is tilted, the avatar is made to bow. When the moving image synthesizing device 1 is tilted, the scenery photographed by the out-camera 17 is tilted. Therefore, a still image may be synthesized with a bowing avatar using one frame of the moving image photographed immediately before the tilt is detected as a background. . Instead of a still image, a moving image of about several seconds immediately before the tilt is detected may be used as the background.

The avatar control unit 13 may control the avatar based on information from a sensor attached to the distributor, or may control the avatar using an external input device such as a keyboard. The sensor/input device and the video synthesizing device 1 communicate wirelessly.

If the facial expression button is not operated (YES in step S33), the avatar control unit 13 performs face tracking on the face of the broadcaster captured by the in-camera 16, and reflects the facial expression of the broadcaster on the avatar (step S35). By reflecting the expression of the broadcaster captured by the in-camera 16 on the avatar, the avatar can be controlled expressively without bothering the broadcaster.

The avatar control unit 13 may cause the avatar to speak based on the voice of the distributor collected by the microphone.

If the distributor is moving (YES in step S31) and the avatar is not in a position-fixed state (NO in step S32), the avatar control unit 13 directs the avatar 100 toward the distributor (step S36). ), and the standing position of the avatar 100 is moved in the advancing direction (step S37). Specifically, the avatar control unit 13 moves the position of the avatar so that the distance between the position of the video synthesizer 1 and the standing position of the avatar in the real space coordinate system is maintained at a predetermined distance. The avatar control unit 13 moves the avatar away from the distributor when the distributor (video synthesizer 1) moves forward, moves the avatar closer to the distributor when the distributor retreats, and moves the avatar closer to the distributor when the distributor stops. to stop The processing from step S33 to step S35 may be performed independently of the movement of the avatar. Specifically, the process of step S33 may be executed after the process of step S37.

The avatar control unit 13 may fix the orientation of the avatar, or may direct the orientation of the avatar to the traveling direction of the avatar. For example, if the position of the avatar is not fixed but the direction of the avatar is fixed, when the distributor moves forward in the state of FIG. Move your avatar backwards. If the position of the avatar is not fixed and the direction of the avatar is not fixed, when the distributor moves forward in the state of FIG. Move your avatar forward.

If the microphone is collecting the voice of a person other than the distributor, the avatar control unit 13 may direct the avatar in the direction of the voice.

When the destination of the avatar is not the floor, for example, when the destination is a wall, the avatar control unit 13 may cause the avatar to stop at the edge of the floor without moving. When a distributor distributes an AR video while walking on the road, the space measurement unit 11 detects a flat portion in the direction of travel from the image of the out-camera 17, and the avatar control unit 13 moves the avatar to the flat portion in the direction of travel. Let As a result, when the distributor distributes the AR video while walking down the road, it is possible to generate an AR video in which the avatar appears to be walking down the road.

When the position of the avatar is not fixed, even if the broadcaster is not moving, when the shooting direction of the out-camera 17 is panned left or right, the avatar control unit 13 moves the avatar left or right according to the shooting direction. good too.

As described above, according to the present embodiment, the out-camera 17 shoots a live-action video, the in-camera 16 shoots the distributor, and the avatar control unit 13 captures an image of the distributor shot with the in-camera 16. The avatar is controlled based on the image, and the synthesizer 14 arranges the avatar at a predetermined position in the real space coordinate system and synthesizes the avatar with the live-action video. You can generate an AR video that synthesizes the reflected avatar. By performing face tracking of the face of the distributor photographed by the in-camera 16 and reflecting it on the avatar, an avatar with rich expressions can be synthesized.

According to the present embodiment, the space measurement unit 11 acquires the three-dimensional space information of the shooting location, detects an avatar placeable area that permits placement of the avatar, and the initial placement unit 12 places the avatar in the avatar placeable area. By determining the position of the avatar in the real space coordinate system, the avatar can be fixed in the real space without installing a marker for determining the standing position of the avatar.

According to the present embodiment, the avatar control unit 13 moves the position of the avatar in the real space coordinate system according to the position in the real space coordinate system of the video synthesizer 1 , thereby allowing the distributor to hold the video synthesizer 1 . If you shoot while walking, you can generate an AR video that synthesizes an avatar that moves according to the movement of the broadcaster.

In the present embodiment, the video synthesizer 1 generates the AR video, but the server generates the AR video such as the live-action video captured by the out-camera 17 and the image captured by the in-camera 16. You may transmit the data necessary for the generation of and generate the AR video on the server or the cloud.

REFERENCE SIGNS LIST 1 motion picture synthesis device 11 space measurement unit 12 initial placement unit 13 avatar control unit 14 synthesis unit 15 position detection unit 16 in-camera 17 out-camera 18 input unit 19 display unit 20 communication control unit 21... Storage unit 3... Video distribution server 5... Avatar management server 7... Comment management server 9... Viewer terminal

Claims (10)

A video synthesizer for generating an augmented reality video by synthesizing an avatar with a live-action video,
a video acquisition unit that acquires a live-action video that serves as a background of an augmented reality video and an image of an operator who operates an avatar;
a control unit that controls an avatar based on the image of the operator;
a placement unit that places the avatar at an arbitrary initial position in a coordinate system corresponding to the real space of the live-action video based on the operation of the operator;
a synthesizing unit that generates an augmented reality video by synthesizing the avatar with the live-action video ;
The synthesizing unit changes a synthetic position of the avatar in the augmented reality moving image according to a position of the avatar in a coordinate system corresponding to the real space when a shooting direction of the live-action moving image changes. A moving image synthesizer. A video synthesizer for generating an augmented reality video by synthesizing an avatar with a live-action video,
a video acquisition unit that acquires a live-action video that serves as a background of an augmented reality video and an image of an operator who operates an avatar;
a control unit that controls an avatar based on the image of the operator;
a space measurement unit that acquires three-dimensional space information of the shooting location of the live-action video and detects an area where the avatar can be placed ;
a placement unit that places the avatar at an initial position where the avatar can be placed in a coordinate system corresponding to the real space of the live-action video based on the operation of the operator;
and a synthesizing unit that generates an augmented reality video by synthesizing the avatar with the live-action video.
A video synthesizer characterized by: The space measurement unit detects a flat portion as an area where the avatar can be placed.
3. The moving picture synthesizing device according to claim 2, wherein: 4. The moving picture synthesizing apparatus according to claim 2, wherein said space measuring section acquires three-dimensional spatial information of a shooting location from movements of feature points detected from said photographed moving picture. 5. The moving image synthesizing device according to claim 1, wherein the control unit moves the avatar according to movements of the operator. 6. The moving image synthesizing device according to claim 5 , wherein the controller directs the avatar in the direction in which the operator has advanced, and moves the position of the avatar according to the movement of the operator. . A video synthesis method for generating an augmented reality video by synthesizing an avatar with a live-action video,
by computer,
a step of obtaining an image of an operator operating an avatar and a live-action video as a background of the augmented reality video;
controlling an avatar based on the image of the operator;
a step of arranging the avatar at an arbitrary initial position in a coordinate system corresponding to the real space of the live-action video based on the operation of the operator;
generating an augmented reality video by synthesizing the avatar with the live-action video ;
In the step of generating the augmented reality video, when the shooting direction of the live-action video changes, the composite position of the avatar in the augmented reality video is changed according to the position of the avatar in the coordinate system corresponding to the real space. A method of synthesizing moving images characterized by changing . A video synthesis method for generating an augmented reality video by synthesizing an avatar with a live-action video,
by computer,
a step of obtaining an image of an operator operating an avatar and a live-action video as a background of the augmented reality video;
controlling an avatar based on the image of the operator;
obtaining three-dimensional spatial information of the shooting location of the live-action video, and detecting an area where the avatar can be placed;
arranging the avatar at an initial position where the avatar can be arranged in a coordinate system corresponding to the real space of the live-action video, based on the operation of the operator;
and generating an augmented reality video by synthesizing the avatar with the live-action video.
A moving image synthesis method characterized by: A video synthesis program for generating an augmented reality video by synthesizing an avatar with a live-action video,
a process of acquiring an image of an operator operating an avatar and a live-action video as a background of the augmented reality video;
a process of controlling an avatar based on the image of the operator;
A process of arranging the avatar at an arbitrary initial position in a coordinate system corresponding to the real space of the live-action video based on the operation of the operator;
causing a computer to execute a process of generating an augmented reality video by synthesizing the avatar with the live-action video;
In the process of generating the augmented reality video, when the shooting direction of the live-action video changes, the composite position of the avatar in the augmented reality video is changed according to the position of the avatar in the coordinate system corresponding to the real space. A moving picture composition program characterized by changing . A video synthesis program for generating an augmented reality video by synthesizing an avatar with a live-action video,
a process of acquiring an image of an operator operating an avatar and a live-action video as a background of the augmented reality video;
a process of controlling an avatar based on the image of the operator;
a process of acquiring three-dimensional spatial information of the shooting location of the live-action video and detecting an area where the avatar can be placed;
A process of arranging the avatar at an initial position where the avatar can be arranged in a coordinate system corresponding to the real space of the live-action video, based on the operation of the operator;
a process of generating an augmented reality video by synthesizing the avatar with the live-action video;
A video composition program characterized by causing a computer to execute

Images