JP2019179502A - System, method, and program for creating moving image - Google Patents

Abstract

To provide a system, a method, and a program for supporting creation of a moving image including an avatar of a user.SOLUTION: A moving image creation device 10 has a function for creating a moving image including an avatar of a user. The device 10 creates the moving image including the avatar of the user, and an operation of the avatar is controlled based on an arrangement of predetermined parts of a body of the user in an image input via a camera 131. Therefore, the user can easily create, for example, by moving the body in front of the camera 131, the moving image including the avatar that operates in accordance with a movement of the body.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system, a method, and a program for creating a moving image including a user's avatar.

  2. Description of the Related Art Conventionally, a system for distributing a moving image by a user has been provided, and a moving image distributed through such a system includes a moving image (self-shooting moving image) in which the user himself / herself appears as a subject (for example, a patent) Reference 1). For example, the user can shoot a moving image including the user himself / herself through a camera and a microphone included in a smartphone, a personal computer, and the like, and can distribute the moving image to a plurality of viewers.

JP 2017-121036 A

  However, there may be a user who is reluctant to open his face to an unspecified number of viewers. And such a user delivers only audio | voice, for example, but the content only of audio | voice may lack interest. Therefore, for example, instead of the user himself / herself, it is considered useful to be able to easily create a video in which the user's avatar appears.

  An embodiment of the present invention has an object of supporting creation of a moving image including a user's avatar. Other objects of the embodiments of the present invention will become apparent by referring to the entire specification.

  A system according to an embodiment of the present invention includes one or more computer processors, and is a system for creating a moving image, wherein the one or more computer processors are configured to execute a readable instruction, A process of recognizing one or a plurality of predetermined parts of the user's body included in an image input via the camera, and an avatar action based on at least the arrangement of the recognized predetermined part in the input image A process of controlling and a process of creating a moving image including the avatar are executed.

  A method according to an embodiment of the present invention is a method for creating a moving image that is executed by one or more computers, and includes one or more of a user's body included in an image input via a camera. Recognizing a predetermined part, controlling the operation of the avatar based on at least the arrangement of the recognized predetermined part in the inputted image, and creating a moving image including the avatar. .

  A program according to an embodiment of the present invention is a program for creating a moving image, and is input to the one or more computers via a camera in accordance with execution on the one or more computers. A process for recognizing one or more predetermined parts of the user's body included in the image, a process for controlling the movement of the avatar based on at least the arrangement of the recognized predetermined part in the inputted image, and the avatar And a process of creating a moving image including.

  Various embodiments of the present invention assist in creating a video that includes a user's avatar.

1 is a configuration diagram schematically showing a configuration of a moving image creating apparatus 10 according to an embodiment of the present invention. FIG. 2 is a block diagram schematically showing functions of the moving image creating apparatus 10. The flowchart which illustrates the process which the moving image production apparatus 10 performs. The figure which illustrates typically the input image 50 containing a user's face and both hands. The figure for demonstrating a mode that the user's face and both hands which are contained in the input image 50 are recognized. The figure which illustrates the animation 60 created. The figure which illustrates the input image 50. The figure which illustrates animation 60. The figure which illustrates animation 60 of the state where avatar 62 is expanded and displayed. The figure which illustrates animation 70 of live distribution.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically showing the configuration of a moving image creating apparatus 10 according to an embodiment of the present invention. The moving image creating apparatus 10 has an function of creating a moving image including a user's avatar, and is an example of an apparatus that implements part or all of the system of the present invention.

  The moving image creating apparatus 10 is configured as a general computer. As shown in FIG. 1, a computer processor 11 such as a CPU, a main memory 12, a user I / F 13, a communication I / F 14, a storage ( A storage device) 15, and these components are electrically connected via a bus or the like (not shown).

  The computer processor 11 reads various programs stored in the storage 15 or the like into the main memory 12 and executes various instructions included in the programs. The main memory 12 is configured by, for example, a DRAM.

  The user I / F 13 includes various input / output devices for exchanging information with the user. The user I / F 13 includes, for example, an information input device such as a keyboard and a pointing device (for example, a mouse and a touch panel) and a voice input device such as a microphone. The user I / F 13 includes an image output device such as a display and an audio output device such as a speaker. In the present embodiment, the user I / F 13 includes an image input device (imaging device) such as a camera 131. For example, the camera 131 is configured as an in-camera including a user who views an image output device such as a display in the field of view.

  The communication I / F 14 is implemented as hardware such as a network adapter, various types of communication software, and combinations thereof, and is configured to realize wired or wireless communication.

  The storage 15 is configured by, for example, a magnetic disk, a flash memory, or the like. The storage 15 stores various programs including an operating system, various data, and the like. The program stored in the storage 15 may include an application program for realizing a function for creating a moving image including an avatar (hereinafter, referred to as “moving image creating application”).

  In the present embodiment, the moving image creation apparatus 10 can be configured as a smartphone, a tablet terminal, a personal computer, a wearable device, or the like.

  Next, functions of the moving image creating apparatus 10 according to the present embodiment will be described. FIG. 2 is a block diagram schematically showing functions of the moving image creating apparatus 10. As shown in the figure, the moving image creating apparatus 10 includes an information storage management unit 41 that stores and manages various information, an image analysis unit 43 that analyzes an image input via the camera 131, and a moving image including a user's avatar. And a moving image creation unit 45 for creating These functions are realized by the cooperation of hardware such as the computer processor 11 and the main memory 12 and various programs and data stored in the storage 15, for example, in the main memory 12. This is realized by the computer processor 11 executing instructions included in the read program.

  The information storage management unit 41 stores and manages various information in the storage 15 and the like. The image analysis unit 43 executes various processes related to image analysis. In the present embodiment, the image analysis unit 43 is configured to recognize a predetermined part of the user's body included in an image input via the camera 131. For example, the image analysis unit 43 inputs an image input via the camera 131 to a learned model generated via machine learning, and selects a predetermined part based on information output from the learned model. Configured to recognize.

  The moving image creating unit 45 executes various processes related to creating a moving image including the user's avatar. In the present embodiment, the moving image creation unit 45 controls the operation of the avatar based at least on the arrangement of the predetermined part recognized by the image analysis unit 43 (position and / or size in the input image). Is configured to do. The created moving image is stored in, for example, the storage 15 or transmitted to a server or the like that is communicably connected via a communication network such as the Internet.

  As described above, the moving image creating apparatus 10 according to the present embodiment creates a moving image including an avatar, and the avatar is based on the arrangement of a predetermined part of the user's body in an image input via the camera 131. Since the movement is controlled, the user can easily create a moving image including an avatar that moves in accordance with the movement of the body (change in the arrangement of a predetermined part) by moving the body in front of the camera 131. Can do. Thus, the moving image creating apparatus 10 of the present embodiment supports the creation of a moving image including the user's avatar.

  In the present embodiment, the moving image creation unit 45 can be configured to deliver the created moving image in real time. For example, the moving image creating apparatus 10 is connected to a moving image distribution server for distributing a moving image to a plurality of viewers in real time (performing live distribution) via a communication network. Is configured to transmit the created moving image to the moving image distribution server in real time (for example, in a streaming format). Such a configuration makes it possible to easily realize live distribution of a moving image including the user's avatar.

  In the present embodiment, the predetermined part recognized by the image analysis unit 43 (in other words, the part of the body used for controlling the movement of the avatar) may include one or more arbitrary parts of the user's body. For example, the image analysis unit 43 may be configured to recognize the user's face and both hands as the predetermined part. Such a configuration makes it possible to operate the avatar according to the movement of the user's face and both hands.

  In addition, the moving image creating unit 45 may be configured to execute a predetermined command specified based on at least a part of movement of the recognized predetermined part. The predetermined command is, for example, a command for controlling an avatar (for example, a command for changing an avatar's facial expression or a setting item), and a command for controlling creation of a moving image (for example, a moving image). Command etc. for temporary suspension of the creation of For example, when the moving image creation unit 45 monitors the movement of a predetermined part in an image input via the camera 131 and detects the movement of a predetermined part that is predetermined as a movement corresponding to a predetermined command. , Configured to execute the corresponding command. Such a configuration makes it possible to input a command based on the movement of the user's body.

  The predetermined command includes a command for stopping the control of the avatar's operation based on the arrangement of the predetermined part. For example, when the moving image creation unit 45 detects the movement of a predetermined part corresponding to the command, the moving picture creation unit 45 controls the operation of the avatar regardless of the arrangement of the predetermined part in the input image (to make the avatar still. Is configured.) Such a configuration enables the user to move his / her body in a state that is not reflected in the avatar's motion (stopping the linkage between the user's motion and the avatar's motion).

  In the present embodiment, the image analysis unit 43 detects the predetermined marker (mark) having a predetermined appearance (color, pattern, and / or shape) provided in a predetermined part of the user's body. It may be configured to recognize the site. Such a configuration can improve the recognition accuracy of a predetermined part and reduce the processing load related to the recognition.

  In this case, the predetermined part includes a hand, and the image analysis unit 43 includes a marker having a first appearance provided on the back side of the user's hand and a marker having a second appearance provided on the palm side of the user's hand. The direction of the user's hand is recognized by detecting each, and the moving image creation unit 45 may be configured to control the operation of the avatar based at least on the direction of the user's hand recognized by the image analysis unit 43. Such a configuration makes it possible to control the movement of the avatar based on the orientation of the user's hand.

  In addition, the moving image creating unit 45 can be configured to control the operation of the avatar so as to follow the movement of a predetermined part in the input image. Such a configuration enables creation of a moving image including an avatar that moves following the movement of the user's body.

  Next, a specific example of the moving image creating apparatus 10 of the present embodiment having such a function will be described. The moving image creating apparatus 10 in this example is configured as a smartphone, a tablet terminal, a personal computer, or the like, and a moving image creating application is installed therein.

  FIG. 3 is a flowchart illustrating processing executed by the moving image creating apparatus 10 when creating a moving image including an avatar in this example. For example, when the user gives an instruction to start creating a moving image via a screen displayed on the moving image creating apparatus 10, the process illustrated in FIG. 3 is executed.

  First, as shown in FIG. 3, the moving image creating apparatus 10 recognizes the user's face and both hands included in the input image input via the camera 131 (step S100). In this example, the camera 131 is configured as an in-camera, and the camera 131 includes a user who views the screen displayed on the apparatus 10 in the field of view. The user moves the body in front of the camera 131 to operate the avatar while watching the created animation displayed on the device 10.

  FIG. 4 schematically illustrates an input image 50 input via the camera 131. As shown in the figure, in this example, a circular marker MK1 of the first color (for example, red) is provided on the palm of the user's right hand RH, and a second marker is provided on the palm of the user's left hand LH. A circular marker MK2 of color (for example, yellow) is provided. These markers MK1 and MK2, for example, are configured as stickers to be attached to the palm of the hand, and the stickers are provided to the user by, for example, a provider of a moving image creating application. The markers MK1 and MK2 are directly drawn with ink or the like on the palm of the hand, for example. In this case, for example, the user draws the markers MK1 and MK2 on the palms of both hands according to instructions distributed by the provider of the moving image creation application.

  FIG. 5 is a diagram for explaining how the user's face and both hands included in the input image 50 illustrated in FIG. 4 are recognized. As shown in the figure, in this example, the user's face FC is detected and recognized as a rectangular detection area DA1 surrounding the outline of the face FC. In addition, the user's hands RH and LH are detected and recognized as rectangular detection areas DA2 and DA3 surrounding the contours of the markers MK1 and MK2 provided on the palms of the hands RH and LH, respectively. Such recognition of face FC and both hands RH, LH (markers MK1, 2) (and subsequent tracking) is realized using a well-known object tracking technique, for example, a learned learning generated through machine learning. Realized using a model.

  Returning to the flowchart of FIG. 3, when the user's face and both hands included in the input image are recognized, the moving image creating apparatus 10 next determines whether or not a command is input (step S110). The presence / absence of command input is determined based on the movement of the user's hand included in the input image. Details of the determination of presence / absence of command input will be described later.

  Then, when there is no command input (NO in step S110), the moving image creating apparatus 10 controls the operation of the avatar based on the user's face and the arrangement of both hands (step S120). The control of the avatar movement based on the user's face and the arrangement of both hands in the input image is repeated until the creation of the moving image ends (NO in step S140). In other words, during the period of creating (recording and recording) a moving image, the moving image creating apparatus 10 includes the moving image according to the change in the arrangement of the user's face and both hands recognized and tracked in the image input via the camera 131. Operate the avatar.

  FIG. 6 illustrates a moving image 60 (one scene) to be created. The moving image 60 is configured as an image obtained by viewing a three-dimensional virtual space with a specific field of view (via a virtual camera at a specific position). The virtual space includes the humanoid avatar 62 and the avatar 62. It includes a desk object 64 positioned in front. In this example, the relative positions of both hands RH and LH (the positional relationship between the actual user's face and both hands) with respect to the position of the user's face FC in the input image 50 input via the camera 131 are reproduced. As described above, the operation of the avatar 62 is controlled. For example, as illustrated in FIG. 7, the actual user takes a “many years” posture, and the user's hands RH and LH (markers MK 1 and 2) in the input image 50 move obliquely above the user's face FC. In this case, as illustrated in FIG. 8, the avatar 62 in the moving image 60 also takes a “many” posture (moves both hands diagonally above the face) in the same manner as the user.

  In this example, the avatar 62 is enlarged or reduced according to the change in the size of the user's face FC in the input image 50. In other words, according to the change in the distance between the real user and the camera 131, the distance between the avatar 62 and the virtual camera in the virtual space changes.

  For example, when a real user approaches the camera 131 and the size of the face FC in the input image 50 increases, the virtual camera in the virtual space approaches the avatar 62 and the avatar 62 is enlarged and displayed. FIG. 9 illustrates the moving image 60 in a state where the avatar 62 is enlarged and displayed as the virtual camera approaches the avatar 62 as compared with the states of FIGS. As a result of the virtual camera approaching the avatar 62, the desk object 64 is removed from the field of view of the virtual camera and is no longer displayed. Further, as shown in the figure, in this example, when the enlargement ratio of the avatar 62 is equal to or greater than a predetermined value (when the distance between the avatar 62 and the virtual camera is equal to or smaller than the predetermined value), the facial expression of the avatar 62 changes. It has become.

  In this way, in this example, the user can easily create a moving image including the avatar 62 that moves following the movement of the body by moving the body in front of the camera 131.

  Here, the determination of presence / absence of command input in step S110 will be described. In this example, the movement of the user's hand corresponding to a predetermined command is preset. For example, the movement of the right hand that the state in which the marker MK1 of the right hand RH is not recognized in the input image 50 continues for a predetermined time (for example, 5 seconds) is set as the movement corresponding to the expression switching command for switching the expression of the avatar 62. ing. For example, when detecting a state in which the marker MK1 is not recognized, the apparatus 10 counts the duration, and when the duration has reached a predetermined time, determines in step S110 that a facial expression switching command has been input (step S110). In step S130, the command is executed. The user can switch the facial expression of the avatar 62 by, for example, continuing the state of holding the right hand for a predetermined time. The switching of the facial expression of the avatar 62 is performed according to a predetermined order for a plurality of facial expressions, for example.

  Further, for example, the movement of the left hand in which the marker MK2 of the left hand LH in the input image 50 reciprocates a predetermined number of times in the left-right direction (for example, three times) controls the movement of the avatar 62 based on the face and both hands arrangement in the input image 50 Is set as a movement corresponding to a control stop command for stopping the operation. For example, when the movement of the marker MK2 reciprocating a predetermined number of times in the left-right direction is detected, the apparatus 10 determines that the control stop command is input in step S110 (YES in step S110), and executes the command (step S130). Various operations that are not preferable to be reflected in the operation of the avatar 62 using the period during which the control of the operation of the avatar 62 based on the input image 50 is stopped, which is realized by the execution of the control stop command. You can continue to create videos while In addition, in the period when the control of the operation of the avatar 62 based on the input image 50 is stopped, the avatar 62 is controlled to perform a predetermined operation, and the sound recording is continued. In addition, in the period when the control of the operation of the avatar 62 based on the input image 50 is stopped, the operation of the avatar 62 based on the input image 50 is performed by an input operation by the user via the information input device such as the touch panel of the moving image creation device 10. Control is resumed.

  As described above, the control of the movement of the avatar 62 based on the face and the arrangement of both hands included in the input image 50 and the input and execution of the command based on the hand movement included in the input image 50 are repeated (in step S140). NO) When the user instructs the end of the creation of the moving image (YES in step S140), the creation of the moving image is terminated. The created moving image is stored in the storage 15 or the like.

  In the above-described example, at least one of the right hand and the left hand is provided with a marker having an appearance different from the marker on the palm side on the back side of the hand. The user's palm is facing or the back of the hand is facing), and the movement (hand orientation) of the avatar 62 may be controlled so that the orientation of the user's hand is reproduced.

  In the example described above, the created video may be distributed live. In this case, for example, the video creation device 10 transmits the created video to the video distribution server in a streaming format, and the video distribution server transmits the video in a streaming format to a plurality of viewer user terminals (smartphones or the like). To deliver. FIG. 10 illustrates a moving-distributed moving image 70. The moving image 70 displays a three-dimensional virtual space 200, and in the virtual space 200, an avatar 62 of a user (distributor) who creates and distributes a moving image is arranged on the stage 204, and a plurality of viewers Each avatar 208 is arranged in the audience area 206. The operation of the distributor's avatar 62 is controlled based on the distributor's face and the arrangement of both hands in the input image 50 described above. In the virtual space 200, the distributor's avatar 62 touches another object (for example, an item (gift) thrown by the viewer's avatar 208 or an inputted comment) placed in the virtual space 200. Anyway. For example, an object placed in the virtual space 200 by the viewer's avatar 208 disappears when the distributor's avatar 62 touches the object (the object is driven out of the screen by the avatar 62). Alternatively, when a distributor's avatar 62 touches an object arranged in the virtual space 200, a predetermined effect (a change in shape, light emission, etc.) may be generated on the object. Furthermore, the operation of the distributor's avatar 62 may be controlled in accordance with an input operation by the distributor for the object. For example, the avatar 62 grips the object by a long press tap operation on the object, and then the object is arranged at the position of the tap operation by a tap operation at an arbitrary position in the virtual space 200 (an arbitrary position of the avatar 62). The object is attached to the position of the tap operation by the tap operation.).

  In the example described above, the markers MK1 and MK2 are provided on both hands of the user. However, markers may also be provided on the user's face. On the contrary, as with the user's face, both hands may be recognized without providing a marker. Further, instead of or in addition to at least a part of the face and both hands, other parts of the user's body may be recognized and used to control the operation of the avatar 62.

  The moving image creating apparatus 10 according to the present embodiment described above creates a moving image including the user's avatar, and the avatar is a predetermined part (for example, a face) of the user's body in an image input via the camera 131. Since the movement is controlled based on the arrangement of the two hands), the user can easily create a moving image including an avatar that moves according to the movement of the body by moving the body in front of the camera 131. Can do. Thus, the embodiment of the present invention supports creation of a moving image including a user's avatar.

  In another embodiment of the present invention, at least a part of the above-described functions of the moving image creating apparatus 10 is a server (for example, the above-described moving image distribution) connected to the apparatus 10 via a communication network such as the Internet. The server) and the moving image creating apparatus 10 cooperate. For example, the moving image creating apparatus 10 may transmit an image input via the camera 131 to a server, and the server side may analyze the received image and create (and distribute) the moving image. good.

  The processes and procedures described in the present specification are implemented by software, hardware, or any combination thereof other than those explicitly described. For example, the processes and procedures described in this specification are realized by mounting logic corresponding to the processes and procedures on a medium such as an integrated circuit, a volatile memory, a nonvolatile memory, and a magnetic disk. The processing and procedure described in this specification can be implemented as a computer program corresponding to the processing / procedure and executed by various computers.

  Even if the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple Component and / or multiple modules. Also, the software and hardware elements described herein can be implemented by integrating them into fewer components or by disassembling them into more components.

  In the present specification, when the constituent elements of the invention are described as one or a plurality, or when they are described without being limited to one or a plurality of cases, they should be understood separately in context. The component may be either singular or plural.

DESCRIPTION OF SYMBOLS 10 Movie creation apparatus 11 Computer processor 131 Camera 41 Information storage management part 43 Image analysis part 45 Movie creation part 50 Input image 60, 70 Movie 62 Avatar

Claims (10)

A system comprising one or more computer processors for creating a video,
The one or more computer processors are responsive to execution of readable instructions,
Processing for recognizing one or more predetermined parts of the user's body included in an image input via the camera;
A process for controlling the operation of the avatar based at least on the arrangement of the recognized predetermined part in the input image;
Executing a process of creating a video including the avatar;
system. The one or more computer processors further execute a process of distributing the created moving image in real time.
The system of claim 1. The predetermined part includes a face and both hands,
The system according to claim 1 or 2. The one or more computer processors further execute a process of executing a predetermined command specified based on at least a part of movement of the predetermined part recognized.
The system according to claim 1. The predetermined command includes a command for stopping control of the operation of the avatar based on the arrangement of the predetermined portion in the input image.
The system of claim 4. The recognizing process includes recognizing the predetermined part by detecting a marker having a predetermined appearance provided in the predetermined part.
The system according to any one of claims 1 to 5. The predetermined part includes a hand,
The recognition process recognizes the orientation of the user's hand by detecting a marker having a first appearance provided on the back side of the user's hand and a marker having a second appearance provided on the palm side of the user's hand. Including
The controlling process includes controlling the movement of the avatar based at least on the orientation of the recognized user's hand,
The system of claim 6. The controlling process includes controlling the movement of the avatar so as to follow the movement of the predetermined part.
The system according to claim 1. A method for creating a video that is executed by one or more computers, comprising:
Recognizing one or more predetermined parts of a user's body included in an image input via a camera;
Controlling the movement of the avatar based at least on the arrangement of the recognized predetermined site in the input image;
Creating a video that includes the avatar,
Method. A program for creating videos,
In response to execution on one or more computers, the one or more computers
A process of recognizing one or more predetermined parts of the user's body included in an image input via the camera;
A process for controlling the operation of the avatar based at least on the arrangement of the recognized predetermined part in the input image;
A process of creating a video including the avatar,
program.

Images