JP2019207509A - Moving image creation system, method, and program - Google Patents

Abstract

To improve entertainment related to output of sound included in a moving image.SOLUTION: A moving image creation apparatus 10 has a function for creating a moving image. The apparatus 10 creates the moving image including an image corresponding to a virtual space and an input sound, converts the input sound into text in response to a touch operation on a predetermined area for displaying the image and arranges the corresponding text object in the virtual space, thereby making it possible to easily create a moving image in which an object corresponding to the input sound is arranged in the virtual space.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system, a method, and a program for creating a moving image.

  Conventionally, a system for a user to distribute a moving image has been provided (see, for example, Patent Document 1). For example, a user shoots a moving image including an image input through a camera included in a user terminal such as a smartphone and a personal computer, and a sound input through a microphone included in the user terminal. It can be distributed to a plurality of viewers.

JP 2017-121036 A

  However, in the above-described conventional system, the audio included in the moving image is only output together with the image included in the moving image, and may not be interesting. In this way, there is room for improvement in the entertainment characteristics of the audio output included in the moving image.

  An object of the embodiment of the present invention is to improve entertainment related to output of audio included in a moving image. Other objects of the embodiments of the present invention will become apparent by referring to the entire specification.

  A system according to an embodiment of the present invention includes one or more computer processors, and is a system for creating a moving image, wherein the one or more computer processors are configured to execute a readable instruction, A process of presenting a screen having a predetermined area for displaying an image corresponding to the virtual space to the user, a process of creating a moving image including an image corresponding to the virtual space and sound input by the user; In response to a touch operation on the predetermined area by the user, the input voice is converted into text, and a text object corresponding to the converted text is placed at a position where the touch operation is performed in the predetermined area. And a process of arranging at a position in the virtual space based on.

  A method according to an embodiment of the present invention is a method for creating a moving image that is executed by one or more computers, and presents a screen having a predetermined area for displaying an image corresponding to a virtual space to a user. And a step of creating a moving image including an image corresponding to the virtual space and a sound input by the user, and the sound input according to a touch operation on the predetermined area by the user Is converted to text, and a text object corresponding to the converted text is arranged at a position in the virtual space based on the position where the touch operation is performed in the predetermined area.

  A program according to an embodiment of the present invention is a program for creating a moving image, and an image corresponding to a virtual space is displayed on the one or more computers in accordance with execution on the one or more computers. A process of presenting a screen having a predetermined area to be displayed to a user; a process of creating a moving image including an image corresponding to the virtual space; and a sound input by the user; and the predetermined area by the user A position in the virtual space based on a position where the touch operation is performed in the predetermined area by converting the input voice into text in response to a touch operation on And processing to be arranged in.

  Various embodiments of the present invention improve entertainment related to the output of audio contained in a video.

1 is a configuration diagram schematically showing a configuration of a moving image creating apparatus 10 according to an embodiment of the present invention. FIG. 2 is a block diagram schematically showing functions of the moving image creating apparatus 10. The figure which illustrates screen 60 for animation creation. The flowchart which illustrates the process which the moving image production apparatus 10 performs in order to control operation | movement of the avatar 102. FIG. The figure which illustrates typically the input image 50 input via an in-camera. The figure for demonstrating a mode that the user's face and both hands which are contained in the input image 50 are recognized. The figure which illustrates the input image 50. The figure which illustrates screen 60 for animation creation. The flowchart which illustrates the process which the moving image production apparatus 10 performs according to the detection of the touch operation with respect to the image display area 62. The figure which illustrates screen 60 for animation creation. The figure which illustrates screen 60 for animation creation. The figure which illustrates screen 60 for animation creation. The figure which illustrates the distributor screen 80.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram schematically showing the configuration of a moving image creating apparatus 10 according to an embodiment of the present invention. The moving image creating apparatus 10 is an example of an apparatus that has a function for creating a moving image and that implements part or all of the system of the present invention.

  The moving image creating apparatus 10 is configured as a general computer, and as shown in FIG. 1, a computer processor 11 such as a CPU or a GPU, a main memory 12, a user I / F 13, a communication I / F 14, A storage (storage device) 15 is provided, and these components are electrically connected via a bus or the like (not shown).

  The computer processor 11 reads various programs stored in the storage 15 or the like into the main memory 12 and executes various instructions included in the programs. The main memory 12 is configured by, for example, a DRAM.

  The user I / F 13 includes various input / output devices for exchanging information with the user. The user I / F 13 includes, for example, an information input device such as a keyboard and a pointing device (for example, a mouse and a touch panel), a voice input device such as a microphone, and an image input device such as a camera. The user I / F 13 includes an image output device such as a display and an audio output device such as a speaker.

  The communication I / F 14 is implemented as hardware such as a network adapter, various types of communication software, and combinations thereof, and is configured to realize wired or wireless communication.

  The storage 15 is configured by, for example, a magnetic disk, a flash memory, or the like. The storage 15 stores various programs including an operating system, various data, and the like. The program stored in the storage 15 may include an application program for realizing a function for creating a moving image (hereinafter, also referred to as “moving image creating application”).

  In the present embodiment, the moving image creation apparatus 10 can be configured as a smartphone, a tablet terminal, a personal computer, a wearable device, or the like.

  Next, functions of the moving image creating apparatus 10 according to the present embodiment will be described. FIG. 2 is a block diagram schematically showing functions of the moving image creating apparatus 10. As shown in the figure, the moving image creating apparatus 10 includes an information storage management unit 41 that stores and manages various information, a moving image creating unit 43 that creates a moving image, and a virtual space control unit 45 that controls a virtual space. These functions are realized by the cooperation of hardware such as the computer processor 11 and the main memory 12 and various programs and data stored in the storage 15, for example, in the main memory 12. This is realized by the computer processor 11 executing instructions included in the read program.

  The information storage management unit 41 stores and manages various information in the storage 15 and the like. The moving image creation unit 43 executes various processes related to the creation of a moving image. In the present embodiment, the moving image creation unit 43 is configured to present a screen having a predetermined area for displaying an image corresponding to the virtual space to the user. For example, the moving image creation unit 43 is configured to display a screen having the predetermined area on a display or the like.

  In addition, the moving image creation unit 43 is configured to create a moving image including an image corresponding to the virtual space and input audio. For example, the moving image creating unit 43 is configured to create (record) a moving image including an image of a virtual space displayed in a predetermined area and sound input via a microphone. The created moving image is stored in the storage 15 or the like, for example.

  The virtual space control unit 45 executes various processes related to the control of the virtual space. In the present embodiment, the virtual space control unit 45 converts input speech into text in response to a user's touch operation on the predetermined area, and places a text object corresponding to the converted text in the virtual space. Is configured to do. The text object is arranged at a position in the virtual space based on the position where the touch operation is performed in a predetermined area.

  As described above, the moving image creating apparatus 10 according to the present embodiment creates a moving image including an image corresponding to the virtual space and the input audio, and the touch operation is performed on a predetermined area in which the image is displayed. Since the input sound is converted into text and the corresponding text object is arranged in the virtual space, it is possible to easily create a moving image in which the object corresponding to the input sound is arranged in the virtual space. That is, the moving image creating apparatus 10 according to the present embodiment improves entertainment related to output of audio included in the moving image.

  In the present embodiment, the virtual space control unit 45 may be configured to convert a voice input during a period from when the touch state with respect to the predetermined area is started to cancellation to text. For example, the virtual space control unit 45 starts recording the input voice in response to the start of the touch state, and converts the recorded voice to text in response to the cancellation of the touch state. Configured to place a text object corresponding to the rendered text. Such a configuration can realize the arrangement of the text object with a simple operation.

  In addition, the virtual space control unit 45 provides a text corresponding to the converted text so that a visual effect based on the direction of the flick operation and / or the slide operation performed after the touch state on the predetermined area is started is given. It can be configured to place objects. For example, when the direction of the flick operation / slide operation performed when the touch state on the predetermined area is canceled is the first direction (for example, the right direction), the virtual space control unit 45 performs the first visual When an effect (for example, fade-in effect) is given to the text object, and the direction of the flick operation / slide operation is the second direction (for example, left direction), the second visual effect (for example, fade-out) Is applied to the text object. With such a configuration, the visual effect can be imparted to the text object with a simple operation.

  In the present embodiment, the virtual space is configured to include an object that displays an image input (captured) via a camera, for example. In this case, the created moving image is configured as a moving image in which a real user appears (appears), for example. Further, the virtual space is configured to include an avatar operated by the user, for example. In this case, the created moving image is configured as a moving image in which an avatar appears instead of a real user, and the virtual space control unit 45 is configured to control the operation of the avatar in the virtual space. In this case, the virtual space can be configured such that the avatar can touch the arranged text object. Such a configuration allows the text object to be touched via the avatar, so that the entertainment of the created moving image can be improved.

  In the case where the user's avatar is included in the virtual space, the virtual space control unit 45 may be configured to control the operation of the avatar according to the operation by the user on the touch panel or the physical controller, for example. Further, the virtual space control unit 45 may be configured to control the operation of the avatar based at least on the posture of the user included in the image input via the camera (for example, in conjunction with the posture of the user). . Detection of the posture (bone) of the user included in the image can be realized by applying a known human posture estimation technique, for example. Further, the virtual space control unit 45 is based at least on an arrangement in the image of one or more predetermined parts (for example, face and both hands) of the user's body included in the input image (for example, the predetermined part). It may be configured to control the behavior of the avatar (as linked to the placement). Such a configuration makes it possible to operate the avatar based on the actual movement of the user.

  Next, a specific example of the moving image creating apparatus 10 of the present embodiment having such a function will be described. The moving image creating apparatus 10 in this example is configured as a smartphone, a tablet terminal, a personal computer, or the like, and a moving image creating application is installed therein. The moving image creating apparatus 10 in this example is configured to create a moving image including an avatar.

  FIG. 3 illustrates a moving image creation screen 60 displayed on the display or the like of the moving image creation apparatus 10. The screen 60 is a screen for creating a moving image. As shown in the drawing, an image display area (predetermined area) 62 for displaying an image included in the moving image to be created, and creation (recording, recording) of the moving image. And an instruction button 64 for instructing the start and end of the operation.

  In the image display area 62, an image obtained by viewing the virtual space 100 with a specific field of view (via a virtual camera at a specific position) is displayed. The virtual space 100 is configured as a three-dimensional virtual space, and includes a humanoid avatar 102 and a desk object 104 positioned in front of the avatar 102.

  Here, the process regarding control of the operation | movement of the avatar 102 contained in the virtual space 100 is demonstrated. FIG. 4 is a flowchart illustrating a process executed by the moving image creating apparatus 10 in order to control the operation of the avatar 102 in this example. For example, the apparatus 10 executes the process illustrated in FIG. 4 according to the display of the moving image creation screen 60.

  First, as shown in FIG. 4, the moving image creating apparatus 10 recognizes the user's face and both hands included in the input image input through the in-camera (step S100). The in-camera is configured to include a user who views the screen displayed on the device 10 in the field of view. The user moves the body in front of the in-camera to operate the avatar 102 while viewing the image of the virtual space 100 included in the image display area 62 of the moving image creation screen 60.

  FIG. 5 schematically illustrates an input image 50 input via the in-camera. As shown in the figure, in this example, a circular marker MK1 of the first color (for example, red) is provided on the palm of the user's right hand RH, and a second marker is provided on the palm of the user's left hand LH. A circular marker MK2 of color (for example, yellow) is provided. These markers MK1 and MK2, for example, are configured as stickers to be attached to the palm of the hand, and the stickers are provided to the user by, for example, a provider of a moving image creation application. The markers MK1 and MK2 are directly drawn with ink or the like on the palm of the hand, for example. In this case, for example, the user draws the markers MK1 and MK2 on the palms of both hands according to instructions distributed by the provider of the moving image creation application.

  FIG. 6 is a diagram for explaining how the user's face and both hands included in the input image 50 illustrated in FIG. 5 are recognized. As shown in the figure, in this example, the user's face FC is detected and recognized as a rectangular detection area DA1 surrounding the outline of the face FC. In addition, the user's hands RH and LH are detected and recognized as rectangular detection areas DA2 and DA3 surrounding the contours of the markers MK1 and MK2 provided on the palms of the hands RH and LH, respectively. Such recognition of the face FC and both hands RH, LH (markers MK1, 2) (and subsequent tracking) is realized using a known object tracking technique, for example, a learned learning generated through machine learning. Realized using a model.

  Returning to the flowchart of FIG. 4, when the user's face and both hands included in the input image are recognized, the moving image creating apparatus 10 next controls the operation of the avatar based on the arrangement of the user's face and both hands in the input image. (Step S110). The control of the avatar movement based on the user's face and the arrangement of both hands in the input image is repeated until the creation of the moving image is completed (for example, until the display of the moving image creation screen 60 is completed) ( NO in step S120).

  In this example, the relative positions of both hands RH and LH (the positional relationship between the actual user's face and both hands) with respect to the position of the user's face FC in the input image 50 input via the in-camera are reproduced. Thus, the operation of the avatar 102 is controlled. For example, as illustrated in FIG. 7, the actual user takes a “many years” posture, and the user's hands RH and LH (markers MK 1 and 2) in the input image 50 move obliquely above the user's face FC. In this case, as illustrated in FIG. 8, the avatar 102 in the virtual space 100 also assumes a “many” posture (moves both hands diagonally above the face) in the same manner as the user.

  When the user selects the instruction button 64, recording of a moving image is started. Specifically, a moving image including an image displayed in the image display area 62 and sound input through a microphone is recorded. When the user selects the instruction button 64 again, the recording of the moving image is stopped. The created moving image is stored in a predetermined area such as the storage 15. Thus, in this example, the user speaks while moving his / her body in front of the in-camera, so that the image corresponding to the virtual space 100 including the avatar 102 that operates following the movement of the body and the user himself / herself Can be easily created.

  Here, an operation related to the arrangement of the text objects in this example will be described. FIG. 9 is a flowchart illustrating processing executed by the moving image creating apparatus 10 in response to detection of a touch operation by the user on the image display area 62 of the moving image creating screen 60. When the touch operation on the image display area 62 is detected, the apparatus 10 first records the input voice input through the microphone until the touch state on the image display area 62 is canceled as shown in the figure ( In step S200 and step S210, NO).

  When the touch state on image display area 62 is resolved (YES in step S210), apparatus 10 stops recording (step S220) and converts the recorded voice into text (step S230). The text conversion of the recorded voice is realized by applying a known voice recognition technique. In this example, the input sound is also included in the created moving image (not muted) in the period from the start of the touch state to the image display area 62 to the cancellation. Note that during the period, sound in the created moving image may be muted.

  Subsequently, the moving image creating apparatus 10 places a text object corresponding to the converted text in the virtual space 100 (step S240). The text object is obtained by converting the converted text as a three-dimensional object in the virtual space 100, and is arranged at a position on the virtual space 100 based on the position where the touch operation is performed on the image display area 62.

  FIG. 10 illustrates the moving image creation screen 60 in a state where the touch operation is performed on the image display area 62 and the touch state is continued. In this case, a circular touch position display object 70 centered on the position where the touch operation is performed is displayed in the image display area 62. In this state, the user inputs (speaks) a voice to be converted into a text object.

  FIG. 11 illustrates the moving image creation screen 60 in a state where the touch state with respect to the image display area 62 is canceled from the state of FIG. 10 and the text object 106 is arranged in the virtual space 100. In the example of FIG. 11, are audio input of "Hello" in the period that the touch state is continued for the image display area 62, text object 106 is configured as an object corresponding to the text "Hello" . In this example, the text object 106 corresponds to the position where the touch operation was performed (the display position of the touch position display object 70) in the example (the object corresponding to the character “K” in the example of FIG. 11). It arrange | positions so that it may become a position in the virtual space 100. The position where the text object 106 is arranged is not limited to this. For example, the text object 106 can be arranged at various positions in the virtual space 100 that can be specified based on the position where the touch operation is performed in the image display area 62.

  In this example, the arranged text object 106 disappears after a lapse of a predetermined time (for example, 5 seconds). The avatar 102 can touch the text object 106. For example, the text object 106 is configured to move independently for each character (configured as a separate object for each character). For example, as shown in FIG. When the object of the character “ko” at the head of the object 106 is touched, only the object can be moved. Thus, the user can move the placed text object 106 via the avatar 102 while placing the text object 106 in the virtual space 100 by a touch operation on the image display area 62. The operation of the placed text object 106 is not limited to this. For example, the text object 106 may be controlled to move downward (drop) at a predetermined speed after being placed. In addition, the entire text object 106 or an object corresponding to each character disappears when the avatar 102 touches, or when the avatar 102 touches, a predetermined effect (change in shape, light emission, etc.) May be generated.

  In this example, when a flick operation is performed when the touch state on the image display area 62 is canceled, a visual effect based on the direction of the flick operation is given to the text object to be arranged. For example, if a flicking operation in the right direction is performed when canceling the touch state, the text object 106 is arranged (displayed) with a fade-in effect, while the text object 106 is moved in the left direction when canceling the touch state. When the flick operation is performed, after the text object 106 is arranged, the text object 106 is deleted with a fade-out effect.

  In the example described above, the created video may be distributed live. In this case, for example, the video creation device 10 transmits the created video to the video distribution server in a streaming format, and the video distribution server transmits the video in a streaming format to a plurality of viewer user terminals (smartphones or the like). To deliver. FIG. 13 exemplifies a distributor screen 80 displayed on the display or the like of the moving image creating apparatus 10 configured as a user terminal or the like of a distributor who performs live distribution of moving images. The screen 80 displays a three-dimensional virtual space 200 on the entire screen. In the virtual space 200, a distributor's avatar 202 is arranged on a stage 204, and each viewer's avatar 208 is a spectator. Arranged in area 206. The avatar 202 is operated by the distributor (for example, controlled based on the distributor's face and the positions of both hands included in the image input via the camera). Further, in response to a touch operation on the distributor screen 80 by the distributor, the input voice is converted into text, and the corresponding text object is arranged in the virtual space 200. The avatar 202 can touch the arranged text object in the virtual space 200. The avatar 202 may be able to touch other objects (for example, items (gifts) thrown by the viewer's avatar 208) placed in the virtual space 200 in the same manner as the text object. .

  In the example described above, the user's avatars 102 and 202 are included in the virtual spaces 100 and 200. However, in another example of the present embodiment, the user's avatar is not included in the virtual space. An object or the like for displaying an input image can be arranged.

  The moving image creating apparatus 10 according to the present embodiment described above creates a moving image including an image corresponding to the virtual space and the input sound, and displays a predetermined area (for example, the moving image creating screen 60). In response to a touch operation on the image display area 62), the input voice is converted into text and the corresponding text object is arranged in the virtual space, so the object corresponding to the input voice is arranged in the virtual space. It is possible to easily create a video to be played. That is, the moving image creating apparatus 10 according to the present embodiment improves entertainment related to output of audio included in the moving image.

  In another embodiment of the present invention, at least a part of the above-described functions of the moving image creating apparatus 10 is a server (for example, the above-described moving image distribution) that is connected to the apparatus 10 via a communication network such as the Internet. The server) and the moving image creating apparatus 10 cooperate. For example, on the server side, analysis of an image input via the camera of the moving image creation device 10, text conversion of audio input via a microphone, creation (and distribution) of a moving image, and the like are performed. You may do it.

  The processes and procedures described in the present specification are implemented by software, hardware, or any combination thereof other than those explicitly described. For example, the processes and procedures described in this specification are realized by mounting logic corresponding to the processes and procedures on a medium such as an integrated circuit, a volatile memory, a nonvolatile memory, and a magnetic disk. The processing and procedure described in this specification can be implemented as a computer program corresponding to the processing / procedure and executed by various computers.

  Even if the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple Component and / or multiple modules. Also, the software and hardware elements described herein can be implemented by integrating them into fewer components or by disassembling them into more components.

  In the present specification, when the constituent elements of the invention are described as one or a plurality, or when they are described without being limited to one or a plurality of cases, they should be understood separately in context. The component may be either singular or plural.

DESCRIPTION OF SYMBOLS 10 Movie creation apparatus 11 Computer processor 41 Information storage management part 43 Movie creation part 45 Virtual space control part 50 Input image 60 Movie creation screen 62 Image display area (predetermined area)
80 Distributor screen 100, 200 Virtual space 102, 202 Avatar 106 Text object

Claims (8)

A system comprising one or more computer processors for creating a video,
The one or more computer processors are responsive to execution of readable instructions,
A process of presenting a screen having a predetermined area for displaying an image corresponding to the virtual space to the user;
A process of creating a moving image including an image corresponding to the virtual space and sound input by the user;
In response to a touch operation on the predetermined area by the user, the input voice is converted into text, and a text object corresponding to the converted text is placed at a position where the touch operation is performed in the predetermined area. A process of arranging at a position in the virtual space based on,
system. The process of arranging includes converting the input voice into text in a period from the start of the touch state to the predetermined area to the cancellation.
The system of claim 1. The arranging process includes arranging the corresponding text object so that a visual effect based on a direction of a flick operation and / or a slide operation performed after a touch state on the predetermined area is started is given. ,
The system according to claim 1 or 2. The one or more computer processors further execute a process of controlling an operation of the avatar operated by the user in the virtual space,
The virtual space is configured so that the avatar can touch the arranged text object.
The system according to claim 1. The process of controlling the movement of the avatar includes controlling the movement of the avatar based on at least the posture of the user included in an image input via a camera.
The system of claim 4. The one or more computer processors further execute a process of distributing the created moving image in real time.
The system according to any one of claims 1 to 5. A method for creating a video that is executed by one or more computers, comprising:
Presenting a screen having a predetermined area for displaying an image corresponding to the virtual space to the user;
Creating a moving image including an image corresponding to the virtual space and sound input by the user;
In response to a touch operation on the predetermined area by the user, the input voice is converted into text, and a text object corresponding to the converted text is placed at a position where the touch operation is performed in the predetermined area. Arranging at a position in the virtual space based on,
Method. A program for creating videos,
In response to execution on one or more computers, the one or more computers
A process of presenting a screen having a predetermined area for displaying an image corresponding to the virtual space to the user;
A process of creating a moving image including an image corresponding to the virtual space and sound input by the user;
In response to a touch operation on the predetermined area by the user, the input voice is converted into text, and a text object corresponding to the converted text is placed at a position where the touch operation is performed in the predetermined area. A process of arranging at a position in the virtual space based on
program.

Images