JP7038602B2 - Systems, methods, and programs for creating videos - Google Patents

Description

The present invention relates to a system, a method, and a program for creating a moving image.

Conventionally, a system for a user to distribute a moving image has been provided (see, for example, Patent Document 1). For example, the user shoots a video including an image input through a camera of a user terminal such as a smartphone or a personal computer and a voice input through a microphone of the user terminal, and captures the shot video. It can be distributed to multiple viewers.

Japanese Unexamined Patent Publication No. 2017-120136

However, in the above-mentioned conventional system, the sound included in the moving image is only output together with the image also included in the moving image, and may be uninteresting. As described above, there is room for improvement in the entertainment property of the audio output included in the moving image.

One of the objects of the embodiment of the present invention is to improve the entertainment property of the output of the sound included in the moving image. Other objects of the embodiments of the present invention will become apparent by reference to the entire specification.

The system according to an embodiment of the present invention comprises one or more computer processors for creating moving images, wherein the one or more computer processors respond to the execution of readable instructions. A process of presenting a screen having a predetermined area for displaying an image corresponding to a virtual space to a user, a process of creating a moving image including an image corresponding to the virtual space and a voice input by the user, and a process of creating a moving image. In response to the touch operation on the predetermined area by the user, the input voice is converted into text, and the text object corresponding to the converted text is placed at the position where the touch operation is performed in the predetermined area. The process of arranging at a position in the virtual space based on the above is executed.

A method according to an embodiment of the present invention is a method executed by one or more computers to create a moving image, and presents a screen having a predetermined area for displaying an image corresponding to a virtual space to a user. A step of creating a moving image including an image corresponding to the virtual space and a voice input by the user, and the input voice in response to a touch operation on the predetermined area by the user. Is converted into text, and the text object corresponding to the converted text is provided at a position in the virtual space based on the position where the touch operation is performed in the predetermined area.

The program according to the embodiment of the present invention is a program for creating a moving image, and depending on execution on one or a plurality of computers, the image corresponding to the virtual space is displayed on the one or a plurality of computers. A process of presenting a screen having a predetermined area to be displayed to a user, a process of creating a moving image including an image corresponding to the virtual space and a voice input by the user, and a process of creating the predetermined area by the user. In response to the touch operation on the computer, the input voice is converted into text, and the text object corresponding to the converted text is placed in the virtual space based on the position where the touch operation is performed in the predetermined area. To execute the process to be placed in.

Various embodiments of the present invention improve entertainment with respect to the output of audio contained in the moving image.

FIG. 6 is a configuration diagram schematically showing a configuration of a moving image creating device 10 according to an embodiment of the present invention. The block diagram which shows the function of the moving image creation apparatus 10. The figure which illustrates the screen 60 for moving image. FIG. 6 is a flow chart illustrating a process executed by the moving image creation device 10 in order to control the operation of the avatar 102. The figure which schematically exemplifies the input image 50 input through the in-camera. The figure for demonstrating how the face and both hands of the user included in the input image 50 are recognized. The figure which illustrates the input image 50. The figure which illustrates the screen 60 for moving image. FIG. 6 is a flow chart illustrating a process executed by the moving image creating device 10 in response to the detection of a touch operation on the image display area 62. The figure which illustrates the screen 60 for moving image. The figure which illustrates the screen 60 for moving image. The figure which illustrates the screen 60 for moving image. The figure which illustrates the distributor screen 80.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a configuration diagram schematically showing a configuration of a moving image creating device 10 according to an embodiment of the present invention. The moving image creation device 10 has a function for creating a moving image, and is an example of a device that implements a part or all of the system of the present invention.

The moving image creation device 10 is configured as a general computer, and as shown in FIG. 1, a computer processor 11 such as a CPU or GPU, a main memory 12, a user I / F13, a communication I / F14, and the like. A storage (storage device) 15 is provided, and each of these components is electrically connected via a bus or the like (not shown).

The computer processor 11 reads various programs stored in the storage 15 and the like into the main memory 12 and executes various instructions included in the programs. The main memory 12 is composed of, for example, a DRAM or the like.

The user I / F 13 includes various input / output devices for exchanging information with the user. The user I / F 13 includes, for example, an information input device such as a keyboard and a pointing device (for example, a mouse, a touch panel, etc.), a voice input device such as a microphone, and an image input device such as a camera. Further, the user I / F 13 includes an image output device such as a display and an audio output device such as a speaker.

The communication I / F 14 is implemented as hardware such as a network adapter, various communication software, and a combination thereof, and is configured to realize wired or wireless communication.

The storage 15 is composed of, for example, a magnetic disk, a flash memory, or the like. The storage 15 stores various programs including an operating system, various data, and the like. The program stored in the storage 15 may include an application program for realizing a function for creating a moving image (hereinafter, may be referred to as a “moving image creating application”).

In the present embodiment, the moving image creating device 10 may be configured as a smartphone, a tablet terminal, a personal computer, a wearable device, or the like.

Next, the function of the moving image creating device 10 of the present embodiment will be described. FIG. 2 is a block diagram schematically showing the functions of the moving image creating device 10. As shown in the figure, the moving image creation device 10 has an information storage management unit 41 for storing and managing various information, a moving image creating unit 43 for creating a moving image, and a virtual space control unit 45 for controlling a virtual space. These functions are realized by the hardware such as the computer processor 11 and the main memory 12, and various programs and data stored in the storage 15 and the like operating in cooperation with each other, for example, in the main memory 12. This is realized by the computer processor 11 executing the instructions included in the read program.

The information storage management unit 41 stores and manages various information in the storage 15 and the like. The moving image creation unit 43 executes various processes related to the creation of the moving image. In the present embodiment, the moving image creation unit 43 is configured to present the user with a screen having a predetermined area for displaying an image corresponding to the virtual space. For example, the moving image creation unit 43 is configured to display a screen having the predetermined area on a display or the like.

Further, the moving image creation unit 43 is configured to create a moving image including an image corresponding to the virtual space and input voice. For example, the moving image creation unit 43 is configured to create (record) a moving image including an image of a virtual space displayed in a predetermined area and a sound input via a microphone. The created moving image is stored in, for example, a storage 15 or the like.

The virtual space control unit 45 executes various processes related to the control of the virtual space. In the present embodiment, the virtual space control unit 45 converts the input voice into text in response to a touch operation on the predetermined area by the user, and arranges a text object corresponding to the converted text in the virtual space. It is configured to do. The text object is placed at a position in the virtual space based on the position where the touch operation is performed in a predetermined area.

As described above, the moving image creating device 10 of the present embodiment creates a moving image including an image corresponding to the virtual space and input voice, and responds to a touch operation on a predetermined area for displaying the image. Since the input voice is converted into text and the corresponding text object is placed in the virtual space, it is possible to easily create a moving image in which the object corresponding to the input voice is placed in the virtual space. That is, the moving image creation device 10 of the present embodiment improves the entertainment property regarding the output of the sound included in the moving image.

In the present embodiment, the virtual space control unit 45 may be configured to convert the voice input in the period from the start of the touch state to the predetermined area to the elimination of the touch state into text. For example, the virtual space control unit 45 starts recording the input voice in response to the start of the touch state, and converts the recorded voice into text in response to the cancellation of the touch state. It is configured to place a text object that corresponds to the recorded text. Such a configuration can realize the arrangement of text objects by a simple operation.

Further, the virtual space control unit 45 is a text corresponding to the converted text so as to give a visual effect based on the direction of the flick operation and / or the slide operation performed after the touch state for the predetermined area is started. It can be configured to place objects. For example, the virtual space control unit 45 has a first visual sense when the direction of the flick operation / slide operation performed when the touch state for a predetermined area is canceled is the first direction (for example, the right direction). If the effect (eg, fade-in effect) is given to the text object, while the direction of the flick / slide operation is in the second direction (eg, left), the second visual effect (eg, fade out). Effect) is configured to be given to the text object. Such a configuration can realize the addition of a visual effect to a text object by a simple operation.

In the present embodiment, the virtual space is configured to include, for example, an object that displays an image input (photographed) via a camera. In this case, the created moving image is configured as, for example, a moving image in which a real user appears (appears). Further, the virtual space is configured to include, for example, an avatar operated by the user. In this case, the created moving image is configured as a moving image in which an avatar appears in place of a real user, and the virtual space control unit 45 is configured to control the operation of the avatar in the virtual space. In this case, the virtual space may be configured so that the avatar can touch the placed text object. Such a configuration allows the text object to be touched through the avatar, which may improve the entertainment of the created video.

When the virtual space includes a user's avatar, the virtual space control unit 45 may be configured to control the operation of the avatar in response to an operation by the user on, for example, a touch panel or a physical controller. Further, the virtual space control unit 45 may be configured to control the movement of the avatar based on at least the posture of the user included in the image input via the camera (for example, to be linked to the posture of the user). .. The detection of the posture (bone) of the user included in the image can be realized by applying, for example, a known person posture estimation technique. Further, the virtual space control unit 45 is based on at least the arrangement in the image of one or a plurality of predetermined parts (for example, face and both hands) of the user's body included in the input image (for example, of the predetermined part). It can be configured to control the behavior of the avatar (to be linked to the placement). Such a configuration makes it possible to operate the avatar based on the movement of the actual user.

Next, a specific example of the moving image creating device 10 of the present embodiment having such a function will be described. The moving image creation device 10 of this example is configured as a smartphone, a tablet terminal, a personal computer, or the like, and an application for creating a moving image is installed. The moving image creation device 10 of this example is configured to create a moving image including an avatar.

FIG. 3 illustrates a moving image creation screen 60 displayed on a display or the like of the moving image creating device 10. The screen 60 is a screen for creating a moving image, and as shown in the figure, an image display area (predetermined area) 62 for displaying an image included in the moving image to be created, and creating (recording, recording) the moving image. It has an instruction button 64 for instructing the start and end of.

In the image display area 62, an image of the virtual space 100 viewed in a specific field of view (via a virtual camera at a specific position) is displayed. The virtual space 100 is configured as a three-dimensional virtual space, and includes a humanoid avatar 102 and a desk object 104 located in front of the avatar 102.

Here, a process related to control of the operation of the avatar 102 included in the virtual space 100 will be described. FIG. 4 is a flow chart illustrating the process executed by the moving image creating device 10 in order to control the operation of the avatar 102 in this example. For example, the device 10 executes the process illustrated in FIG. 4 according to the display of the moving image creation screen 60.

First, as shown in FIG. 4, the moving image creating device 10 recognizes the user's face and both hands included in the input image input via the in-camera (step S100). The in-camera is configured to include the user viewing the screen displayed on the device 10 in the field of view. The user moves his / her body in front of the in-camera to operate the avatar 102 while viewing the image of the virtual space 100 included in the image display area 62 of the moving image creation screen 60.

FIG. 5 schematically illustrates an input image 50 input via the in-camera. As shown in this example, in this example, the palm of the user's right hand RH is provided with a circular marker MK1 of the first color (for example, red), and the palm of the user's left hand LH is provided with a second color. A circular marker MK2 of color (eg, yellow) is provided. These markers MK1 and 2 are configured as, for example, a sticker to be attached to the palm, and the sticker is provided to the user by, for example, a provider of a moving image creation application. Further, the markers MK1 and 2 are drawn directly on the palm, for example, with ink or the like. In this case, for example, the user draws the markers MK1 and MK2 on the palms of both hands according to the instructions distributed by the provider of the video creation application.

FIG. 6 is a diagram for explaining how the user's face and both hands included in the input image 50 illustrated in FIG. 5 are recognized. As shown in the figure, in this example, the user's face FC is detected and recognized as a rectangular detection area DA1 surrounding the contour of the face FC. Further, the user's two-handed RH and LH are detected and recognized as rectangular detection areas DA2 and DA3 surrounding the contours of the markers MK1 and 2 provided on the palms of the two-handed RH and LH, respectively. Such recognition (and subsequent tracking) of face FC and two-handed RH, LH (markers MK1, 2) is realized using known object tracking techniques, for example, trained generated via machine learning. It is realized using a model.

Returning to the flow chart of FIG. 4, when the user's face and both hands included in the input image are recognized, the moving image creating device 10 then controls the operation of the avatar based on the arrangement of the user's face and both hands in the input image. (Step S110). The control of the movement of the avatar based on the arrangement of the user's face and both hands in such an input image is repeated until the creation of the moving image is completed (for example, until the display of the moving image creation screen 60 is completed) (for example, until the display of the moving image creation screen 60 is completed). NO in step S120).

In this example, the relative positions of both hands RH and LH (the positional relationship between the actual user's face and both hands) with respect to the position of the user's face FC in the input image 50 input via the in-camera are reproduced. As described above, the operation of the avatar 102 is controlled. For example, as illustrated in FIG. 7, the actual user takes a “hurray” posture, and the user's two-handed RH and LH (markers MK1, 2) in the input image 50 move diagonally upward of the user's face FC. In this case, as illustrated in FIG. 8, the avatar 102 in the virtual space 100 also takes a “hurray” posture (moves both hands diagonally upward of the face) like the user.

When the user selects the instruction button 64, recording of the moving image is started, and specifically, a moving image including an image displayed in the image display area 62 and a voice input via a microphone is recorded. When the user selects the instruction button 64 again, the recording of the moving image is stopped. The created moving image is stored in a predetermined area such as the storage 15. As described above, in this example, the user speaks while moving his / her body in front of the in-camera, and the image corresponding to the virtual space 100 including the avatar 102 that operates following the movement of the body, and himself / herself. You can easily create a video that includes the audio of.

Here, the operation related to the arrangement of the text object in this example will be described. FIG. 9 is a flow chart illustrating a process executed by the moving image creating device 10 in response to the detection of a touch operation by the user on the image display area 62 of the moving image creating screen 60. Upon detecting the touch operation on the image display area 62, the device 10 first records the input voice input via the microphone until the touch state on the image display area 62 is canceled, as shown in the figure. NO in steps S200 and S210).

Then, when the touch state with respect to the image display area 62 is eliminated (YES in step S210), the device 10 stops recording (step S220) and converts the recorded voice into text (step S230). Text conversion of recorded speech is realized by applying known speech recognition technology. In this example, the input voice is also included in the created moving image (not muted) in the period from the start of the touch state to the image display area 62 to the elimination of the touch state. It should be noted that the sound in the created moving image may be muted during the period.

Subsequently, the moving image creation device 10 arranges a text object corresponding to the converted text in the virtual space 100 (step S240). The text object is such that the converted text is configured as a three-dimensional object in the virtual space 100, and is arranged at a position on the virtual space 100 based on the position where the touch operation with respect to the image display area 62 is performed.

FIG. 10 illustrates a moving image creation screen 60 in which a touch operation is performed on the image display area 62 and the touch state continues. In this case, in the image display area 62, the circular touch position display object 70 centered on the position where the touch operation is performed is displayed. In this state, the user inputs (speaks) the voice to be converted into a text object.

FIG. 11 illustrates a moving image creation screen 60 in a state where the touch state with respect to the image display area 62 is eliminated from the state of FIG. 10 and the text object 106 is arranged in the virtual space 100. In the example of FIG. 11, the voice "hello" is input while the touch state with respect to the image display area 62 is continued, and the text object 106 is configured as an object corresponding to the text "hello". .. In this example, the head of the text object 106 (the object corresponding to the "ko" character in the example of FIG. 11) corresponds to the position where the touch operation was performed (the display position of the touch position display object 70). It is arranged so as to be located in the virtual space 100. The position where the text object 106 is arranged is not limited to this. For example, the text object 106 may be located in various virtual spaces 100 that can be identified based on the position in the image display area 62 where the touch operation was performed.

In this example, the placed text object 106 disappears after a predetermined time (eg, 5 seconds). Further, the avatar 102 can touch the text object 106. For example, the text object 106 is configured to move independently for each character (composed as a separate object for each character), for example, as shown in FIG. 12, with the right hand of the avatar 102, the text. If you touch the object with the character "ko" at the beginning of the object 106, you can move only that object. In this way, the user can move the arranged text object 106 via the avatar 102 while arranging the text object 106 in the virtual space 100 by touching the image display area 62. The operation of the arranged text object 106 is not limited to this. For example, the text object 106 may be controlled to move (fall) downward at a predetermined speed after being placed. Further, the entire text object 106 or the object corresponding to each character disappears when the avatar 102 touches it, or when the avatar 102 touches it, a predetermined effect (shape change, light emission, etc.) May occur.

In this example, when a flick operation is performed when the touch state with respect to the image display area 62 is canceled, a visual effect based on the direction of the flick operation is given to the text object to be arranged. For example, if a flick operation is performed to the right when the touch state is canceled, the text object 106 is placed (displayed) with the effect of fading in, while the text object 106 is placed (displayed) to the left when the touch state is canceled. When the flick operation is performed, after the text object 106 is placed, the text object 106 is erased with the effect of fading out.

In the above example, the created video may be delivered live. In this case, for example, the video creation device 10 transmits the created video to the video distribution server in a streaming format, and the video distribution server sends the video to the user terminals (smartphones, etc.) of a plurality of viewers in the streaming format. To deliver. FIG. 13 illustrates a distributor screen 80 displayed on a display or the like of a moving image creation device 10 configured as a user terminal or the like of a distributor who performs live distribution of a moving image. The screen 80 displays a three-dimensional virtual space 200 on the entire screen, in which the distributor's avatar 202 is arranged on the stage 204, and each avatar 208 of the plurality of viewers is an audience. It is located in area 206. The avatar 202 is manipulated by the distributor (eg, controlled based on the distributor's face and the placement of both hands contained in the image input via the camera). Further, in response to a touch operation on the distributor screen 80 by the distributor, the input voice is converted into text and the corresponding text object is arranged in the virtual space 200. The avatar 202 can touch the arranged text object in the virtual space 200. It should be noted that the avatar 202 may be able to touch other objects (for example, items (gifts) thrown by the viewer's avatar 208) arranged in the virtual space 200 as well as the text object. ..

In the above-mentioned example, the user's avatars 102 and 202 are included in the virtual spaces 100 and 200, but in another example of the present embodiment, the virtual space does not include the user's avatar and is transmitted through the camera. An object or the like that displays the input image may be arranged.

The moving image creation device 10 according to the present embodiment described above creates a moving image including an image corresponding to a virtual space and input voice, and displays a predetermined area (for example, a moving image creating screen 60). In response to the touch operation on the image display area 62) of the above, the input voice is converted into text and the corresponding text object is placed in the virtual space. Therefore, the object corresponding to the input voice is placed in the virtual space. It will be possible to easily create a video to be created. That is, the moving image creation device 10 of the present embodiment improves the entertainment property regarding the output of the sound included in the moving image.

In another embodiment of the present invention, at least a part of the above-mentioned functions of the moving image creation device 10 is a server (for example, the above-mentioned moving image distribution) that is communicably connected to the device 10 via a communication network such as the Internet. It is realized by the cooperation between the server) and the moving image creating device 10. For example, on the server side, analysis of an image input via the camera of the moving image creation device 10, text conversion of voice input via a microphone, creation (and distribution) of a moving image, and the like are performed. You may do so.

The processes and procedures described herein are implemented by software, hardware or any combination thereof, other than those expressly described. For example, the processes and procedures described herein are realized by implementing logic corresponding to the processes and procedures on a medium such as an integrated circuit, a volatile memory, a non-volatile memory, or a magnetic disk. Further, the processes and procedures described in the present specification can be implemented as a computer program corresponding to the processes and procedures, and can be executed by various computers.

Even if it is described that the processes and procedures described herein are performed by a single device, software, component, module, such processes or procedures may be performed by multiple devices, multiple software, multiple devices. Can be performed by a component of, and / or multiple modules. The software and hardware elements described herein can also be realized by integrating them into fewer components or by breaking them down into more components.

In the present specification, even if the components of the invention are described as either singular or plural, or even if they are described without limitation to either singular or plural, they should be understood separately in the context. Except for, the component may be singular or plural.

10 Video creator 11 Computer processor 41 Information storage management unit 43 Video creation unit 45 Virtual space control unit 50 Input image 60 Video creation screen 62 Image display area (predetermined area)
80 Distributor screen 100, 200 Virtual space 102, 202 Avatar 106 Text object

Claims (8)

A system for creating moving images with one or more computer processors.
The one or more computer processors said, depending on the execution of readable instructions,
A process of presenting a screen having a predetermined area for displaying an image corresponding to a virtual space to a user,
A process of creating a moving image including an image corresponding to the virtual space and a voice input by the user.
In response to the touch operation on the predetermined area by the user, the input voice is converted into text, and the text object corresponding to the converted text is placed at the position where the touch operation is performed in the predetermined area. Executes the process of arranging the object at a position in the virtual space based on the above.
system. The arranging process includes converting the input voice into text in the period from the start of the touch state to the predetermined area to the elimination of the touch state.
The system of claim 1. The arranging process includes arranging the corresponding text object so as to give a visual effect based on the direction of a flick operation and / or a slide operation performed after the touch state for the predetermined area is started. ,
The system of claim 1 or 2. The one or more computer processors further execute a process of controlling the operation of the avatar operated by the user in the virtual space.
The virtual space is configured so that the avatar can touch the arranged text object.
The system according to any one of claims 1 to 3. The process of controlling the movement of the avatar includes controlling the movement of the avatar based on at least the posture of the user included in the image input through the camera.
The system of claim 4. The one or more computer processors further execute a process of delivering the created moving image in real time.
The system according to any one of claims 1 to 5. A method for making videos, run by one or more computers.
A step of presenting the user with a screen having a predetermined area for displaying an image corresponding to the virtual space, and
A step of creating a moving image including an image corresponding to the virtual space and a voice input by the user.
In response to the touch operation on the predetermined area by the user, the input voice is converted into text, and the text object corresponding to the converted text is placed at the position where the touch operation is performed in the predetermined area. A step of arranging a position in the virtual space based on the above.
Method. A program for creating videos
Depending on the execution on one or more computers, the one or more computers
A process of presenting a screen having a predetermined area for displaying an image corresponding to a virtual space to a user,
A process of creating a moving image including an image corresponding to the virtual space and a voice input by the user.
In response to the touch operation on the predetermined area by the user, the input voice is converted into text, and the text object corresponding to the converted text is placed at the position where the touch operation is performed in the predetermined area. To execute the process of arranging the object at the position in the virtual space based on the above.
program.

Images