JP2022047036A - Moving image creation device, moving image creation system, and moving image creation program - Google Patents

Abstract

To provide a moving image creation device, a moving image creation system, and a moving image creation program that support creation of moving images in which an avatar moves and sings with simple and little effort.SOLUTION: A moving image creation device for creating a moving image including an avatar includes: basic information acquisition means for acquiring basic information about the avatar; schedule information generation means for generating at least part of schedule information in which a plurality of operation instructions for causing the avatar to perform predetermined operations are arranged in accordance with the elapse of time; voice information acquisition means for acquiring voice information; and moving image data generation means for generating moving image data based on the basic information, the schedule information, and the voice information. As the schedule information generation means, schedule information automatic generation means for automatically generating at least part of the schedule information is provided so that the mouth of the avatar moves in synchronization with the voice.SELECTED DRAWING: Figure 1

Description

Application for application of Article 30, Paragraph 2 of the Patent Law Published on the website Publication date: July 6, 2nd year Reiwa Publication address: https: // www. youtube. com / watch? v = gv6s3xzZY7s

Application for application of Article 30, Paragraph 2 of the Patent Law Published on the website Distribution date: July 7, Reiwa 2 Publication address: https: // apps. apple. com / jp / app / id15135338556

Application for application of Article 30, Paragraph 2 of the Patent Law Published on the website Publication date: July 7, 2nd year Reiwa Publication address: https: // www. copyperoid. com /

Application for application of Article 30, Paragraph 2 of the Patent Law Published on the website Publication date: July 15, 2nd year Reiwa Publication address: https: // www. youtube. com / watch? v = vkyy_xSvj1M

Application for application of Article 30, Paragraph 2 of the Patent Law Published on the website Date of publication: July 21, 2nd year of Reiwa Publication address: https: // SEGA. jp / product / copyperoid /

Application for application of Article 30, Paragraph 2 of the Patent Law Published on the website Date of publication: July 21, 2nd year of Reiwa Publication address: https: // www. Segatoys. co. jp /

Application for application of Article 30, Paragraph 2 of the Patent Law Sega Toys Co., Ltd. invented "Video Creation Device, Video Creation System and Video Creation Program" by Naoko Miyazaki for each media on July 21, 2nd year of Reiwa. I sent a news release document containing the above by e-mail or fax.

Application for application of Article 30, Paragraph 2 of the Patent Law Published on the website Date of publication: July 26, 2nd year of Reiwa Publication address: https: // twitter. com / SEGA_OFFICIAL /

The present invention relates to a video creation device, a video creation system, and a video creation program for creating a video in which an avatar is displayed.

Conventionally, the video creation system described in Patent Document 1 and the like, which supports the creation of a video with voice sung by an avatar by setting avatar information and voice information to which motion information is added and synthesizing each information, has been conventionally used. It is known.

Japanese Unexamined Patent Publication No. 2011-133882

In Document 1, by inputting avatar information, video information, melody information, and lyrics information input by the user from the terminal device, it is possible to smoothly create a video with a voice sung by the avatar. However, in order to move the avatar more humanly, it is necessary to input more motion information, and there is a problem that it takes time and effort to improve the quality of the moving image.

An object of the present invention is to provide a video creation device, a video creation system, and a video creation program that support the creation of a video in which an avatar moves or sings easily and with little effort.

In order to solve the above problems, the moving image creating device of the present invention is a moving image creating device that creates a moving image including an avatar, and obtains basic information about the avatar from its own storage unit or another via a network. Basic information acquisition means acquired from a computer, schedule information generation means for generating at least a part of schedule information in which a plurality of operation instructions for causing the avatar to perform a predetermined operation are arranged in accordance with the passage of time, and voice information. Based on the voice information acquisition means acquired from the own recording means or the own storage unit, or from another computer via the network, the basic information, the schedule information, and the voice information. The avatar is provided with a video data generation means for generating video data including a video with audio that performs a plurality of actions with the passage of time, and the mouth of the avatar synchronizes with the voice as the schedule information generation means. It is characterized in that a schedule information automatic generation means for automatically generating at least a part of the schedule information is provided so as to operate.

A recording means for recording the user's voice is provided, and the user's voice recorded by the recording means is used as the voice information acquired by the voice acquisition means.

It is characterized by providing a synthetic voice generation means capable of changing a user's voice information recorded by the recording means into a synthetic voice.

A music information acquisition means for acquiring music information used for video creation from its own storage unit or from another computer via a network is provided, and the voice information acquisition means is a music acquired by the music information acquisition means. It is characterized by being configured to acquire voice information from information.

The music information includes lyrics information and voice information of the music, and a lyrics information changing means for editing the lyrics information of the music selected by the voice information acquisition means is provided, and the schedule information automatic generation means is the lyrics information. When the lyrics information is edited by the changing means, it is characterized in that it is configured to change the movement of the avatar's mouth according to the changed lyrics.

The voice information acquisition means is characterized in that it can acquire voice information regarding the type of voice uttered by the avatar from its own storage unit or from another computer via a network. There is.

The schedule information generated by the schedule information generation means is characterized by including position information in which a plurality of information regarding left and right positions in a screen displaying the avatar are arranged in accordance with the passage of time.

The schedule information generated by the schedule information generation means is characterized by including scale information in which a plurality of information regarding the size of displaying the avatar is arranged in accordance with the passage of time.

The avatar is configured to have three-dimensional information, and the schedule information generated by the schedule information generation means is rotation position information in which a plurality of information regarding the direction in which the avatar is displayed are arranged in accordance with the passage of time. It is characterized by including.

Further, the moving image creation system according to the embodiment of the present invention includes a storage unit, basic information acquisition means for acquiring basic information about the avatar from the storage unit, and an operation instruction for causing the avatar to perform a predetermined operation. A schedule information generation means for generating at least a part of schedule information arranged in a plurality of cases according to the passage of time, a voice information acquisition means for acquiring voice information from its own recording means or the storage unit, the basic information, and the basic information. The schedule is provided with a moving image data generation means for generating moving image data including an image with sound in which the avatar performs a plurality of actions with the passage of time based on the schedule information and the audio information. As the information generation means, a schedule information automatic generation means for automatically generating at least a part of the schedule information is provided so that the mouth of the avatar operates in synchronization with the voice.

Further, the moving image creation program according to the embodiment of the present invention includes a basic information acquisition process of acquiring basic information about the avatar from its own storage unit or acquiring basic information from another computer via a network. A schedule information generation process that generates at least a part of schedule information in which a plurality of operation instructions for causing an avatar to perform a predetermined operation are arranged in accordance with the passage of time, and voice information is stored in one's own recording means or one's own storage. Based on the voice information acquisition process acquired from the unit or from another computer via the network, the basic information, the schedule information, and the voice information, the avatar has a plurality of avatars over time. As the video data generation process for generating video data including a video with audio for operation and the schedule information generation process, at least a part of the schedule information is used so that the mouth of the avatar operates in synchronization with the audio. It is characterized by executing the schedule information automatic generation process that is automatically generated.

As the schedule information generation means, the schedule information automatic generation means for automatically generating at least a part of the schedule information is provided so that the mouth of the avatar operates in synchronization with the voice, so that the avatar can be easily and with little effort. You can create videos that make your avatar more human.

By configuring the schedule information generation means so that information on the display position, scale, and rotation position of the avatar can be set as the schedule information set according to the passage of time in the moving image, the avatar can be easily operated. Can be moved more dynamically in the video.

It is a block diagram which shows the example of the structure of the moving image creation processing apparatus of this invention. It is a block diagram which shows an example of the system configuration corresponding to the embodiment of this invention. It is a flowchart which shows the example of the moving image creation process corresponding to at least one of the Embodiments of this invention. It is a figure which showed the example of the setting operation screen for acquiring the basic information. It is a figure which showed the example of the setting operation screen for setting schedule information. It is a figure which showed the example of the avatar display edit screen. It is a flowchart which shows the example of the moving image creation process corresponding to at least one of the embodiments of another Example. It is a figure which showed the example of the operation screen for acquiring the music information by the music information acquisition means. It is a figure which showed the example of the setting operation screen for setting the schedule information in another embodiment.

Hereinafter, the moving image creation processing device according to the embodiment of the present invention will be described. FIG. 1 is a block diagram showing an example of the configuration of the moving image creation processing device of the present invention. As shown in FIG. 1, the moving image creation processing device (moving image creating device) 10 includes a background information acquisition means 11, a basic information acquisition means 12, an audio information acquisition means 13, a schedule information manual generation means 14, and a schedule. The information automatic generation means 15, the moving image data generation means 16, and the storage unit 17 are provided.

FIG. 2 is a block diagram showing an example of a system configuration corresponding to an embodiment of the present invention. In the illustrated example, the terminal device 301 ... 30n used by the user is configured to be connectable to the server device 20 via the communication network 40, and the function of the moving image creation processing device 10 is on the server device 20 side. It is configured to aggregate a part of. The moving image creation processing device 10 may be realized in one terminal device 301 and used offline.

The server device 20 is managed by a system administrator and is configured by an information processing device such as a WWW server. Further, the server device 20 includes a control unit for executing various processes, a storage medium for storing various information, a communication unit, and a general configuration as a computer, and various processes are performed on a plurality of terminal devices. It has various functions to provide information about.

The system configuration is not limited to the above example, and one terminal device functioning as the moving image creation processing device 10 may be used by a plurality of users, or may be a configuration including a plurality of server devices.

The plurality of terminal devices 301 ... 30n are information processing terminals such as smartphones and tablet terminals having input means and output means composed of touch-operable touch panels, each of which is connected to a communication network 40 and is connected to the server device 20. It is equipped with hardware and software for executing various processes by communicating with. It should be noted that each of the plurality of terminal devices 301 ... 30n may be configured to be able to directly communicate with each other without going through the server device 20.

The background information acquisition means 11 has a function of acquiring image data or moving image data used as a background of a moving image to be created. The image data or moving image data used as the background may be acquired from the storage unit of the terminal device 301 ... 30n used by the user, or may be acquired from another computer via the communication network 40. Further, the image data or the moving image data may be photographed and acquired on the spot by a photographing means (not shown) such as a camera provided in the terminal device 301 ... 30n used by the user.

The basic information acquisition means 12 has a function of acquiring basic information regarding the appearance of an avatar to be displayed in a moving image to be created. In this embodiment, the basic information acquisition means 12 presents to the user a plurality of types of designs for each configuration of the avatar's hairstyle, costume, and color as basic information regarding the avatar, and the user selects an arbitrary design for each configuration. By doing so, it is possible to create an original avatar according to the user's preference. In this embodiment, a three-dimensional avatar is generated. An example of a specific setting operation of the basic information constituting the avatar will be described later.

The voice information acquisition means 13 has a function of acquiring voice information emitted by an avatar to be displayed in a moving image to be created. In this embodiment, the voice information acquisition means 13 is recorded by a recording means for recording a voice emitted by the user by using a microphone or the like provided in the terminal device 301 ... 30n used by the user, and the recording means. It has a synthetic voice generation means for editing the user's own voice information into a synthetic voice, and uses the user's voice information recorded by the recording means or the voice information obtained by editing the voice information into a synthetic voice for creating a moving image. It is configured so that it can be acquired as voice information.

The schedule information manual generation means 14 causes the avatar displayed in the video to be created to perform a predetermined operation, the telop information displayed in the video, the effect information, the voice information, and the video. It has a function to generate schedule information in which a plurality of music information related to music (BGM, etc.) is displayed side by side according to the passage of time of a moving image to be created, and the schedule information can be input and edited by a user operation. Has been done. An example of the schedule information setting operation will be described later.

The schedule information automatic generation means 15 automatically generates operation information (schedule information) for causing an avatar displayed in a moving image to perform a predetermined operation based on the schedule information acquired by the schedule information manual generation means 14. It has a function to generate. Specifically, the schedule information automatic generation means 15 automatically moves the mouth constituting the face of the avatar according to the voice information composed of the user's voice acquired by the recording means (lip-sync). ) Acquired by the lip-sync means (function), the blinking means (function) that automatically blinks the eyes constituting the face of the avatar at a predetermined timing based on the voice information, the effect information, etc., and the recording means. There is a synthetic voice setting means (function) for changing the user's own voice to a specific voice sound (synthetic voice).

The video data generation means 16 synthesizes (renders) all of the background information, basic information, audio information, schedule information, etc. set for creating a video including an avatar, so that the avatar sings or sends a message. It has a function of generating a moving image and storing it in the storage unit 17.

The storage unit 17 has a function of storing information necessary for processing of each means (each unit) in the moving image creation processing device 10, and also storing various information generated by the processing of each means.

Next, the flow of the moving image creation processing in the moving image creation processing apparatus 10 corresponding to the embodiment of the present invention will be described with reference to FIG. FIG. 3 is a flowchart showing an example of a moving image creation process corresponding to at least one of the embodiments of the present invention. In the illustrated example, the moving image creation process proceeds to step S101 when the execution of the moving image creation project is started.

In step S101, the background information acquisition means 11 acquires image data or moving image data used for the background of the moving image (executes the background information acquisition process), and then proceeds to step S102. In step S102, the basic information acquisition means 12 acquires basic information about the avatar displayed in the moving image (executes the basic information acquisition process), and then proceeds to step S103.

In step S103, the schedule information is generated (schedule information generation processing is executed) by the schedule information manual generation means 14 and the schedule information automatic generation means 15, and then the process proceeds to step S104. In step S104, a message video by an avatar is generated (video) by synthesizing (rendering) various information set by the background acquisition means 11, the basic information acquisition means 12, and the schedule information generation means 14 and 15. Execute the generation process), and then end the process.

The schedule information generation process includes a schedule information manual generation process that provides an environment for manually editing schedule information, and a schedule information automatic generation process that automatically generates other schedules based on the input of schedule information. Have. By the way, the order of various processes constituting the above process flow is random as long as there is no contradiction in the process contents.

Next, an example of an acquisition method of information necessary for creating a moving image will be described with reference to FIGS. 4 to 6. FIG. 4 is a diagram showing an example of a setting operation screen for acquiring basic information. The basic information acquisition means 12 provides an operation environment (hereinafter, basic information input screen 20) for setting basic information about an avatar on a terminal device (touch panel) 301 ... 30n on the user side (see FIG. 4). The process of acquiring the basic information about the avatar via the basic information input screen 20 is configured to be executable.

In the illustrated example, as the basic information input screen 20 displayed on the touch panel on the user side, an avatar display unit 21 that displays an avatar generated based on the set basic information and a component that can change the appearance of the avatar. The element switching operation unit 22 for switching (basic information), the design selection operation unit 23 for displaying the design (basic information) that can be changed in the selected component side by side, and the basic information about the selected avatar. The save operation unit 24 for saving is displayed at the same time (see FIG. 4).

The element switching operation unit 22 includes a strip-shaped display unit in the left-right direction and tab-shaped switching operation units 22A, 22B, 22C arranged side by side along the display unit. In the illustrated example, the hairstyle switching operation unit 22A, the costume switching operation unit 22B, and the color switching operation unit 22C related to the avatar are arranged side by side in the left-right direction on the display unit, and the user can use the display unit. By selecting and operating any one of the switching operation units 22A, 22B, and 22C by tapping or the like, the design selection operation unit 23 can display information about the selected switching operation unit 22. It is configured (see FIG. 4).

Specifically, when the switching operation unit 22A for the hairstyle is selected, a plurality of basic information (designs) regarding the hairstyle are displayed side by side on the design selection operation unit 23, and the switching operation unit 22B for the costume is displayed. When is selected, a plurality of basic information (designs) related to costumes are displayed side by side on the design selection operation unit 23, and when the color switching operation unit 22C related to the avatar is selected, the design selection operation is performed. A plurality of basic information (color information) regarding the main color of the avatar is displayed side by side in the unit 23.

Further, the element switching operation unit 22 slides the band-shaped display unit in the left-right direction (flick operation) so that the switching operation unit 22 displayed in the display unit slides in the display unit. It may be configured. As a result, more types of switching operation units 22 can be arranged in the display unit. According to the configuration, as basic information (element switching operation unit 22) constituting the avatar whose design can be selected, in addition to the above example, the race, gender, etc. of the avatar, as well as the contour, eyes, and nose constituting the face. , Mouth, physique, etc. can be finely selected. The types of the switching operation unit 22 (basic information) that can be selected are not limited to these.

In the design change operation unit 23, among the components of the avatar selected by the element switching operation unit 22, the window-shaped design display unit 25 on which the changeable design is displayed is arranged side by side. One design display unit 25A selected by a user's touch operation or the like intuitively confirms which design display unit is selected by highlighting (thickening or blinking) the outer frame. It is configured so that it can be done (see FIG. 4).

By the way, in the design display unit 25, a design display unit (design) 25B that cannot be selected by displaying a light ink display, a display with a key, etc. 26 is arranged, and the server device recognizes that the predetermined charge amount has been paid. On the condition that the design is not selectable, the design display unit may be opened and used (see FIG. 4).

The avatar display unit 21 can display an avatar that reflects all the designs (basic information) of each component of the avatar selected by the element switching operation unit 22 and the design change operation unit 23. .. The avatar displayed on the avatar display unit 21 is configured so that when the selection of the design display unit 25 is changed, the design of the changed part is immediately reflected, so that the user's favorite avatar can be used. It can be created efficiently.

With the above configuration, the appearance of the avatar to be displayed in the created moving image can be determined by acquiring the basic information about the avatar by the basic information acquisition means 12.

Next, an example of a method of generating schedule information by the schedule information manual generation means will be described with reference to FIG. FIG. 5 is a diagram showing an example of a setting operation screen for manually setting schedule information. The schedule information manual generation means 14 provides an operation environment (hereinafter, schedule information input screen 30) for setting the schedule information on the terminal device (touch panel) 301 ... 30n on the user side (see FIG. 5). The process of acquiring schedule information via the schedule information input screen 30 is configured to be executable.

In the illustrated example, the schedule information input screen 30 on the user's touch panel is a strip-shaped timeline display unit 32 in the left-right direction and a vertical direction in which the display position is fixed at the center of the timeline display unit 32 in the left-right direction. The display reference line 33, the edit screen display unit 31 that displays the image of the playback position of the moving image specified by the display reference line 33 and the timeline display unit 32, and the schedule edit arranged on the display reference line 33. The track 34 and the track 34 are displayed at the same time (see FIG. 5).

In the timeline display unit 32, the playback time of moving images is described in order on the strip-shaped display unit side by side in the left-right direction (every 10 seconds in the illustrated example), and the timeline display unit 32 or the schedule editing track 34. By sliding the display portion of the above in the left-right direction (flick operation) or the like, the display of the reproduction time in the timeline display unit 32 is configured to slide and move.

At this time, the playback time (00:10 seconds in the illustrated example) displayed in the center of the timeline display unit 32 in the left-right direction on the edit screen display unit 31 is shown on the display reference line 33. It is configured to display the state of the reproduction time on the timeline display unit 32.

In the illustrated example, the schedule editing track 34 is predetermined to the telop editing track 36 for editing the telop displayed in the moving image, the background editing track 37 for editing the image data and the moving image data as the background of the moving image, and the avatar. An avatar editing track 38 that edits operation instructions and the like for executing the operation of, an effect editing track 39 that edits the effect in the video, and a voice editing track (voice information) that acquires and edits voice information attached to the video. The acquisition means) 41 and the music editing track 42 for attaching a BGM to the moving image are configured to be displayed side by side in the vertical direction. Further, each edit track 34 is configured so that an edit block 40 indicating that the schedule information has been input is displayed in the reproduction range in which the schedule information is input (see FIG. 5).

Further, the schedule editing track 34 is configured to be integrally slid left and right together with the notation of the reproduction time of the timeline display unit 32 by a slide operation or the like by the user, and is in the center of each editing track in the left-right direction. , The display reference line 33 fixedly displayed at the center of the left and right is arranged. Along with this, each editing track is configured to be able to add or edit each schedule information at the reproduction position displayed by the timeline display unit 32 and the display reference line 33.

Specifically, in the example shown in FIG. 5, the telop editing track 36 does not have the editing block 40 at the time when the playback time starts 10 seconds (the schedule information is not set). At the right end of the telop editing track 36, an additional button 43 for adding schedule information regarding the telop is arranged (see FIG. 5). Similarly, in the background editing track 37, since the editing block 40 is present (the schedule information is set) when the playback time starts 10 seconds, the schedule information regarding the background is at the right end of the background editing track. An edit button 44 for editing (changing / deleting, etc.) is arranged (see FIG. 5).

Incidentally, the editing block 40 is provided with an editing range operation unit 40a on the end side thereof so that the length of the editing block in the left-right direction can be edited by a slide operation. The user can easily set the designated range of the editing block 40 on each editing track by long-pressing the editing range operation unit 40a and performing an operation (drag operation) of grasping and sliding the editing range operation unit 40a by a tap operation or a click operation. Can be edited to. As a result, it is not necessary to specify a numerical value or the like for specifying the range of each editing block, so that the editing work becomes easier and smoother.

In the voice editing track 41, the voice emitted by the user can be recorded by a recording means provided in the user's terminal device or the like and acquired as the voice information. In other words, the recorded user's voice can be used as a message, song, or the like spoken by the avatar in the video. Further, the acquired voice information is configured to be edited into a synthetic voice so that the voice of whom is not known by the synthetic voice generation means.

Further, when the voice information to be played in the moving image is acquired by the voice editing track 41, the operation information of the mouth of the avatar is automatically obtained by the schedule information automatic generation means 15 based on the acquired voice information. It is configured to perform the lip-sync process that is generated in. In the lip sync process, for example, the shape of the mouth of the avatar when pronouncing a vowel (a i u e o) is prepared in advance, and the timing at which the vowel is uttered is acquired from the acquired voice information. By combining these information, the shape of the mouth of the avatar may be configured to move in an appropriate shape at an appropriate timing. The processing method for moving the mouth of the avatar by the lip-sync processing is not limited to these.

Further, when the add button 43 or the edit button 44 of the avatar edit track 38 is pressed, the avatar display edit screen 50 for editing the schedule information related to the avatar display on the user's terminal device (touch panel). Is configured to be displayed (see FIG. 6). Hereinafter, a specific description will be given.

FIG. 6 is a diagram showing an example of an avatar display editing screen. As shown in the figure, on the avatar display editing screen 50, the editing screen display unit 51 that displays the state of the moving image at the playback position being edited and information on the components of the avatar's operation are displayed in a tab format in the left-right direction. The avatar operation element switching operation unit 52 displayed side by side is displayed.

In the illustrated example, the avatar operation element switching display unit 52 has a pose switching unit 52A for setting information (schedule information) regarding the pose of the avatar and a facial expression switching unit for setting information regarding the facial expression of the avatar (schedule information). A unit 52B and an avatar display position switching unit 52C for setting information (schedule information) regarding the display position of the avatar in the moving image are provided.

When the operation tab of the avatar display position switching unit 52C is selected, the information regarding the display position of the avatar in the editing block 40 being edited is edited by an operation tool (slider operation tool) composed of a plurality of slide bars. It is configured to be able to. Specifically, as shown in FIG. 6, as the slider operation tool, a position information setting slider 57 for operating the left and right positions of the avatar in the video, and a scale information setting for operating the size of the avatar in the video. The slider 58 and the rotation information setting slider 59 that operates the rotation position of the avatar expressed three-dimensionally are arranged side by side in the vertical direction (see FIG. 6).

According to each slider operation tool, the position, size, and orientation of the avatar can be freely set in the created moving image, so that the avatar can be moved more dynamically. Specifically, it is possible to make the avatar appear to be in a three-dimensional space by displaying the avatar in a small scale while walking the avatar in the depth direction of the screen in the video.

Next, another embodiment of the moving image creation processing apparatus will be described with reference to FIGS. 7 to 9 and different from the above-mentioned example. In another embodiment, as the voice information acquisition means 13, instead of the recording means described above, data (music information) of the music sung by the avatar in the moving image is stored from the storage unit 17 of the user's terminal device 301 ... 30n. A music information acquisition means 71 is provided for acquiring music information or acquiring music information from another computer via a network. Further, instead of the above-mentioned synthetic voice generation means, a voice information acquisition means 72 for acquiring the type of voice (synthetic voice) when the avatar sings a song is provided.

Incidentally, in the music information acquired by the music information acquisition means 71, the lyrics information of the music can be acquired at the same time as the voice information when the avatar sings the music.

Along with this, the schedule information automatic generation means 15 matches the music information acquired by the music information acquisition means 71, specifically, the lyrics information included in the music information, and the mouth of the avatar in the moving image. You can automatically set the movement information of the avatar's mouth so that you can move. In other words, by selecting the music sung by the avatar by the music information acquisition means 71, it is not necessary to separately input information regarding the movement of the mouth of the avatar. Therefore, the avatar in the video can be moved more humanly with less effort.

FIG. 7 is a flowchart showing an example of the moving image creation process corresponding to at least one of the embodiments of another embodiment. In the illustrated example, the moving image creation process proceeds to step S201 when the execution of the moving image creation project is started.

In step S201, the voice information acquisition means 13 (music information acquisition means 71 and voice information acquisition means 72) acquires music information and voice information to be sung by the avatar in the moving image, and then proceeds to step S202.

In step S202, image data or moving image data used for the background of the moving image created by the background information acquiring means is acquired (background information acquisition processing is executed), and then the process proceeds to step S203. In step S203, basic information about the avatar displayed in the moving image created by the basic information acquisition means is acquired (basic information acquisition process is executed), and then the process proceeds to step S204.

In step S204, the schedule information generation means generates the schedule information (executes the schedule information generation process), and then proceeds to step S205. In step S205, a message video by the avatar is generated (execution of the video generation process) based on various information set by the background acquisition means, the basic information acquisition means, and the schedule information generation means, and then the process is terminated. do.

That is, in the voice information acquisition means 13 of another embodiment, the music information is acquired by the music information acquisition means 71 before the schedule information is edited by the schedule information manual generation means 14, and the avatar is displayed in a moving image. It is configured to simultaneously acquire the voice information to be sung and the lyrics information to be uttered by the avatar. According to the configuration, by first acquiring the basic information of the avatar and the music information sung by the avatar, the time and effort for inputting the schedule information can be significantly reduced.

FIG. 8 is a diagram showing an example of an operation screen for acquiring music information by the music information acquisition means. The voice information acquisition means 13 provides an operation environment (hereinafter, music information setting screen 70) for inputting the music information and the voice information to the terminal device (touch panel) on the user side (see FIG. 8). The process of acquiring the music information and the voice information via the music information setting screen 70 is configured to be executable.

In the illustrated example, the music information setting screen 70 has a radio button for selecting one from a plurality of songs displayed side by side, and a radio button for selecting one from a plurality of voice types displayed side by side. Are arranged side by side, but they are not limited to these as long as one is selected from multiple options.

Further, the voice information acquisition means 13 increases the types of music information that can be selected and the types of voice information (synthetic voice) on condition that the server device 20 recognizes that the predetermined billing amount has been paid. It may be configured so that

FIG. 9 is a diagram showing an example of a setting operation screen for setting schedule information in another embodiment. In the illustrated example, as the schedule information input screen 30 on the touch panel on the user side, the edit screen display unit 31, the timeline display unit 32, the display reference line 33, and the schedule are the same as in the above example. The edit track 34 and the edit track 34 are displayed at the same time (see FIG. 9).

The schedule editing track 34 includes the telop editing track 36, a lyrics editing track 81 for editing lyrics information acquired from the music information, a dance editing track 82 for designating a dance motion to the avatar, and the avatar. The editing track 38, the effect editing track 39, and the background editing track 37 are configured to be displayed side by side in the vertical direction.

The lyrics editing track 81 is configured such that the lyrics information included in the music information acquired by the music information acquisition means is input in advance, but the user can edit a part of the lyrics to make a substitute song. Has been done.

Along with this, when the lyrics information is edited by the lyrics editing track 81, the schedule information automatic generation means 15 uses the voice information included in the acquired music information as the edited voice. It is configured to execute a synthetic voice generation process that automatically generates (edits) synthetic voice so that it becomes information.

Further, when the synthetic voice constituting the voice information is edited by editing the lyrics information, the schedule information automatic generation means 15 moves in the moving image according to the lyrics of the edited parody. It is configured to be able to perform a lip-sync process that automatically corrects the movement of the avatar's mouth.

That is, when the lyrics information is edited by the lyrics editing track 81, the schedule information automatic generation means 15 includes a part of the schedule information, specifically, the voice information sung by the avatar in the moving image and the avatar. It is configured to automatically edit the motion information related to the movement of the mouth according to the content of the edited lyrics. Therefore, it is possible to edit the music selected by the user into a favorite parody and generate a moving image to be sung by the avatar with smoother and higher quality.

The dance editing track 82 is configured such that the pose operation of the avatar can be specified in the avatar editing track 38, but the dance motion according to the music can be specified.

12 Basic information acquisition means 13 Voice information acquisition means 14 Schedule information manual generation means (schedule information generation means)
15 Schedule information automatic generation means (schedule information generation means)
16 Video data generation means 17 Storage unit 40 Communication network 71 Music information acquisition means 72 Voice information acquisition means (voice information acquisition means)

The present invention relates to a video creation device, a video creation system, and a video creation program for creating a video in which an avatar is displayed.

Conventionally, the video creation system described in Patent Document 1 and the like, which supports the creation of a video with voice sung by an avatar by setting avatar information and voice information to which motion information is added and synthesizing each information, has been conventionally used. It is known.

Japanese Unexamined Patent Publication No. 2011-133882

In Document 1, by inputting avatar information, video information, melody information, and lyrics information input by the user from the terminal device, it is possible to smoothly create a video with a voice sung by the avatar. However, in order to move the avatar more humanly, it is necessary to input more motion information, and there is a problem that it takes time and effort to improve the quality of the moving image.

An object of the present invention is to provide a video creation device, a video creation system, and a video creation program that support the creation of a video in which an avatar moves or sings easily and with little effort.

In order to solve the above problems, the moving image creating device of the present invention is a moving image creating device that creates a moving image including an avatar, and obtains basic information about the avatar from its own storage unit or another via a network. Basic information acquisition means acquired from a computer, schedule information generation means for generating at least a part of schedule information in which a plurality of operation instructions for causing the avatar to perform a predetermined operation are arranged in accordance with the passage of time, and voice information. Based on the voice information acquisition means acquired from the own recording means or the own storage unit, or from another computer via the network, the basic information, the schedule information, and the voice information. The avatar is provided with a video data generation means for generating video data including a video with audio that performs a plurality of actions with the passage of time, and the mouth of the avatar synchronizes with the voice as the schedule information generation means. A schedule information automatic generation means for automatically generating at least a part of the schedule information is provided so that the information can be acquired from its own storage unit or from another computer via a network. The music information acquisition means to be acquired is provided, the voice information acquisition means is configured to acquire voice information from the music information acquired by the music information acquisition means, and the schedule information automatic generation means obtains the music information. The schedule information is configured to automatically set the motion information of the avatar's mouth based on the music information acquired by the means, and the schedule information generated by the schedule information generation means is acquired by the music information acquisition means. It is characterized by including information on the pose of the avatar singing the song or information on the dance motion .

A recording means for recording the user's voice is provided, and the user's voice recorded by the recording means is used as the voice information acquired by the voice acquisition means.

It is characterized by providing a synthetic voice generation means capable of changing a user's voice information recorded by the recording means into a synthetic voice.

The music information includes lyrics information and voice information of the music, and a lyrics information changing means for editing the lyrics information of the music selected by the voice information acquisition means is provided, and the schedule information automatic generation means is the lyrics information. When the lyrics information is edited by the changing means, it is characterized in that it is configured to change the movement of the avatar's mouth according to the changed lyrics.

The voice information acquisition means is characterized in that it can acquire voice information regarding the type of voice uttered by the avatar from its own storage unit or from another computer via a network. There is.

The schedule information generated by the schedule information generation means is characterized by including position information in which a plurality of information regarding left and right positions in a screen displaying the avatar are arranged in accordance with the passage of time.

The schedule information generated by the schedule information generation means is characterized by including scale information in which a plurality of information regarding the size of displaying the avatar is arranged in accordance with the passage of time.

The avatar is configured to have three-dimensional information, and the schedule information generated by the schedule information generation means is rotation position information in which a plurality of information regarding the direction in which the avatar is displayed are arranged in accordance with the passage of time. It is characterized by including.

Further, the moving image creation system according to the embodiment of the present invention includes a storage unit, basic information acquisition means for acquiring basic information about the avatar from the storage unit, and an operation instruction for causing the avatar to perform a predetermined operation. A schedule information generation means for generating at least a part of schedule information arranged in a plurality of cases according to the passage of time, a voice information acquisition means for acquiring voice information from its own recording means or the storage unit, the basic information, and the basic information. The schedule is provided with a moving image data generation means for generating moving image data including an image with sound in which the avatar performs a plurality of actions with the passage of time based on the schedule information and the audio information. As an information generation means, a schedule information automatic generation means for automatically generating at least a part of the schedule information is provided so that the mouth of the avatar operates in synchronization with the voice, and the music information used for moving image is stored in its own storage unit. A music information acquisition means acquired from another computer or acquired from another computer via a network is provided, and the audio information acquisition means is configured to acquire audio information from the music information acquired by the music information acquisition means. The schedule information automatic generation means is configured to automatically set the motion information of the mouth of the avatar based on the music information acquired by the music information acquisition means, and is generated by the schedule information generation means. The schedule information is characterized by including information regarding a pose of an avatar singing a song acquired by the music information acquisition means or information regarding a dance motion .

Further, the moving image creation program according to the embodiment of the present invention includes a basic information acquisition process of acquiring basic information about the avatar from its own storage unit or acquiring basic information from another computer via a network. A schedule information generation process that generates at least a part of schedule information in which a plurality of operation instructions for causing an avatar to perform a predetermined operation are arranged in accordance with the passage of time, and voice information is stored in a recording means of the user or the storage of the computer. Based on the voice information acquisition process acquired from the unit or from another computer via the network, the basic information, the schedule information, and the voice information, the avatar has a plurality of avatars over time. As the video data generation process for generating video data including a video with audio for operation and the schedule information generation process, at least a part of the schedule information is used so that the mouth of the avatar operates in synchronization with the audio. As the schedule information automatic generation process that is automatically generated , the music information acquisition process that acquires the music information used for moving image from its own storage unit, or the music information acquisition process that is acquired from another computer via the network, and the audio information acquisition process. , The process of acquiring voice information from the music information acquired by the music information acquisition process, and the operation information of the avatar's mouth based on the music information acquired by the music information acquisition process as the schedule information automatic generation process. It is characterized in that a process of automatically setting and a process of generating information about a pose of an avatar singing a song acquired by the music information acquisition means or information about a dance motion are executed as the schedule information generation means . ..

As the schedule information generation means, the schedule information automatic generation means for automatically generating at least a part of the schedule information is provided so that the mouth of the avatar operates in synchronization with the voice, so that the avatar can be easily and with little effort. You can create videos that make your avatar more human.

By configuring the schedule information generation means so that information on the display position, scale, and rotation position of the avatar can be set as the schedule information set according to the passage of time in the moving image, the avatar can be easily operated. Can be moved more dynamically in the video.

It is a block diagram which shows the example of the structure of the moving image creation processing apparatus of this invention. It is a block diagram which shows an example of the system configuration corresponding to the embodiment of this invention. It is a flowchart which shows the example of the moving image creation process corresponding to at least one of the Embodiments of this invention. It is a figure which showed the example of the setting operation screen for acquiring the basic information. It is a figure which showed the example of the setting operation screen for setting schedule information. It is a figure which showed the example of the avatar display edit screen. It is a flowchart which shows the example of the moving image creation process corresponding to at least one of the embodiments of another Example. It is a figure which showed the example of the operation screen for acquiring the music information by the music information acquisition means. It is a figure which showed the example of the setting operation screen for setting the schedule information in another embodiment.

Hereinafter, the moving image creation processing device according to the embodiment of the present invention will be described. FIG. 1 is a block diagram showing an example of the configuration of the moving image creation processing device of the present invention. As shown in FIG. 1, the moving image creation processing device (moving image creating device) 10 includes a background information acquisition unit 11, a basic information acquisition unit 12, an audio information acquisition unit 13, a schedule information manual generation unit 14, and a schedule. It includes an information automatic generation means 15, a moving image data generation means 16, and a storage unit 17.

FIG. 2 is a block diagram showing an example of a system configuration corresponding to an embodiment of the present invention. In the illustrated example, the terminal device 301 ... 30n used by the user is configured to be connectable to the server device 20 via the communication network 40, and the function of the moving image creation processing device 10 is on the server device 20 side. It is configured to aggregate a part of. The moving image creation processing device 10 may be realized in one terminal device 301 and used offline.

The server device 20 is managed by a system administrator and is configured by an information processing device such as a WWW server. Further, the server device 20 includes a control unit for executing various processes, a storage medium for storing various information, a communication unit, and a general configuration as a computer, and various processes are performed on a plurality of terminal devices. It has various functions to provide information about.

The system configuration is not limited to the above example, and one terminal device functioning as the moving image creation processing device 10 may be used by a plurality of users, or may be a configuration including a plurality of server devices.

The plurality of terminal devices 301 ... 30n are information processing terminals such as smartphones and tablet terminals having input means and output means composed of touch-operable touch panels, each of which is connected to a communication network 40 and is connected to the server device 20. It is equipped with hardware and software for executing various processes by communicating with. It should be noted that each of the plurality of terminal devices 301 ... 30n may be configured to be able to directly communicate with each other without going through the server device 20.

The background information acquisition means 11 has a function of acquiring image data or moving image data used as a background of a moving image to be created. The image data or moving image data used as the background may be acquired from the storage unit of the terminal device 301 ... 30n used by the user, or may be acquired from another computer via the communication network 40. Further, the image data or the moving image data may be photographed and acquired on the spot by a photographing means (not shown) such as a camera provided in the terminal device 301 ... 30n used by the user.

The basic information acquisition means 12 has a function of acquiring basic information regarding the appearance of an avatar to be displayed in a moving image to be created. In this embodiment, the basic information acquisition means 12 presents to the user a plurality of types of designs for each configuration of the avatar's hairstyle, costume, and color as basic information regarding the avatar, and the user selects an arbitrary design for each configuration. By doing so, it is possible to create an original avatar according to the user's preference. In this embodiment, a three-dimensional avatar is generated. An example of a specific setting operation of the basic information constituting the avatar will be described later.

The voice information acquisition means 13 has a function of acquiring voice information emitted by an avatar to be displayed in a moving image to be created. In this embodiment, the voice information acquisition means 13 is recorded by a recording means for recording a voice emitted by the user by using a microphone or the like provided in the terminal device 301 ... 30n used by the user, and the recording means. It has a synthetic voice generation means for editing the user's own voice information into a synthetic voice, and uses the user's voice information recorded by the recording means or the voice information obtained by editing the voice information into a synthetic voice for creating a moving image. It is configured so that it can be acquired as voice information.

The schedule information manual generation means 14 causes the avatar displayed in the video to be created to perform a predetermined operation, the telop information displayed in the video, the effect information, the voice information, and the video. It has a function to generate schedule information in which a plurality of music information related to music (BGM, etc.) is displayed side by side according to the passage of time of a moving image to be created, and the schedule information can be input and edited by a user operation. Has been done. An example of the schedule information setting operation will be described later.

The schedule information automatic generation means 15 automatically generates operation information (schedule information) that causes an avatar displayed in a moving image to perform a predetermined operation based on the schedule information acquired by the schedule information manual generation means 14. It has a function to generate. Specifically, the schedule information automatic generation means 15 automatically moves the mouth constituting the face of the avatar according to the voice information composed of the user's voice acquired by the recording means (lip-sync). ) Acquired by the lip-sync means (function), the blinking means (function) that automatically blinks the eyes constituting the face of the avatar at a predetermined timing based on the voice information, the effect information, etc., and the recording means. There is a synthetic voice setting means (function) for changing the user's own voice to a specific voice sound (synthetic voice).

The video data generation means 16 synthesizes (renders) all of the background information, basic information, audio information, schedule information, etc. set for creating a video including an avatar, so that the avatar sings or sends a message. It has a function of generating a moving image and storing it in the storage unit 17.

The storage unit 17 has a function of storing information necessary for processing of each means (each unit) in the moving image creation processing device 10, and also storing various information generated by the processing of each means.

Next, the flow of the moving image creation processing in the moving image creation processing apparatus 10 corresponding to the embodiment of the present invention will be described with reference to FIG. FIG. 3 is a flowchart showing an example of a moving image creation process corresponding to at least one of the embodiments of the present invention. In the illustrated example, the moving image creation process proceeds to step S101 when the execution of the moving image creation project is started.

In step S101, the background information acquisition means 11 acquires image data or moving image data used for the background of the moving image (executes the background information acquisition process), and then proceeds to step S102. In step S102, the basic information acquisition means 12 acquires basic information about the avatar displayed in the moving image (executes the basic information acquisition process), and then proceeds to step S103.

In step S103, the schedule information is generated (schedule information generation processing is executed) by the schedule information manual generation means 14 and the schedule information automatic generation means 15, and then the process proceeds to step S104. In step S104, a message video by the avatar is generated (video) by synthesizing (rendering) various information set by the background acquisition means 11, the basic information acquisition means 12, and the schedule information generation means 14 and 15. Execute the generation process), and then end the process.

The schedule information generation process includes a schedule information manual generation process that provides an environment for manually editing schedule information, and a schedule information automatic generation process that automatically generates other schedules based on the input of schedule information. Have. Incidentally, the order of the various processes constituting the above process flow is random as long as there is no contradiction in the process contents.

Next, an example of an acquisition method of information necessary for creating a moving image will be described with reference to FIGS. 4 to 6. FIG. 4 is a diagram showing an example of a setting operation screen for acquiring basic information. The basic information acquisition means 12 provides an operation environment (hereinafter, basic information input screen 20) for setting basic information about an avatar on a terminal device (touch panel) 301 ... 30n on the user side (see FIG. 4). The process of acquiring the basic information about the avatar via the basic information input screen 20 is configured to be executable.

In the illustrated example, as the basic information input screen 20 displayed on the touch panel on the user side, an avatar display unit 21 that displays an avatar generated based on the set basic information and a component that can change the appearance of the avatar. The element switching operation unit 22 for switching (basic information), the design selection operation unit 23 for displaying the design (basic information) that can be changed in the selected component side by side, and the basic information about the selected avatar. The save operation unit 24 for saving is displayed at the same time (see FIG. 4).

The element switching operation unit 22 includes a strip-shaped display unit in the left-right direction and tab-shaped switching operation units 22A, 22B, 22C arranged side by side along the display unit. In the illustrated example, the switching operation unit 22A for the hairstyle, the switching operation unit 22B for the costume, and the switching operation unit 22C for the color related to the avatar are arranged side by side in the left-right direction on the display unit, and the user can use the display unit. By selecting and operating any one of the switching operation units 22A, 22B, and 22C by a tap operation or the like, the design selection operation unit 23 can display information about the selected switching operation unit 22. It is configured (see FIG. 4).

Specifically, when the switching operation unit 22A for the hairstyle is selected, a plurality of basic information (designs) regarding the hairstyle are displayed side by side on the design selection operation unit 23, and the switching operation unit 22B for the costume is displayed. When is selected, a plurality of basic information (designs) related to costumes are displayed side by side on the design selection operation unit 23, and when the color switching operation unit 22C related to the avatar is selected, the design selection operation is performed. A plurality of basic information (color information) regarding the main color of the avatar is displayed side by side in the unit 23.

Further, the element switching operation unit 22 slides the band-shaped display unit in the left-right direction (flick operation) so that the switching operation unit 22 displayed in the display unit slides in the display unit. It may be configured. As a result, more types of switching operation units 22 can be arranged in the display unit. According to the configuration, as basic information (element switching operation unit 22) constituting the avatar whose design can be selected, in addition to the above example, the race, gender, etc. of the avatar, as well as the contour, eyes, and nose constituting the face. , Mouth, physique, etc. can be finely selected. The types of the switching operation unit 22 (basic information) that can be selected are not limited to these.

In the design change operation unit 23, among the components of the avatar selected by the element switching operation unit 22, the window-shaped design display unit 25 on which the changeable design is displayed is arranged side by side. One design display unit 25A selected by a user's touch operation or the like intuitively confirms which design display unit is selected by highlighting (thickening or blinking) the outer frame. It is configured so that it can be done (see FIG. 4).

By the way, in the design display unit 25, a design display unit (design) 25B that cannot be selected by displaying a light ink display or a display with a key 26 or the like is arranged, and the server device recognizes that the predetermined charge amount has been paid. On the condition that the design is not selectable, the design display unit may be opened so that it can be used (see FIG. 4).

The avatar display unit 21 can display an avatar that reflects all the designs (basic information) of each component of the avatar selected by the element switching operation unit 22 and the design change operation unit 23. .. The avatar displayed on the avatar display unit 21 is configured so that when the selection of the design display unit 25 is changed, the design of the changed part is immediately reflected, so that the user's favorite avatar can be used. It can be created efficiently.

With the above configuration, the appearance of the avatar to be displayed in the created moving image can be determined by acquiring the basic information about the avatar by the basic information acquisition means 12.

Next, an example of a method of generating schedule information by the schedule information manual generation means will be described with reference to FIG. FIG. 5 is a diagram showing an example of a setting operation screen for manually setting schedule information. The schedule information manual generation means 14 provides an operation environment (hereinafter, schedule information input screen 30) for setting the schedule information on the terminal device (touch panel) 301 ... 30n on the user side (see FIG. 5). The process of acquiring schedule information via the schedule information input screen 30 is configured to be executable.

In the illustrated example, the schedule information input screen 30 on the user's touch panel is a strip-shaped timeline display unit 32 in the left-right direction and a vertical direction in which the display position is fixed at the center of the timeline display unit 32 in the left-right direction. The display reference line 33, the edit screen display unit 31 that displays the image of the playback position of the moving image specified by the display reference line 33 and the timeline display unit 32, and the schedule edit arranged on the display reference line 33. The track 34 and the track 34 are displayed at the same time (see FIG. 5).

In the timeline display unit 32, the playback time of moving images is described in order on the strip-shaped display unit side by side in the left-right direction (every 10 seconds in the illustrated example), and the timeline display unit 32 or the schedule editing track 34. By sliding the display portion of the above in the left-right direction (flick operation) or the like, the display of the reproduction time in the timeline display unit 32 is configured to slide and move.

At this time, the playback time (00:10 seconds in the illustrated example) displayed in the center of the timeline display unit 32 in the left-right direction on the edit screen display unit 31 is shown on the display reference line 33. It is configured to display the state of the reproduction time on the timeline display unit 32.

In the illustrated example, the schedule editing track 34 is predetermined to the telop editing track 36 for editing the telop displayed in the moving image, the background editing track 37 for editing the image data and the moving image data as the background of the moving image, and the avatar. An avatar editing track 38 that edits operation instructions and the like for executing the operation of, an effect editing track 39 that edits the effect in the video, and a voice editing track (voice information) that acquires and edits voice information attached to the video. The acquisition means) 41 and the music editing track 42 for attaching a BGM to the moving image are configured to be displayed side by side in the vertical direction. Further, each edit track 34 is configured so that an edit block 40 indicating that the schedule information has been input is displayed in the reproduction range in which the schedule information is input (see FIG. 5).

Further, the schedule editing track 34 is configured to slide integrally to the left and right together with the notation of the reproduction time of the timeline display unit 32 by a slide operation or the like by the user, and the schedule editing track 34 is configured to slide in the center of each editing track in the left-right direction. , The display reference line 33 fixedly displayed at the center of the left and right is arranged. Along with this, each editing track is configured to be able to add or edit each schedule information at the reproduction position displayed by the timeline display unit 32 and the display reference line 33.

Specifically, in the example shown in FIG. 5, the telop editing track 36 does not have the editing block 40 at the time when the playback time starts 10 seconds (the schedule information is not set). At the right end of the telop editing track 36, an additional button 43 for adding schedule information regarding the telop is arranged (see FIG. 5). Similarly, in the background editing track 37, since the editing block 40 is present (the schedule information is set) when the playback time starts 10 seconds, the schedule information regarding the background is at the right end of the background editing track. An edit button 44 for editing (changing / deleting, etc.) is arranged (see FIG. 5).

Incidentally, the editing block 40 is provided with an editing range operation unit 40a on the end side thereof so that the length of the editing block in the left-right direction can be edited by a slide operation. The user can easily set the designated range of the editing block 40 on each editing track by long-pressing the editing range operation unit 40a and performing an operation (drag operation) of grasping and sliding the editing range operation unit 40a by a tap operation or a click operation. Can be edited to. As a result, it is not necessary to specify a numerical value or the like for specifying the range of each editing block, so that the editing work becomes easier and smoother.

In the voice editing track 41, the voice emitted by the user can be recorded by a recording means provided in the user's terminal device or the like and acquired as the voice information. In other words, the recorded user's voice can be used as a message, song, or the like spoken by the avatar in the video. Further, the acquired voice information is configured to be edited into a synthetic voice so that the voice of whom is not known by the synthetic voice generation means.

Further, when the voice information to be played in the moving image is acquired by the voice editing track 41, the operation information of the mouth of the avatar is automatically obtained by the schedule information automatic generation means 15 based on the acquired voice information. It is configured to perform the lip-sync process that is generated in. In the lip sync process, for example, the shape of the mouth of the avatar when pronouncing a vowel (a i u e o) is prepared in advance, and the timing at which the vowel is uttered is acquired from the acquired voice information. By combining these information, the shape of the mouth of the avatar may be configured to move in an appropriate shape at an appropriate timing. The processing method for moving the mouth of the avatar by the lip-sync processing is not limited to these.

Further, when the add button 43 or the edit button 44 of the avatar edit track 38 is pressed, the avatar display edit screen 50 for editing the schedule information related to the avatar display on the user's terminal device (touch panel). Is configured to be displayed (see FIG. 6). Hereinafter, a specific description will be given.

FIG. 6 is a diagram showing an example of an avatar display editing screen. As shown in the figure, on the avatar display editing screen 50, the editing screen display unit 51 that displays the state of the moving image at the playback position being edited and information on the components of the avatar's operation are displayed in a tab format in the left-right direction. The avatar operation element switching operation unit 52 displayed side by side is displayed.

In the illustrated example, the avatar operation element switching display unit 52 has a pose switching unit 52A for setting information (schedule information) regarding the pose of the avatar and a facial expression switching unit for setting information regarding the facial expression of the avatar (schedule information). A unit 52B and an avatar display position switching unit 52C for setting information (schedule information) regarding the display position of the avatar in the moving image are provided.

When the operation tab of the avatar display position switching unit 52C is selected, the information regarding the display position of the avatar in the editing block 40 being edited is edited by an operation tool (slider operation tool) composed of a plurality of slide bars. It is configured to be able to. Specifically, as shown in FIG. 6, as the slider operation tool, a position information setting slider 57 for operating the left and right positions of the avatar in the video, and a scale information setting for operating the size of the avatar in the video. The slider 58 and the rotation information setting slider 59 that operates the rotation position of the avatar expressed three-dimensionally are arranged side by side in the vertical direction (see FIG. 6).

According to each slider operation tool, the position, size, and orientation of the avatar can be freely set in the created moving image, so that the avatar can be moved more dynamically. Specifically, it is possible to make the avatar appear to be in a three-dimensional space by displaying the avatar in a small scale while walking the avatar in the depth direction of the screen in the video.

Next, another embodiment of the moving image creation processing apparatus will be described with reference to FIGS. 7 to 9 and different from the above-mentioned example. In another embodiment, as the voice information acquisition means 13, instead of the recording means described above, data (music information) of the music sung by the avatar in the moving image is stored from the storage unit 17 of the user's terminal device 301 ... 30n. A music information acquisition means 71 is provided for acquiring music information or acquiring music information from another computer via a network. Further, instead of the above-mentioned synthetic voice generation means, a voice information acquisition means 72 for acquiring the type of voice (synthetic voice) when the avatar sings a song is provided.

Incidentally, in the music information acquired by the music information acquisition means 71, the lyrics information of the music can be acquired at the same time as the voice information when the avatar sings the music.

Along with this, the schedule information automatic generation means 15 matches the music information acquired by the music information acquisition means 71, specifically, the lyrics information included in the music information, and the mouth of the avatar in the moving image. You can automatically set the movement information of the avatar's mouth so that you can move. In other words, by selecting the music sung by the avatar by the music information acquisition means 71, it is not necessary to separately input information regarding the movement of the mouth of the avatar. Therefore, the avatar in the video can be moved more humanly with less effort.

FIG. 7 is a flowchart showing an example of the moving image creation process corresponding to at least one of the embodiments of another embodiment. In the illustrated example, the moving image creation process proceeds to step S201 when the execution of the moving image creation project is started.

In step S201, the voice information acquisition means 13 (music information acquisition means 71 and voice information acquisition means 72) acquires music information and voice information to be sung by the avatar in the moving image, and then proceeds to step S202.

In step S202, image data or moving image data used for the background of the moving image created by the background information acquiring means is acquired (background information acquisition processing is executed), and then the process proceeds to step S203. In step S203, basic information about the avatar displayed in the moving image created by the basic information acquisition means is acquired (basic information acquisition process is executed), and then the process proceeds to step S204.

In step S204, the schedule information generation means generates the schedule information (executes the schedule information generation process), and then proceeds to step S205. In step S205, a message video by the avatar is generated (execution of the video generation process) based on various information set by the background acquisition means, the basic information acquisition means, and the schedule information generation means, and then the process is terminated. do.

That is, in the voice information acquisition means 13 of another embodiment, the music information is acquired by the music information acquisition means 71 before the schedule information is edited by the schedule information manual generation means 14, and the avatar is displayed in a moving image. It is configured to simultaneously acquire the voice information to be sung and the lyrics information to be uttered by the avatar. According to the configuration, by first acquiring the basic information of the avatar and the music information sung by the avatar, the time and effort for inputting the schedule information can be significantly reduced.

FIG. 8 is a diagram showing an example of an operation screen for acquiring music information by the music information acquisition means. The voice information acquisition means 13 provides an operation environment (hereinafter, music information setting screen 70) for inputting the music information and the voice information to the terminal device (touch panel) on the user side (see FIG. 8). The process of acquiring the music information and the voice information via the music information setting screen 70 is configured to be executable.

In the illustrated example, the music information setting screen 70 has a radio button for selecting one from a plurality of songs displayed side by side, and a radio button for selecting one from a plurality of voice types displayed side by side. Are arranged side by side, but they are not limited to these as long as one is selected from multiple options.

Further, the voice information acquisition means 13 increases the types of music information that can be selected and the types of voice information (synthetic voice) on condition that the server device 20 recognizes that the predetermined billing amount has been paid. It may be configured so that

FIG. 9 is a diagram showing an example of a setting operation screen for setting schedule information in another embodiment. In the illustrated example, as the schedule information input screen 30 on the touch panel on the user side, the edit screen display unit 31, the timeline display unit 32, the display reference line 33, and the schedule are the same as in the above example. The edit track 34 and the edit track 34 are displayed at the same time (see FIG. 9).

The schedule editing track 34 includes the telop editing track 36, a lyrics editing track 81 for editing lyrics information acquired from the music information, a dance editing track 82 for designating a dance motion to the avatar, and the avatar. The editing track 38, the effect editing track 39, and the background editing track 37 are configured to be displayed side by side in the vertical direction.

The lyrics editing track 81 is configured such that the lyrics information included in the music information acquired by the music information acquisition means is input in advance, but the user can edit a part of the lyrics to make a substitute song. Has been done.

Along with this, when the lyrics information is edited by the lyrics editing track 81, the schedule information automatic generation means 15 uses the voice information included in the acquired music information as the edited voice. It is configured to execute a synthetic voice generation process that automatically generates (edits) synthetic voice so that it becomes information.

Further, when the synthetic voice constituting the voice information is edited by editing the lyrics information, the schedule information automatic generation means 15 moves in the moving image according to the lyrics of the edited parody. It is configured to be able to perform a lip-sync process that automatically corrects the movement of the avatar's mouth.

That is, when the lyrics information is edited by the lyrics editing track 81, the schedule information automatic generation means 15 includes a part of the schedule information, specifically, the voice information sung by the avatar in the moving image and the avatar. It is configured to automatically edit the motion information related to the movement of the mouth according to the content of the edited lyrics. Therefore, it is possible to edit the music selected by the user into a favorite parody and generate a moving image to be sung by the avatar with smoother and higher quality.

The dance editing track 82 is configured such that the pose operation of the avatar can be specified in the avatar editing track 38, but the dance motion according to the music can be specified.

12 Basic information acquisition means 13 Voice information acquisition means 14 Schedule information manual generation means (schedule information generation means)
15 Schedule information automatic generation means (schedule information generation means)
16 Video data generation means 17 Storage unit 40 Communication network 71 Music information acquisition means 72 Voice information acquisition means (voice information acquisition means)

The present invention relates to a video creation device, a video creation system, and a video creation program for creating a video in which an avatar is displayed.

Conventionally, the video creation system described in Patent Document 1 and the like, which supports the creation of a video with voice sung by an avatar by setting avatar information and voice information to which motion information is added and synthesizing each information, has been conventionally used. It is known.

Japanese Unexamined Patent Publication No. 2011-133882

In Document 1, by inputting avatar information, video information, melody information, and lyrics information input by the user from the terminal device, it is possible to smoothly create a video with a voice sung by the avatar. However, in order to move the avatar more humanly, it is necessary to input more motion information, and there is a problem that it takes time and effort to improve the quality of the moving image.

An object of the present invention is to provide a video creation device, a video creation system, and a video creation program that support the creation of a video in which an avatar moves or sings easily and with little effort.

In order to solve the above problems, the moving image creating device of the present invention is a moving image creating device that creates a moving image including an avatar, and obtains basic information about the avatar from its own storage unit or another via a network. Basic information acquisition means acquired from a computer, schedule information generation means for generating at least a part of schedule information in which a plurality of operation instructions for causing the avatar to perform a predetermined operation are arranged in accordance with the passage of time, and voice information. The avatar is the avatar based on the voice information acquisition means, the basic information, the schedule information, and the voice information, which are acquired from the storage unit of the self or from another computer via the network. It is equipped with a video data generation means for generating video data including a video with sound that performs a plurality of actions with the passage of time, and as the schedule information generation means, the mouth of the avatar operates in synchronization with the voice. Is provided with a schedule information automatic generation means that automatically generates at least a part of the schedule information, and the music information used for video creation is acquired from its own storage unit, or music information acquired from another computer via a network. The acquisition means is provided, the voice information acquisition means is configured to acquire voice information from the music information acquired by the music information acquisition means, and the schedule information automatic generation means is acquired by the music information acquisition means. The schedule information is configured to automatically set the motion information of the avatar's mouth based on the music information, and the schedule information generated by the schedule information generation means is an avatar singing a music acquired by the music information acquisition means. The music information includes information about the pose or dance motion of the music, and the music information includes the lyrics information and the voice information of the music, and the lyrics information changing means for editing the lyrics information of the music selected by the voice information acquisition means. The schedule information automatic generation means is characterized in that, when the lyrics information is edited by the lyrics information changing means, the movement of the avatar's mouth is changed according to the changed lyrics. There is.

The voice information acquisition means is characterized in that it can acquire voice information regarding the type of voice uttered by the avatar from its own storage unit or from another computer via a network. There is.

The schedule information generated by the schedule information generation means is characterized by including position information in which a plurality of information regarding left and right positions in a screen displaying the avatar are arranged in accordance with the passage of time.

The schedule information generated by the schedule information generation means is characterized by including scale information in which a plurality of information regarding the size of displaying the avatar is arranged in accordance with the passage of time.

The avatar is configured to have three-dimensional information, and the schedule information generated by the schedule information generation means is rotation position information in which a plurality of information regarding the direction in which the avatar is displayed are arranged in accordance with the passage of time. It is characterized by including.

Further, the moving image creation system according to the embodiment of the present invention includes a storage unit, basic information acquisition means for acquiring basic information about the avatar from the storage unit, and an operation instruction for causing the avatar to perform a predetermined operation. The schedule information generation means for generating at least a part of the schedule information arranged in a plurality of cases according to the passage of time, the voice information acquisition means for acquiring the voice information from the storage unit, the basic information, and the schedule information. As the schedule information generation means, the avatar is provided with a video data generation means for generating video data including a video with voice that performs a plurality of actions with the passage of time based on the voice information. , Whether to provide a schedule information automatic generation means that automatically generates at least a part of the schedule information so that the mouth of the avatar operates in synchronization with the voice, and acquire the music information used for video creation from its own storage unit. Alternatively, a music information acquisition means acquired from another computer via the network is provided, and the voice information acquisition means is configured to acquire voice information from the music information acquired by the music information acquisition means, and the schedule information. The automatic generation means is configured to automatically set the motion information of the avatar's mouth based on the music information acquired by the music information acquisition means, and the schedule information generated by the schedule information generation means is The music information includes information on a pose of an avatar singing a music acquired by the music information acquisition means or information on a dance motion , and the music information includes lyrics information and voice information of the music, and is selected by the voice information acquisition means. A means for changing lyrics information for editing the lyrics information of the song is provided, and the means for automatically generating schedule information is such that when the lyrics information is edited by the means for changing the lyrics information, the avatar's mouth is adjusted to the changed lyrics. It is characterized by being configured to change movement .

Further, the moving image creation program according to the embodiment of the present invention includes a basic information acquisition process of acquiring basic information about the avatar from its own storage unit or acquiring basic information from another computer via a network. Schedule information generation processing for generating at least a part of schedule information in which a plurality of operation instructions for causing an avatar to perform a predetermined operation are arranged in accordance with the passage of time, and voice information is acquired from the storage unit of the computer. Alternatively, based on the voice information acquisition process acquired from another computer via the network, the basic information, the schedule information, and the voice information, the voice that the avatar performs a plurality of actions with the passage of time. As the video data generation process for generating video data including the attached video and the schedule information generation process, at least a part of the schedule information is automatically generated so that the mouth of the avatar operates in synchronization with the voice. Schedule information automatic generation processing, music information acquisition processing for acquiring music information used for moving image from its own storage unit, or acquisition from another computer via a network, and music information as the audio information acquisition processing. As the process of acquiring voice information from the music information acquired by the acquisition process and the schedule information automatic generation process, the motion information of the avatar's mouth is automatically set based on the music information acquired by the music information acquisition process. And, as the schedule information generation means, the process of generating the information about the pose of the avatar singing the song acquired by the music information acquisition means or the information about the dance motion, and the process of acquiring the lyrics information of the music from the music information. When the lyrics information is edited by the lyrics information changing means as the processing, the lyrics information changing processing for editing the lyrics information of the music selected by the voice information acquisition means, and the schedule information automatic generation processing, the change is made. It is characterized by executing a process of changing the movement of the avatar's mouth according to the lyrics .

As the schedule information generation means, the schedule information automatic generation means for automatically generating at least a part of the schedule information is provided so that the mouth of the avatar operates in synchronization with the voice, so that the avatar can be easily and with little effort. You can create videos that make your avatar more human.

By configuring the schedule information generation means so that information on the display position, scale, and rotation position of the avatar can be set as the schedule information set according to the passage of time in the moving image, the avatar can be easily operated. Can be moved more dynamically in the video.

It is a block diagram which shows the example of the structure of the moving image creation processing apparatus of this invention. It is a block diagram which shows an example of the system configuration corresponding to the embodiment of this invention. It is a flowchart which shows the example of the moving image creation process corresponding to at least one of the Embodiments of this invention. It is a figure which showed the example of the setting operation screen for acquiring the basic information. It is a figure which showed the example of the setting operation screen for setting schedule information. It is a figure which showed the example of the avatar display edit screen. It is a flowchart which shows the example of the moving image creation process corresponding to at least one of the embodiments of another Example. It is a figure which showed the example of the operation screen for acquiring the music information by the music information acquisition means. It is a figure which showed the example of the setting operation screen for setting the schedule information in another embodiment.

Hereinafter, the moving image creation processing device according to the embodiment of the present invention will be described. FIG. 1 is a block diagram showing an example of the configuration of the moving image creation processing device of the present invention. As shown in FIG. 1, the moving image creation processing device (moving image creating device) 10 includes a background information acquisition means 11, a basic information acquisition means 12, an audio information acquisition means 13, a schedule information manual generation means 14, and a schedule. The information automatic generation means 15, the moving image data generation means 16, and the storage unit 17 are provided.

FIG. 2 is a block diagram showing an example of a system configuration corresponding to an embodiment of the present invention. In the illustrated example, the terminal device 301 ... 30n used by the user is configured to be connectable to the server device 20 via the communication network 40, and the function of the moving image creation processing device 10 is on the server device 20 side. It is configured to aggregate a part of. The moving image creation processing device 10 may be realized in one terminal device 301 and used offline.

The server device 20 is managed by a system administrator and is configured by an information processing device such as a WWW server. Further, the server device 20 includes a control unit for executing various processes, a storage medium for storing various information, a communication unit, and a general configuration as a computer, and various processes are performed on a plurality of terminal devices. It has various functions to provide information about.

The system configuration is not limited to the above example, and one terminal device functioning as the moving image creation processing device 10 may be used by a plurality of users, or may be a configuration including a plurality of server devices.

The plurality of terminal devices 301 ... 30n are information processing terminals such as smartphones and tablet terminals having input means and output means composed of touch-operable touch panels, each of which is connected to a communication network 40 and is connected to the server device 20. It is equipped with hardware and software for executing various processes by communicating with. It should be noted that each of the plurality of terminal devices 301 ... 30n may be configured to be able to directly communicate with each other without going through the server device 20.

The background information acquisition means 11 has a function of acquiring image data or moving image data used as a background of a moving image to be created. The image data or moving image data used as the background may be acquired from the storage unit of the terminal device 301 ... 30n used by the user, or may be acquired from another computer via the communication network 40. Further, the image data or the moving image data may be photographed and acquired on the spot by a photographing means (not shown) such as a camera provided in the terminal device 301 ... 30n used by the user.

The basic information acquisition means 12 has a function of acquiring basic information regarding the appearance of an avatar to be displayed in a moving image to be created. In this embodiment, the basic information acquisition means 12 presents to the user a plurality of types of designs for each configuration of the avatar's hairstyle, costume, and color as basic information regarding the avatar, and the user selects an arbitrary design for each configuration. By doing so, it is possible to create an original avatar according to the user's preference. In this embodiment, a three-dimensional avatar is generated. An example of a specific setting operation of the basic information constituting the avatar will be described later.

The voice information acquisition means 13 has a function of acquiring voice information emitted by an avatar to be displayed in a moving image to be created. In this embodiment, the voice information acquisition means 13 is recorded by a recording means for recording a voice emitted by the user by using a microphone or the like provided in the terminal device 301 ... 30n used by the user, and the recording means. It has a synthetic voice generation means for editing the user's own voice information into a synthetic voice, and uses the user's voice information recorded by the recording means or the voice information obtained by editing the voice information into a synthetic voice for creating a moving image. It is configured so that it can be acquired as voice information.

The schedule information manual generation means 14 causes the avatar displayed in the video to be created to perform a predetermined operation, the telop information displayed in the video, the effect information, the voice information, and the video. It has a function to generate schedule information in which a plurality of music information related to music (BGM, etc.) is displayed side by side according to the passage of time of a moving image to be created, and the schedule information can be input and edited by a user operation. Has been done. An example of the schedule information setting operation will be described later.

The schedule information automatic generation means 15 automatically generates operation information (schedule information) that causes an avatar displayed in a moving image to perform a predetermined operation based on the schedule information acquired by the schedule information manual generation means 14. It has a function to generate. Specifically, the schedule information automatic generation means 15 automatically moves the mouth constituting the face of the avatar according to the voice information composed of the user's voice acquired by the recording means (lip-sync). ) Acquired by the lip-sync means (function), the blinking means (function) that automatically blinks the eyes constituting the face of the avatar at a predetermined timing based on the voice information, the effect information, etc., and the recording means. There is a synthetic voice setting means (function) for changing the user's own voice to a specific voice sound (synthetic voice).

The video data generation means 16 synthesizes (renders) all of the background information, basic information, audio information, schedule information, etc. set for creating a video including an avatar, so that the avatar sings or sends a message. It has a function of generating a moving image and storing it in the storage unit 17.

The storage unit 17 has a function of storing information necessary for processing of each means (each unit) in the moving image creation processing device 10, and also storing various information generated by the processing of each means.

Next, the flow of the moving image creation processing in the moving image creation processing apparatus 10 corresponding to the embodiment of the present invention will be described with reference to FIG. FIG. 3 is a flowchart showing an example of a moving image creation process corresponding to at least one of the embodiments of the present invention. In the illustrated example, the moving image creation process proceeds to step S101 when the execution of the moving image creation project is started.

In step S101, the background information acquisition means 11 acquires image data or moving image data used for the background of the moving image (executes the background information acquisition process), and then proceeds to step S102. In step S102, the basic information acquisition means 12 acquires basic information about the avatar displayed in the moving image (executes the basic information acquisition process), and then proceeds to step S103.

In step S103, the schedule information is generated (schedule information generation processing is executed) by the schedule information manual generation means 14 and the schedule information automatic generation means 15, and then the process proceeds to step S104. In step S104, a message video by the avatar is generated (video) by synthesizing (rendering) various information set by the background acquisition means 11, the basic information acquisition means 12, and the schedule information generation means 14 and 15. Execute the generation process), and then end the process.

The schedule information generation process includes a schedule information manual generation process that provides an environment for manually editing schedule information, and a schedule information automatic generation process that automatically generates other schedules based on the input of schedule information. Have. Incidentally, the order of the various processes constituting the above process flow is random as long as there is no contradiction in the process contents.

Next, an example of an acquisition method of information necessary for creating a moving image will be described with reference to FIGS. 4 to 6. FIG. 4 is a diagram showing an example of a setting operation screen for acquiring basic information. The basic information acquisition means 12 provides an operation environment (hereinafter, basic information input screen 20) for setting basic information about an avatar on a terminal device (touch panel) 301 ... 30n on the user side (see FIG. 4). The process of acquiring the basic information about the avatar via the basic information input screen 20 is configured to be executable.

In the illustrated example, as the basic information input screen 20 displayed on the touch panel on the user side, an avatar display unit 21 that displays an avatar generated based on the set basic information and a component that can change the appearance of the avatar. The element switching operation unit 22 for switching (basic information), the design selection operation unit 23 for displaying the design (basic information) that can be changed in the selected component side by side, and the basic information about the selected avatar. The save operation unit 24 for saving is displayed at the same time (see FIG. 4).

The element switching operation unit 22 includes a strip-shaped display unit in the left-right direction and tab-shaped switching operation units 22A, 22B, 22C arranged side by side along the display unit. In the illustrated example, the hairstyle switching operation unit 22A, the costume switching operation unit 22B, and the color switching operation unit 22C related to the avatar are arranged side by side in the left-right direction on the display unit, and the user can use the display unit. By selecting and operating any one of the switching operation units 22A, 22B, and 22C by tapping or the like, the design selection operation unit 23 can display information about the selected switching operation unit 22. It is configured (see FIG. 4).

Specifically, when the switching operation unit 22A for the hairstyle is selected, a plurality of basic information (designs) regarding the hairstyle are displayed side by side on the design selection operation unit 23, and the switching operation unit 22B for the costume is displayed. When is selected, a plurality of basic information (designs) related to costumes are displayed side by side on the design selection operation unit 23, and when the color switching operation unit 22C related to the avatar is selected, the design selection operation is performed. A plurality of basic information (color information) regarding the main color of the avatar is displayed side by side in the unit 23.

Further, the element switching operation unit 22 slides the band-shaped display unit in the left-right direction (flick operation) so that the switching operation unit 22 displayed in the display unit slides in the display unit. It may be configured. As a result, more types of switching operation units 22 can be arranged in the display unit. According to the configuration, as basic information (element switching operation unit 22) constituting the avatar whose design can be selected, in addition to the above example, the race, gender, etc. of the avatar, as well as the contour, eyes, and nose constituting the face. , Mouth, physique, etc. can be finely selected. The types of the switching operation unit 22 (basic information) that can be selected are not limited to these.

In the design change operation unit 23, among the components of the avatar selected by the element switching operation unit 22, the window-shaped design display unit 25 on which the changeable design is displayed is arranged side by side. One design display unit 25A selected by a user's touch operation or the like intuitively confirms which design display unit is selected by highlighting (thickening or blinking) the outer frame. It is configured so that it can be done (see FIG. 4).

By the way, in the design display unit 25, a design display unit (design) 25B that cannot be selected by displaying a light ink display or a display with a key 26 or the like is arranged, and the server device recognizes that the predetermined charge amount has been paid. On the condition that the design is not selectable, the design display unit may be opened so that it can be used (see FIG. 4).

The avatar display unit 21 can display an avatar that reflects all the designs (basic information) of each component of the avatar selected by the element switching operation unit 22 and the design change operation unit 23. .. The avatar displayed on the avatar display unit 21 is configured so that when the selection of the design display unit 25 is changed, the design of the changed part is immediately reflected, so that the user's favorite avatar can be used. It can be created efficiently.

With the above configuration, the appearance of the avatar to be displayed in the created moving image can be determined by acquiring the basic information about the avatar by the basic information acquisition means 12.

Next, an example of a method of generating schedule information by the schedule information manual generation means will be described with reference to FIG. FIG. 5 is a diagram showing an example of a setting operation screen for manually setting schedule information. The schedule information manual generation means 14 provides an operation environment (hereinafter, schedule information input screen 30) for setting the schedule information on the terminal device (touch panel) 301 ... 30n on the user side (see FIG. 5). The process of acquiring schedule information via the schedule information input screen 30 is configured to be executable.

In the illustrated example, the schedule information input screen 30 on the user's touch panel is a strip-shaped timeline display unit 32 in the left-right direction and a vertical direction in which the display position is fixed at the center of the timeline display unit 32 in the left-right direction. The display reference line 33, the edit screen display unit 31 that displays the image of the playback position of the moving image specified by the display reference line 33 and the timeline display unit 32, and the schedule edit arranged on the display reference line 33. The track 34 and the track 34 are displayed at the same time (see FIG. 5).

In the timeline display unit 32, the playback time of moving images is described in order on the strip-shaped display unit side by side in the left-right direction (every 10 seconds in the illustrated example), and the timeline display unit 32 or the schedule editing track 34. By sliding the display portion of the above in the left-right direction (flick operation) or the like, the display of the reproduction time in the timeline display unit 32 is configured to slide and move.

At this time, the playback time (00:10 seconds in the illustrated example) displayed in the center of the timeline display unit 32 in the left-right direction on the edit screen display unit 31 is shown on the display reference line 33. It is configured to display the state of the reproduction time on the timeline display unit 32.

In the illustrated example, the schedule editing track 34 is predetermined to the telop editing track 36 for editing the telop displayed in the moving image, the background editing track 37 for editing the image data and the moving image data as the background of the moving image, and the avatar. An avatar editing track 38 for editing an operation instruction for executing the operation of, an effect editing track 39 for editing an effect in a video, and a voice editing track for acquiring and editing audio information attached to a video (audio information). The acquisition means) 41 and the music editing track 42 for attaching a BGM to the moving image are configured to be displayed side by side in the vertical direction. Further, each edit track 34 is configured so that an edit block 40 indicating that the schedule information has been input is displayed in the reproduction range in which the schedule information is input (see FIG. 5).

Further, the schedule editing track 34 is configured to slide integrally to the left and right together with the notation of the reproduction time of the timeline display unit 32 by a slide operation or the like by the user, and the schedule editing track 34 is configured to slide in the center of each editing track in the left-right direction. , The display reference line 33 fixedly displayed at the center of the left and right is arranged. Along with this, each editing track is configured to be able to add or edit each schedule information at the reproduction position displayed by the timeline display unit 32 and the display reference line 33.

Specifically, in the example shown in FIG. 5, the telop editing track 36 does not have the editing block 40 at the time when the playback time starts 10 seconds (the schedule information is not set). At the right end of the telop editing track 36, an additional button 43 for adding schedule information regarding the telop is arranged (see FIG. 5). Similarly, in the background editing track 37, since the editing block 40 is present (the schedule information is set) when the playback time starts 10 seconds, the schedule information regarding the background is at the right end of the background editing track. An edit button 44 for editing (changing / deleting, etc.) is arranged (see FIG. 5).

Incidentally, the editing block 40 is provided with an editing range operation unit 40a on the end side thereof so that the length of the editing block in the left-right direction can be edited by a slide operation. The user can easily set the designated range of the editing block 40 on each editing track by long-pressing the editing range operation unit 40a and performing an operation (drag operation) of grasping and sliding the editing range operation unit 40a by a tap operation or a click operation. Can be edited to. As a result, it is not necessary to specify a numerical value or the like for specifying the range of each editing block, so that the editing work becomes easier and smoother.

In the voice editing track 41, the voice emitted by the user can be recorded by a recording means provided in the user's terminal device or the like and acquired as the voice information. In other words, the recorded user's voice can be used as a message, song, or the like spoken by the avatar in the video. Further, the acquired voice information is configured to be edited into a synthetic voice so that the voice of whom is not known by the synthetic voice generation means.

Further, when the voice information to be played in the moving image is acquired by the voice editing track 41, the operation information of the mouth of the avatar is automatically obtained by the schedule information automatic generation means 15 based on the acquired voice information. It is configured to perform the lip-sync process that is generated in. In the lip sync process, for example, the shape of the mouth of the avatar when pronouncing a vowel (a i u e o) is prepared in advance, and the timing at which the vowel is uttered is acquired from the acquired voice information. By combining these information, the shape of the mouth of the avatar may be configured to move in an appropriate shape at an appropriate timing. The processing method for moving the mouth of the avatar by the lip-sync processing is not limited to these.

Further, when the add button 43 or the edit button 44 of the avatar edit track 38 is pressed, the avatar display edit screen 50 for editing the schedule information related to the avatar display on the user's terminal device (touch panel). Is configured to be displayed (see FIG. 6). Hereinafter, a specific description will be given.

FIG. 6 is a diagram showing an example of an avatar display editing screen. As shown in the figure, on the avatar display editing screen 50, the editing screen display unit 51 that displays the state of the moving image at the playback position being edited and information on the components of the avatar's operation are displayed in a tab format in the left-right direction. The avatar operation element switching operation unit 52 displayed side by side is displayed.

In the illustrated example, the avatar operation element switching display unit 52 has a pose switching unit 52A for setting information (schedule information) regarding the pose of the avatar and a facial expression switching unit for setting information regarding the facial expression of the avatar (schedule information). A unit 52B and an avatar display position switching unit 52C for setting information (schedule information) regarding the display position of the avatar in the moving image are provided.

When the operation tab of the avatar display position switching unit 52C is selected, the information regarding the display position of the avatar in the editing block 40 being edited is edited by an operation tool (slider operation tool) composed of a plurality of slide bars. It is configured to be able to. Specifically, as shown in FIG. 6, as the slider operation tool, a position information setting slider 57 for operating the left and right positions of the avatar in the video, and a scale information setting for operating the size of the avatar in the video. The slider 58 and the rotation information setting slider 59 that operates the rotation position of the avatar expressed three-dimensionally are arranged side by side in the vertical direction (see FIG. 6).

According to each slider operation tool, the position, size, and orientation of the avatar can be freely set in the created moving image, so that the avatar can be moved more dynamically. Specifically, it is possible to make the avatar appear to be in a three-dimensional space by displaying the avatar in a small scale while walking the avatar in the depth direction of the screen in the video.

Next, another embodiment of the moving image creation processing apparatus will be described with reference to FIGS. 7 to 9 and different from the above-mentioned example. In another embodiment, as the voice information acquisition means 13, instead of the recording means described above, data (music information) of the music sung by the avatar in the moving image is stored from the storage unit 17 of the user's terminal device 301 ... 30n. A music information acquisition means 71 is provided for acquiring music information or acquiring music information from another computer via a network. Further, instead of the above-mentioned synthetic voice generation means, a voice information acquisition means 72 for acquiring the type of voice (synthetic voice) when the avatar sings a song is provided.

Incidentally, in the music information acquired by the music information acquisition means 71, the lyrics information of the music can be acquired at the same time as the voice information when the avatar sings the music.

Along with this, the schedule information automatic generation means 15 matches the music information acquired by the music information acquisition means 71, specifically, the lyrics information included in the music information, and the mouth of the avatar in the moving image. You can automatically set the movement information of the avatar's mouth so that you can move. In other words, by selecting the music sung by the avatar by the music information acquisition means 71, it is not necessary to separately input information regarding the movement of the mouth of the avatar. Therefore, the avatar in the video can be moved more humanly with less effort.

FIG. 7 is a flowchart showing an example of the moving image creation process corresponding to at least one of the embodiments of another embodiment. In the illustrated example, the moving image creation process proceeds to step S201 when the execution of the moving image creation project is started.

In step S201, the voice information acquisition means 13 (music information acquisition means 71 and voice information acquisition means 72) acquires music information and voice information to be sung by the avatar in the moving image, and then proceeds to step S202.

In step S202, image data or moving image data used for the background of the moving image created by the background information acquiring means is acquired (background information acquisition processing is executed), and then the process proceeds to step S203. In step S203, basic information about the avatar displayed in the moving image created by the basic information acquisition means is acquired (basic information acquisition process is executed), and then the process proceeds to step S204.

In step S204, the schedule information generation means generates the schedule information (executes the schedule information generation process), and then proceeds to step S205. In step S205, a message video by the avatar is generated (execution of the video generation process) based on various information set by the background acquisition means, the basic information acquisition means, and the schedule information generation means, and then the process is terminated. do.

That is, in the voice information acquisition means 13 of another embodiment, the music information is acquired by the music information acquisition means 71 before the schedule information is edited by the schedule information manual generation means 14, and the avatar is displayed in a moving image. It is configured to simultaneously acquire the voice information to be sung and the lyrics information to be uttered by the avatar. According to the configuration, by first acquiring the basic information of the avatar and the music information sung by the avatar, the time and effort for inputting the schedule information can be significantly reduced.

FIG. 8 is a diagram showing an example of an operation screen for acquiring music information by the music information acquisition means. The voice information acquisition means 13 provides an operation environment (hereinafter, music information setting screen 70) for inputting the music information and the voice information to the terminal device (touch panel) on the user side (see FIG. 8). The process of acquiring the music information and the voice information via the music information setting screen 70 is configured to be executable.

In the illustrated example, the music information setting screen 70 has a radio button for selecting one from a plurality of songs displayed side by side, and a radio button for selecting one from a plurality of voice types displayed side by side. Are arranged side by side, but they are not limited to these as long as one is selected from multiple options.

Further, the voice information acquisition means 13 increases the types of music information that can be selected and the types of voice information (synthetic voice) on condition that the server device 20 recognizes that the predetermined billing amount has been paid. It may be configured so that

FIG. 9 is a diagram showing an example of a setting operation screen for setting schedule information in another embodiment. In the illustrated example, as the schedule information input screen 30 on the touch panel on the user side, the edit screen display unit 31, the timeline display unit 32, the display reference line 33, and the schedule are the same as in the above example. The edit track 34 and the edit track 34 are displayed at the same time (see FIG. 9).

The schedule editing track 34 includes the telop editing track 36, a lyrics editing track 81 for editing lyrics information acquired from the music information, a dance editing track 82 for designating a dance motion to the avatar, and the avatar. The editing track 38, the effect editing track 39, and the background editing track 37 are configured to be displayed side by side in the vertical direction.

The lyrics editing track 81 is configured such that the lyrics information included in the music information acquired by the music information acquisition means is input in advance, but the user can edit a part of the lyrics to make a substitute song. Has been done.

Along with this, when the lyrics information is edited by the lyrics editing track 81, the schedule information automatic generation means 15 uses the voice information included in the acquired music information as the edited voice. It is configured to execute a synthetic voice generation process that automatically generates (edits) synthetic voice so that it becomes information.

Further, when the synthetic voice constituting the voice information is edited by editing the lyrics information, the schedule information automatic generation means 15 moves in the moving image according to the lyrics of the edited parody. It is configured to be able to perform a lip-sync process that automatically corrects the movement of the avatar's mouth.

That is, when the lyrics information is edited by the lyrics editing track 81, the schedule information automatic generation means 15 includes a part of the schedule information, specifically, the voice information sung by the avatar in the moving image and the avatar. It is configured to automatically edit the motion information related to the movement of the mouth according to the content of the edited lyrics. Therefore, it is possible to edit the music selected by the user into a favorite parody and generate a moving image to be sung by the avatar with smoother and higher quality.

The dance editing track 82 is configured such that the pose operation of the avatar can be specified in the avatar editing track 38, but the dance motion according to the music can be specified.

12 Basic information acquisition means 13 Voice information acquisition means 14 Schedule information manual generation means (schedule information generation means)
15 Schedule information automatic generation means (schedule information generation means)
16 Video data generation means 17 Storage unit 40 Communication network 71 Music information acquisition means 72 Voice information acquisition means (voice information acquisition means)

Further, the moving image creation program according to the embodiment of the present invention includes a basic information acquisition process of acquiring basic information about the avatar from its own storage unit or acquiring basic information from another computer via a network. Whether to acquire the schedule information generation process for generating at least a part of the schedule information in which a plurality of operation instructions for causing the avatar to perform a predetermined operation are arranged in accordance with the passage of time, and the voice information from the own storage unit. Or, with voice, the avatar performs a plurality of actions with the passage of time based on the voice information acquisition process acquired from another computer via the network, the basic information, the schedule information, and the voice information. As the video data generation process for generating the video data including the video of the above and the schedule information generation process, at least a part of the schedule information is automatically generated so that the mouth of the avatar operates in synchronization with the voice. The music information acquisition as the schedule information automatic generation processing, the music information acquisition processing of acquiring the music information used for moving image from its own storage unit, or the music information acquisition processing acquired from another computer via the network, and the voice information acquisition processing. As the process of acquiring voice information from the music information acquired by the process and the schedule information automatic generation process, the motion information of the avatar's mouth is automatically set based on the music information acquired by the music information acquisition process. Processing, as the schedule information generation processing , a process of generating information on a pose or a dance motion of an avatar singing a song acquired by the song information acquisition process , and a process of acquiring song lyrics information from the song information. And, when the lyrics information is edited by the lyrics information change processing as the schedule information automatic generation processing and the lyrics information change processing for editing the lyrics information of the music selected by the voice information acquisition processing , the change is made. It is characterized by executing a process of changing the movement of the avatar's mouth according to the lyrics.

Claims (11)

It is a video creation device that creates videos including avatars.
Basic information acquisition means for acquiring basic information about the avatar from its own storage unit or from another computer via a network.
A schedule information generation means for generating at least a part of schedule information in which a plurality of operation instructions for causing the avatar to perform a predetermined operation are arranged in accordance with the passage of time.
A voice information acquisition means that acquires voice information from its own recording means or its own storage unit, or from another computer via a network.
Based on the basic information, the schedule information, and the audio information, the moving image data generation means for generating the moving image data including the moving image with the sound in which the avatar performs a plurality of actions with the passage of time. Prepare,
As the schedule information generation means, a moving image creating device provided with a schedule information automatic generation means for automatically generating at least a part of the schedule information so that the mouth of the avatar operates in synchronization with the voice. A recording means for recording the user's voice is provided.
The moving image creating device according to claim 1, wherein the user's voice recorded by the recording means is used as the voice information acquired by the voice information acquisition means. The moving image creation device according to claim 2, further comprising a synthetic voice generating means capable of changing the voice information of the user recorded by the recording means into synthetic voice. A music information acquisition means for acquiring music information used for video creation from one's own storage unit or from another computer via a network is provided.
The moving image creating device according to claim 1, wherein the voice information acquisition means is configured to acquire voice information from the music information acquired by the music information acquisition means. The music information includes lyrics information and voice information of the music.
A lyric information changing means for editing the lyric information of the music selected by the voice information acquiring means is provided.
The fourth aspect of claim 4, wherein the schedule information automatic generation means is configured to change the movement of the avatar's mouth according to the changed lyrics when the lyrics information is edited by the lyrics information changing means. Video creator. The voice information acquisition means is configured to be able to acquire voice information regarding the type of voice uttered by the avatar from its own storage unit or from another computer via a network. The moving image creation device according to 5. The schedule information generated by the schedule information generation means is described in any one of claims 1 to 6, which includes position information in which a plurality of information regarding left and right positions in a screen displaying the avatar are arranged in accordance with the passage of time. Video creation device. The moving image according to any one of claims 1 to 7, wherein the schedule information generated by the schedule information generation means includes scale information in which a plurality of information regarding the size of displaying the avatar is arranged in accordance with the passage of time. Device. The avatar is configured to have three-dimensional information.
The moving image according to any one of claims 1 to 8, wherein the schedule information generated by the schedule information generating means includes rotation position information in which a plurality of information regarding the direction in which the avatar is displayed is arranged in accordance with the passage of time. Device. A video creation system that creates videos that include avatars.
Memory and
Basic information acquisition means for acquiring basic information about the avatar from the storage unit,
A schedule information generation means for generating at least a part of schedule information in which a plurality of operation instructions for causing the avatar to perform a predetermined operation are arranged in accordance with the passage of time.
A voice information acquisition means for acquiring voice information from its own recording means or the storage unit, and
Based on the basic information, the schedule information, and the audio information, the moving image data generation means for generating the moving image data including the moving image with the sound in which the avatar performs a plurality of actions with the passage of time. Prepare,
As the schedule information generation means, a moving image creating system characterized by providing a schedule information automatic generation means for automatically generating at least a part of the schedule information so that the mouth of the avatar operates in synchronization with the voice. A video creation program that creates videos that include avatars.
On the computer
Basic information acquisition processing to acquire basic information about the avatar from its own memory or from another computer via the network.
A schedule information generation process that generates at least a part of schedule information in which a plurality of operation instructions for causing the avatar to perform a predetermined operation are arranged in accordance with the passage of time.
A voice information acquisition process of acquiring voice information from one's own recording means or one's own storage unit, or from another computer via a network.
A video data generation process for generating video data including a video with audio in which the avatar performs a plurality of actions with the passage of time based on the basic information, the schedule information, and the audio information.
As the schedule information generation process, a moving image creation program for executing a schedule information automatic generation process that automatically generates at least a part of the schedule information so that the mouth of the avatar operates in synchronization with the voice.

Images