JP2021132393A - Moving image distribution system for live distribution of moving image including animation of character object generated on the basis of motion of actor - Google Patents

Abstract

To provide a moving image distribution system, method, and program which facilitate coping with an unexpected situation occurring during live distribution.SOLUTION: A moving image distribution system 1 comprises: a server device 20 which performs live distribution of a moving image including an animation of a character object generated on the basis of actor's motion to a first client device used by a first user; and a supporter computer 40 which prohibits posting a message to a moving image to the first user based on fifth operation input.SELECTED DRAWING: Figure 1

Description

The disclosure in the present specification relates to a video distribution system for live distribution of a video including an animation of a character object generated based on the movement of an actor.

A video distribution system is known that generates an animation of a character object based on the movement of an actor and delivers a video including the animation of the character object live. Such a moving image distribution system is disclosed in, for example, Japanese Patent Application Laid-Open No. 2015-184689 (Patent Document 1).

JP 2015-184689

During live video distribution, unforeseen circumstances such as malfunctions in the equipment used to generate the animation may occur. In the above-mentioned conventional video distribution system, live distribution is performed only by the actor and the operator of the camera. When an actor or a camera operator deals with an unexpected situation, the movement for dealing with the unexpected situation may be reflected in the movement of the character object, or the camera work may become inappropriate. As a result, there may be an undesired effect on the quality of the delivered video.

An object of the present disclosure is to provide technical improvements that solve or alleviate at least some of the problems of the prior art described above. One of the more specific purposes of the present disclosure is to provide a video distribution system that facilitates coping with unforeseen circumstances that occur during live distribution.

The video distribution system according to one aspect is a video distribution system according to one aspect, in which a moving image including an animation of a character object generated based on the movement of an actor is live on a first display device and a first client device used by a first user. The first additional information is displayed on the distribution server device configured for distribution and the moving image displayed on the first display device, while the first additional information is not displayed on the first client device. It is equipped with a supporter computer configured in. The first display device may be used by the actor. The first display device may display an object that can be selected by the actor.

In one aspect, the supporter computer causes the moving image displayed on the first display device and the moving image displayed on the first client device to display the second additional information based on the second operation input. It is composed of.

In one aspect, the supporter computer is configured to generate a modified character object that modifies at least a part of the character object based on a third operation input, and includes the modified character object in the moving image.

In one aspect, the character object has a face portion that is generated to move based on face motion data indicating the movement of the face of the actor, and the distribution server device is based on the third operation input. It is configured to display the third additional information on the face portion.

In one aspect, the supporter computer further includes a storage for storing a decoration object displayed in association with the character object, and the supporter computer displays the decoration object in the moving image based on the fourth operation input. It is configured to display a blind object to hide at least part of it.

In one aspect, the distribution server device is configured to live deliver the video to a second client device used by a second user in addition to the first client device, the first client device being a first type. The second client device is a second type device different from the first type device, and the supporter computer is a display image of the moving image displayed on the first client device and a display image of the moving image. A second display device for displaying a display image of the moving image displayed on the second client device is provided.

In one aspect, the supporter computer is configured to prohibit the distribution of the moving image to the first user based on the fifth operation input.

One aspect relates to a video distribution method executed by one or more computer processors executing computer-readable instructions. The video distribution method includes a step of live distribution of a video including an animation of a character object generated based on the movement of an actor to a first display device and a first client device used by a first user, and the first display device. The first client device is provided with a step of displaying the first additional information on the moving image displayed on the screen and not displaying the first additional information on the first client device.

The distribution program according to one aspect is a live distribution of a moving image including an animation of a character object generated based on the movement of an actor to one or more computer processors to a first display device and a first client device used by a first user. And the step of displaying the first additional information on the moving image displayed on the first display device while not displaying the first additional information on the first client device.

According to an embodiment of the present invention, it is possible to provide a moving image distribution system that facilitates coping with an unexpected situation that occurs during live distribution.

It is a block diagram which shows the moving image distribution system by one Embodiment. It is a schematic diagram which shows typically the equipment of the studio which produces the moving image distributed by the moving image distribution system of FIG. It is a figure explaining the possession list stored in the moving image distribution system of FIG. It is a figure explaining the candidate list stored in the moving image distribution system of FIG. It is a figure which shows the example of the moving image which is displayed on the client apparatus 10a in one Embodiment. FIG. 5 includes an animation of the character object. It is a figure which shows the example of the moving image which is displayed on the client apparatus 10a in one Embodiment. FIG. 6 usually includes an object. It is a figure which shows the example of the moving image which is displayed on the client apparatus 10a in one Embodiment. FIG. 7 includes decorative objects. It is a schematic diagram which shows the example of the decoration object selection screen for selecting a desired selection decoration object from the decoration objects included in a candidate list. It is a figure which shows the example of the image displayed on the display 44 of the supporter computer 40 in one embodiment. It is a figure which shows the example of the moving image which is displayed on the display 39 in one Embodiment. FIG. 10 shows an example of the first additional information. It is a figure which shows the example of the moving image which is displayed on the client apparatus 10a in one Embodiment. FIG. 11a contains the second additional information. It is a figure which shows the example of the moving image which is displayed on the client apparatus 10a in one Embodiment. FIG. 11b contains the second additional information. It is a figure which shows the example of the moving image which is displayed on the client apparatus 10a in one Embodiment. FIG. 12a contains a third additional information. It is a figure which shows the example of the moving image which is displayed on the client apparatus 10a in one Embodiment. FIG. 12b contains a third additional piece of information. It is a figure which shows the example of the moving image which is displayed on the client apparatus 10a in one Embodiment. FIG. 13 includes a blind object. It is a flow figure which shows the flow of the moving image distribution processing in one Embodiment. It is a flow chart which shows the flow of the process which displays the 1st additional information in one Embodiment. It is a flow chart which shows the flow of the process which displays the 2nd additional information in one Embodiment. It is a flow chart which shows the flow of the process which displays the 3rd additional information in one Embodiment. It is a flow chart which shows the flow of the process of displaying a blind object in one Embodiment. It is a flow diagram which shows the flow of the process which prohibits the distribution of a moving image to the distribution prohibition user in one Embodiment.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings as appropriate. The same or similar components are designated by the same reference numerals in a plurality of drawings.

A moving image distribution system according to an embodiment will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram showing a video distribution system 1 according to an embodiment, and FIG. 2 is a schematic diagram schematically showing equipment of a studio in which a video distributed by the video distribution system 1 is produced. 3 and 4 are diagrams for explaining information stored in the moving image distribution system 1.

The video distribution system 1 includes client devices 10a to 10c, a server device 20, a studio unit 30, and a storage 60. The client devices 10a to 10c, the server device 20, and the storage 60 are connected to each other so as to be able to communicate with each other via the network 50. The server device 20 is configured to deliver a moving image including a character animation, as will be described later. The characters included in this moving image may be motion-controlled in the virtual space.

This moving image can be distributed from the server device 20 to each of the client devices 10a to 10c. The first viewing user who is the user of the client device 10a, the second viewing user who is the user of the client device 10b, and the third viewing user who is the user of the client device 10c view the delivered moving image by each client device. be able to. The video distribution system 1 may include a number of client devices less than three, or may include more than three client devices. In the present specification, when it is not necessary to distinguish the client devices 10a to 10c and the client devices other than these from each other, these may be collectively referred to as "client devices".

The client devices 10a to 10c are information processing devices such as smartphones. In addition to smartphones, the client devices 10a to 10c may be mobile phones, tablet terminals, personal computers, electronic book readers, wearable computers, game consoles, and various information processing devices capable of reproducing moving images other than these. Each of the client devices 10a to 10c includes a computer processor, a memory, a communication I / F, a display, a sensor unit having various sensors such as a gyro sensor, a sound collecting device such as a microphone, and a storage for storing various information. May be good.

The viewing user can input a message regarding the delivered moving image via the input interface of the client devices 10a to 10c and post this message to the server device 20. The message posted by each viewing user may be superimposed on the moving image and displayed. As a result, interaction between viewing users is achieved.

In the illustrated embodiment, the server device 20 includes a computer processor 21, a communication I / F 22, and a storage 23.

The computer processor 21 is an arithmetic unit that loads various programs that realize an operating system and various functions from the storage 23 or other storage into a memory and executes instructions included in the loaded programs. The computer processor 21 is, for example, a CPU, an MPU, a DSP, a GPU, various arithmetic units other than these, or a combination thereof. The computer processor 21 may be realized by an integrated circuit such as an ASIC, PLD, FPGA, or MCU. Although the computer processor 21 is illustrated as a single component in FIG. 1, the computer processor 21 may be a collection of a plurality of physically separate computer processors. In the present specification, the program described as being executed by the computer processor 21 or the instructions contained in the program may be executed by a single computer processor or may be executed by a plurality of computer processors in a distributed manner. May be good. Further, the program executed by the computer processor 21 or the instructions included in the program may be executed by a plurality of virtual computer processors.

The communication I / F 22 is implemented as hardware, firmware, communication software such as a TCP / IP driver or PPP driver, or a combination thereof. The server device 20 can send and receive data to and from other devices via the communication I / F 22.

The storage 23 is a storage device accessed by the computer processor 21. The storage 23 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various storage devices other than those capable of storing data. Various programs can be stored in the storage 23. At least a part of the program and various data that can be stored in the storage 23 may be stored in a storage (for example, a storage 60) that is physically separate from the server device 20.

Many of the components of the studio unit 30 are arranged in, for example, the studio room R shown in FIG. As shown in the figure, in the studio room R, actors A1 and A2 are performing. The studio unit 30 is configured to detect the movements and facial expressions of the actors A1 and A2 and output the detected information to the server device 20.

Both the actor A1 and the actor A2 are objects whose movements and facial expressions are captured by a group of sensors described later provided in the studio unit 30. Actor A1 and Actor A2 are, for example, humans, animals, or moving objects performing. The actor A1 and the actor A2 may be, for example, robots capable of independent movement. The number of actors in the studio room R may be 1 or 3 or more.

The studio unit 30 includes six motion sensors 31a to 31f mounted on the actor A1, a controller 33a held by the left hand of the actor A1, a controller 33b held by the right hand of the actor A1, and a fitting 37b. It has a camera 37a attached to the head of the actor A1 via the camera 37a. The studio unit 30 also includes six motion sensors 32a to 32f mounted on the actor A2, a controller 34a held on the left hand of the actor A2, a controller 34b held on the right hand of the actor A2, and a fitting 38b. It has a camera 38a attached to the head of the actor A2 via the camera 38a. A microphone for acquiring audio data may be attached to each of the fittings 37b and the fittings 38b. This microphone can acquire the utterances of the actor A1 and the actor A2 as voice data. The microphone may be a wearable microphone mounted on the actors A1 and A2 via the fittings 37b and 38b, and may be a mounting type microphone installed on the floor, wall, or ceiling of the studio room R. It may be. In addition to the above components, the studio unit 30 has a base station 35a, a base station 35b, a tracking sensor 36a, a tracking sensor 36b, and a display 39. A supporter computer 40 is installed in the adjacent room separated from the studio room R by a glass window. The server device 20 may be installed in the same room where the supporter computer 40 is installed.

The motion sensors 31a to 31f and the motion sensors 32a to 32f cooperate with the base station 35a and the base station 35b to detect their own positions and orientations. In one embodiment, the base station 35a and the base station 35b are multi-axis laser emitters. After emitting the flashing light for synchronization, the base station 35a scans the laser light, for example, around the vertical axis. The base station 35b scans the laser beam, for example, around a horizontal axis. The motion sensors 31a to 31f and the motion sensors 32a to 32f may each include a plurality of optical sensors that detect the incident of blinking light and laser light from the base station 35a and the base station 35b, respectively. The motion sensors 31a to 31f and the motion sensors 32a to 32f have the time difference between the incident timing of the blinking light and the incident timing of the laser light, the light receiving time at each optical sensor, the incident angle of the laser light detected by each optical sensor, and the incident angle of the laser light detected by each optical sensor, respectively. Information other than these may be detected if necessary. The motion sensors 31a to 31f and the motion sensors 32a to 32f may be, for example, a Vive Tracker provided by HTC CORPORATION. The base station 35a and the base station 35b may be, for example, a base station provided by HTC CORPORATION. Three or more base stations may be provided. The position of the base station may be changed as appropriate. For example, in addition to or in place of the base station located in the upper corner of the space to be detected by the tracking sensor shown in FIG. 2, the upper position and the lower position near the floor surface are paired. An arranged base station may be provided.

The detection information (hereinafter, may be referred to as “tracking information”) detected by each of the motion sensors 31a to 31f and the motion sensors 32a to 32f is transmitted to the server device 20. This detection information may be wirelessly transmitted to the server device 20 from each of the motion sensors 31a to 31f and the motion sensors 32a to 32f. Since the base station 35a and the base station 35b emit flashing light and scan the laser light at regular intervals, the tracking information of each motion sensor is updated at each interval. Based on the tracking information detected by the motion sensors 31a to 31f and the motion sensors 32a to 32f, the positions and orientations of the motion sensors 31a to 31f and the motion sensors 32a to 32f can be calculated. The positions and orientations of the motion sensors 31a to 31f and the motion sensors 32a to 32f may be calculated, for example, in the server device 20 based on the tracking information.

In the illustrated embodiment, the actor A1 is equipped with six motion sensors 31a to 31f. The motion sensors 31a, 31b, 31c, 31d, 31e, and 31f are attached to the left wrist, right wrist, left instep, right instep, waist, and crown of the actor A1, respectively. The motion sensors 31a to 31f may be attached to the actor A1 via an attachment. The actor A2 is equipped with six motion sensors 32a to 32f. The motion sensors 32a to 32f can be mounted on the actor A2 at the same positions as the motion sensors 31a to 31f. The motion sensors 31a to 31f and the motion sensors 32a to 32f shown in FIG. 2 are examples. The motion sensors 31a to 31f can be attached to various parts of the actor A1, and the motion sensors 32a to 32f can be attached to various parts of the actor A2. The number of motion sensors mounted on the actors A1 and A2 may be 5 or less, or 7 or more. In this way, by detecting the positions and orientations of the motion sensors 31a to 31f and the motion sensors 32a to 32f mounted on each part of the body of the actor A1 and the actor A2, the movement of the body of the actor A1 and the actor A2 is detected. be able to.

In one embodiment, a large number of infrared LEDs are mounted on each of the plurality of motion sensors mounted on the actor A1 and the actor A2, and the light from the infrared LEDs is provided on the floor or wall of the studio room R. The position and orientation of each of the motion sensors may be detected by detecting with an infrared camera. A visible light LED may be used instead of the infrared LED, and the position and orientation of each of the motion sensors may be detected by detecting the light from the visible light LED with a visible light camera. In this way, each of the plurality of motion sensors mounted on the actor is provided with a light emitting unit (for example, an infrared LED or a visible light LED), and the light from the light emitting unit is provided in the studio room R as a light receiving unit (for example). For example, the position and orientation of each of the motion sensors may be detected by detecting with an infrared camera or a visible light camera).

In one embodiment, a plurality of reflection markers can be used instead of the motion sensors 31a to 31f and the motion sensors 32a to 32f. The reflection marker is attached to each of the actor A1 and the actor A2 with an adhesive tape or the like. The position and orientation of the reflection marker can be detected by photographing the actors A1 and A2 to which the reflection marker is attached in this way to generate shooting data, and performing image processing on the shooting data.

The controller 33a and the controller 33b output a control signal corresponding to the operation of the actor A1 to the server device 20. Similarly, the controller 34a and the controller 34b output a control signal corresponding to the operation of the actor A2 to the server device 20.

The tracking sensor 36a and the tracking sensor 36b generate tracking information for determining the setting information of the virtual camera for constructing the virtual space included in the moving image. The tracking information of the tracking sensor 36a and the tracking sensor 36b is calculated as a position in each three-dimensional Cartesian coordinate system and an angle around each axis. The position and orientation of the tracking sensor 36a can be changed according to the operation of the operator. The tracking sensor 36a transmits tracking information indicating its position and orientation to the tracking information server device 20. Similarly, the position and orientation of the tracking sensor 36b are set according to the operation of the operator. The tracking sensor 36b transmits tracking information indicating its position and orientation to the tracking information server device 20. The tracking sensors 36a, 36b may be supported by a gimbal or stabilizer. The gimbal may be configured in a shape that can be grasped by an actor or a supporter. By supporting the tracking sensors 36a and 36b with a gimbal or a stabilizer, it is possible to suppress camera shake.

The camera 37a is attached to the head of the actor A1 as described above. For example, the camera 37a is arranged so that the face of the actor A1 can be imaged. The camera 37a continuously images the face of the actor A1 and acquires the image data of the face of the actor A1. Similarly, the camera 38a is attached to the head of the actor A2. The camera 38a is arranged so that the face of the actor A2 can be imaged, the face of the actor A2 is continuously imaged, and the imaging data of the face of the actor A2 is acquired. The camera 37a transmits the image capturing data of the face of the actor A1 to the server device 20, and the camera 38a transmits the imaging data of the face of the actor A1 to the server device 20. The camera 37a and the camera 38a may be 3D cameras capable of detecting the depth of a person's face.

The display 39 is configured to display information received from the supporter computer 40. The information transmitted from the supporter computer 40 to the display 39 may include, for example, text information, image information, and various other information. The display 39 is arranged at a position visible to the actors A1 and A2. The display 39 is an example of the first display device.

In the illustrated embodiment, the supporter computer 40 is installed in a room adjacent to the studio room R. Since the room in which the supporter computer 40 is installed and the studio room R are separated by a glass window, the operator of the supporter computer 40 (sometimes referred to as "supporter" in the present specification) is the actor A1 and the actor A2. Can be visually recognized. In the illustrated embodiment, the supporter B1 and the supporter B2 are present as operators of the supporter computer 40. In the present specification, when it is not necessary to distinguish between supporter B1 and supporter B2, they may be collectively referred to as supporter B1 and supporter B2 "supporter".

The supporter computer 40 may be configured so that the settings of the components of the studio unit 30 can be changed according to the operations of the supporter B1 and the supporter B2. The supporter computer 40 can, for example, set the scanning interval by the base station 35a and the base station 35b, set the position or orientation of the tracking sensor 36a and the tracking sensor 36b, and change various settings of various other devices. .. At least one of the supporter B1 and the supporter B2 can input a message to the supporter computer 40 and display the input message on the display 39.

The components and functions of the studio unit 30 shown in FIG. 2 are exemplary. The studio unit 30 applicable to the present invention may include various components (not shown). For example, the studio unit 30 may include a projector. The projector can project a moving image delivered to the client device 10a or other client devices onto the screen S.

Next, in one embodiment, the information stored in the storage 23 will be described. In the illustrated embodiment, the storage 23 stores model data 23a, object data 23b, possession list 23c, candidate list 23d, and various information necessary for generation and distribution of distribution moving images other than the above.

The model data 23a is model data for generating a character animation. The model data 23a may be three-dimensional model data for generating a three-dimensional animation, or may be two-dimensional model data for generating two-dimensional model data. The model data 23a includes, for example, rig data indicating the skeleton of the character (sometimes referred to as “skeleton data”) and surface data indicating the shape and texture of the surface of the character. The model data 23a can include a plurality of model data different from each other. The plurality of model data may have different rig data from each other, or may have the same rig data. The plurality of model data may have different surface data from each other, or may have the same surface data. In the illustrated embodiment, the model data 23a has at least two types of model data that are different from each other in order to generate the character object corresponding to the actor A1 and the character object corresponding to the actor A2. The model data for the character object corresponding to the actor A1 and the model data for the character object corresponding to the actor A2 have, for example, the same rig data, but may have different surface data.

The object data 23b includes asset data for constructing a virtual space constituting a moving image. The object data 23b is data for drawing the background of the virtual space constituting the moving image, data for drawing various objects displayed in the moving image, and various objects displayed in the moving image other than these. Contains data. The object data 23a may include object position information indicating the position of the object in the virtual space.

In addition to the above, the object data 23b may include a gift object to be displayed in the moving image based on a display request from the viewing user of the client devices 10a to 10c. Gift objects can include effect objects, regular objects, and decoration objects. The viewing user can purchase the desired gift object. There may be an upper limit on the number of objects that can be purchased or the purchase price of the objects.

The effect object is an object that affects the impression of the entire viewing screen of the distributed moving image, and is, for example, an object that imitates confetti. The object imitating confetti may be displayed on the entire viewing screen, whereby the impression of the entire viewing screen before and after the display can be changed. Effect objects may appear to overlap with character objects, but differ from decorative objects in that they are not displayed in association with a particular part of the character object.

A normal object is an object that becomes a digital gift from a viewing user to an actor (for example, actor A1 or actor A2), for example, an object that imitates a stuffed animal or a bouquet. In one aspect, the normal object is displayed on the moving image display screen so as not to come into contact with the character object. In one aspect, the normal object is displayed on the moving image display screen so as not to overlap with the character object. Ordinary objects may appear to overlap objects other than character objects in virtual space. Ordinary objects may appear to overlap with character objects, but differ from decorative objects in that they are not displayed in association with a particular part of the character object. In one aspect, when the normal object is displayed overlapping with the character object, the normal object overlaps with a part other than the head including the face of the character object and does not overlap with the head of the character object. Is displayed.

A decoration object is an object displayed on a display screen in association with a specific part of a character object. In one aspect, the decorative object displayed on the display screen in association with a specific part of the character object is displayed on the display screen so as to be in contact with the specific part of the character object. In one aspect, the decorative object displayed on the display screen in association with a specific part of the character object is displayed on the display screen so as to cover a part or all of the specific part of the character object.

The decorative object is, for example, an accessory (catucher, necklace, earring, etc.) attached to the character object, clothes (T-shirt, etc.), a costume, and an object that can be attached to other character objects. The object data 23b corresponding to the decoration object may include mounting position information indicating which part of the character object the decoration object is associated with. The mounting position information of a certain decorative object can indicate to which part of the character object the decorative object is mounted. For example, when the decoration object is a Katyusha, the mounting position information of the decoration object may indicate that the decoration object is mounted on the "head" of the character object. When the decorative object is a T-shirt, the mounting position information of the decorative object may indicate that the decorative object is mounted on the "body" of the character object.

A display time may be set for each of the gift objects according to the type of the gift object. In one aspect, the display time of the decorative object may be set longer than the display time of the effect object and the display time of the normal object. For example, the display time of the decorative object may be set to 60 seconds, the display time of the effect object may be set to 5 seconds, and the display time of the normal object may be set to 10 seconds.

The possession list 23c is a list showing the gift objects possessed by the viewing user of the moving image. An example of the holding list 23c is shown in FIG. As shown in the figure, in the possession list 23c, an object ID that identifies a gift object possessed by the viewing user is stored in association with the account information of the viewing user (for example, the user ID of the viewing user). The viewing users include, for example, first viewing users to third viewing users of the client devices 10a to 10c.

The candidate list 23d is a list of decorative objects for which a display request has been made by the viewing user. As will be described later, the viewing user who owns the decorative object can make a display request for the decorative object. In the candidate list 23d, the object ID that identifies the decoration object is stored in association with the account information of the viewing user who has requested the display of the decoration object. The candidate list 23d may be created for each distributor. The candidate list 23d may be stored, for example, in association with the distributor identification information that identifies the distributor (actor A1, actor A2, supporter B1, and / or supporter B2).

Next, the functions realized by the computer processor 21 will be described more specifically. The computer processor 21 executes a computer-readable instruction included in the distribution program to execute the body motion data generation unit 21a, the face motion data generation unit 21b, the animation generation unit 21c, the video generation unit 21d, the video distribution unit 21e, and the like. It functions as a display request processing unit 21f, a decorative object selection unit 21g, and an object purchase processing unit 21h. At least a part of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the moving image distribution system 1. At least a part of the functions realized by the computer processor 21 may be realized by, for example, the computer processor mounted on the supporter computer 40.

The body motion data generation unit 21a generates first body motion data of each part of the body of the actor A1 based on the tracking information detected by each of the motion sensors 31a to 31f, and also generates the first body motion data of the motion sensors 32a to 32f. Based on the tracking information detected by each, the second body motion data which is a digital representation of the position and orientation of each part of the body of the actor A2 is generated. In the present specification, the first body motion data and the second body motion data may be collectively referred to simply as "body motion data". Body motion data is generated at any time over time. For example, body motion data may be generated at predetermined sampling time intervals. In this way, the body motion data can represent the body movements of the actors A1 and A2 as digital data in chronological order. In the illustrated embodiment, since the motion sensors 31a to 31f and the motion sensors 32a to 32f are attached to the left and right limbs, hips, and head of each of the actors A1 and A2, the motion sensors 31a to 31f and the motion sensors 31a to 31f Based on the tracking information detected by the motion sensors 32a to 32f, it is possible to digitally represent the positions and orientations of the actors A1 and the actors A2 over time. The body motion data can define, for example, the position and rotation angle of the bones constituting the rig data included in the model data 23a.

The face motion data generation unit 21b generates first face motion data which is a digital representation of the face movement of the actor A1 based on the image data of the camera 37a, and also based on each image data of the camera 38a. Generates second face motion data, which is a digital representation of the face movement of actor A2. In the present specification, the first face motion data and the second face motion data may be collectively referred to simply as "face motion data". Face motion data is generated at any time with the passage of time. For example, face motion data may be generated at predetermined sampling time intervals. In this way, the face motion data can digitally represent the facial movements (changes in facial expressions) of the actor A2 and the actor A2 in chronological order.

The animation generation unit 21c applies the body motion data generated by the body motion data generation unit 21a and the face motion data generated by the face motion data generation unit 21b to predetermined model data included in the model data 23a. It is configured to generate animations for character objects that move in virtual space and have changing facial expressions. Specifically, the animation generation unit 21c generates an animation of a character object that moves in synchronization with the movement of the body and facial expression of the actor A1 based on the first body motion data and the first face motion data related to the actor A1. Based on the second body motion data and the second face motion data related to the actor A2, it is possible to generate an animation of a character object that moves in synchronization with the movement of the body and facial expression of the actor A2. In the present specification, the character object generated based on the movement and facial expression of actor A1 is referred to as "first character object", and the character object generated based on the movement and facial expression of actor A2 is referred to as "second character object". There is a case.

The moving image generation unit 21d constructs a virtual space using the object data 23b, and creates the virtual space, the animation of the first character object corresponding to the actor A1, and the animation of the second character object corresponding to the actor A2. Generate a video that contains. The first character object is arranged in the virtual space so as to match the position of the actor A1 with respect to the tracking sensor 36a, and the second character object is arranged in the virtual space so as to match the position of the actor A2 with respect to the tracking sensor 36a. Will be done. Therefore, by changing the position or orientation of the tracking sensor 36a, the positions and orientations of the first character object and the second character object in the virtual space can be changed.

In one aspect, the moving image generation unit 21d constructs a virtual space based on the tracking information of the tracking sensor 36a. For example, the moving image generation unit 21d determines the setting information of the virtual camera (position in the virtual space, gaze position, gaze direction, and angle of view) based on the tracking information of the tracking sensor 36a, and is based on the setting information of the virtual camera. The drawing area of the entire virtual space is defined, and moving image information for displaying the drawing area of this virtual space is generated.

The moving image generation unit 21d replaces the tracking information of the tracking sensor 36a, or based on the tracking information of the tracking sensor 36b in addition to the tracking information of the tracking sensor 36a, the first character object and the second character in the virtual space. It may be configured to determine the position and orientation of the object and the setting information of the virtual camera.

The moving image generation unit 21d can synthesize the sound of the actor A1 and the actor A2 acquired from the microphone of the studio unit 30 with the generated moving image.

As described above, the moving image generation unit 21d has the animation of the first character object that moves in synchronization with the movement of the body and facial expression of the actor A1, and the second character that moves in synchronization with the movement of the body and facial expression of the actor A2. It is possible to generate an animation of an object and generate a moving image for distribution in which the voices of actors A1 and A2 are combined with this animation.

The video distribution unit 21e distributes the video generated by the video generation unit 21d. This moving image is distributed to the client devices 10a to 10c and other client devices via the network 50. The video distribution unit 21e refers to a list of users who have requested distribution of the video (distribution destination list), and distributes the video to the client devices of the users included in the list. As will be described later, the distribution of a specific moving image to a specific user may be prohibited by an instruction from the supporter computer 40. In this case, the video is distributed to users other than the users whose distribution of the video is prohibited. The received moving image is reproduced in the client device 10a to the client device 10c.

This moving image may be distributed to a client device (not shown) installed in the studio room R, and may be projected onto the screen S from this client device via a short focus projector. Further, this moving image may be distributed to the supporter computer 40. As a result, the supporter B1 and the supporter B2 can confirm the viewing screen of the distributed moving image.

FIG. 5 shows a display example of a moving image distributed from the server device 20 to the client device 10a and reproduced in the client device 10a. As shown in the figure, the display image 70 of the moving image delivered from the server device 20 is displayed on the display of the client device 10a. The display image 70 displayed on the client device 10a includes a character object 71A corresponding to the actor A1, a character object 71B corresponding to the actor A2, and an object 72 indicating a table in the virtual space. There is. The object 72 is not a gift object, but one of the objects for constructing the virtual space included in the object data 23b. The character object 71A is generated by applying the first body motion data and the first face motion data of the actor A1 to the model data for the actor A1 included in the model data 23a. The character object 71A is motion-controlled based on the first body motion data and the first face motion data. The character object 71B is generated by applying the second body motion data and the second face motion data of the actor A2 to the model data for the actor A2 included in the model data 23a. The character object 71B is motion-controlled based on the second body motion data and the second face motion data. Therefore, the character object 71A is controlled to move in the screen in synchronization with the movement of the body and facial expression of the actor A1, and the character object 71B is controlled to move in the screen in synchronization with the movement of the body and facial expression of the actor A2. Is controlled by.

As described above, the distribution moving image from the server device 20 may be distributed to the supporter computer 40. The moving image delivered to the supporter computer 40 is displayed on the supporter computer 40 in the same manner as in FIG. The supporter B1 and the supporter B2 can change the settings of the components of the studio unit 30 by watching the moving image being played on the supporter computer 40. In one aspect, when the supporter B1 and the supporter B2 want to change the angles of the character object 71A and the character object 71B in the moving image being distributed, the supporter computer 40 tracks an instruction signal instructing the change of the orientation of the tracking sensor 36a. It can be transmitted to the sensor 36a. The direction of the tracking sensor 36a can be changed according to the instruction signal. For example, the tracking sensor 36a is rotatably attached to a stand via a rotation mechanism having an actuator built around its axis, and when the tracking sensor 36a receives an instruction signal instructing the change of its orientation, the tracking sensor 36a is said to be rotatably attached. The actuator of the rotation mechanism may be driven based on the instruction signal, and the tracking sensor 36a may be rotated by an angle corresponding to the instruction signal. In one aspect, the supporter B1 and the supporter B2 transmit an instruction for using the tracking information from the tracking sensor 36b from the supporter computer 40 to the tracking sensor 36a and the tracking sensor 36b instead of the tracking information from the tracking sensor 36a. May be good. The tracking sensor 36a and the tracking sensor 36b may be configured and installed so as to be movable by an actor or a supporter. As a result, the actor or supporter can move the tracking sensor 36a and the tracking sensor 36b by holding them in their hands.

In one aspect, when the supporter B1 and the supporter B2 see the moving image being played on the supporter computer 40 and determine that an instruction to the actor A1 or the actor A2 is necessary, the supporter B1 and the supporter B2 send a message indicating the instruction to the supporter computer 40. This message can be input and output to the display 39. For example, the supporter B1 and the supporter B2 can instruct the actor A1 or the actor A2 to change the standing position through the message displayed on the display 39.

The display request processing unit 21f receives a display request for the gift object from the client device of the viewing user, and performs processing according to the display request. Each viewing user can send a gift object display request to the server device 20 by operating his / her own client device. For example, the first viewing user can send a gift object display request to the server device 20 by operating the client device 10a. The display request of the gift object may include the user ID of the viewing user and the identification information (object ID) that identifies the object for which the display is requested.

As mentioned above, the gift object may include an effect object, a normal object, and a decoration object. The effect object and the normal object are examples of the first object. A display request that requests the display of an effect object or a normal object is an example of a second display request.

In one aspect, when the display request processing unit 21f receives a display request for a specific effect object from the viewing user, the display request processing unit 21f displays the effect object for which the display is requested on the display image 70 of the moving image based on the display request. Perform processing. For example, when a display request for an effect object corresponding to confetti is made, the display request processing unit 21f displays an effect object 73 indicating confetti based on the display request as shown in FIG. 6. Image 70 To display.

In one aspect, when the display request processing unit 21f receives a display request for a specific normal object from the viewing user, the display request processing unit 21f performs a process for displaying the normal object requested to be displayed on the moving image 70 based on the display request. .. For example, when a display request for a normal object showing a teddy bear is made, the display request processing unit 21f displays a display image of the normal object 74 showing the stuffed bear based on the display request, as shown in FIG. Display on 70.

The display request of the normal object 74 may include a display position specification parameter that specifies the display position of the normal object 74 in the virtual space. In this case, the display request processing unit 21f can display the normal object 74 at the position specified by the display position designation parameter in the virtual space. For example, the display position specification parameter can specify the display position of the normal object 74 above the object 72 indicating the table. By using the display position specification parameter, the viewing user sees the display of the character object 71A, the character object 71B, the gift object, and other objects included in the moving image 70, and specifies the position where the normal object is to be displayed. can do.

In one aspect, the normal object 74 may be displayed to move within the moving image display image 70. For example, the normal object 74 may be displayed so as to fall from the top to the bottom of the screen. In this case, the normal object 74 may be displayed in the display image 70 while falling from the start of the fall to the floor of the virtual space constituting the moving image 70, and may disappear from the display image 70 after falling on the floor. .. The viewing user can see the falling normal object 74 from the start of the fall to the end of the fall. The moving direction of the normal object 74 in the screen can be set arbitrarily. For example, the normal object 74 may be displayed in the display image 70 so as to move from left to right, right to left, upper left to lower left, or other directions of the moving image 70. The normal object 74 can move in various orbits. For example, the normal object 74 can move on a linear orbit, a circular orbit, an elliptical orbit, a spiral orbit, or any other orbit. In the display request of the normal object, the viewing user specifies a movement direction parameter that specifies the movement direction of the normal object 74 and a trajectory in which the normal object 74 moves, in addition to the display position specification parameter or instead of the display position specification parameter. At least one of the orbital parameters can be included. In one embodiment, among the effect object and the normal object, those whose size in the virtual space is smaller than the reference size (for example, a piece of confetti of the effect object 73) are attached to the character object 71A and the character object 71B. Part or all of them may be duplicated and displayed. In one embodiment, among the effect object and the normal object, the one whose size in the virtual space is larger than the reference size (for example, the normal object 74 (a stuffed bear)) is displayed at a position that does not overlap with the character object. .. In one embodiment, the effect object and the normal object whose size in the virtual space is larger than the reference size (for example, the normal object 74 (plush bear)) is at least one of the character object 71A and the character object 71B. If it overlaps with, it is displayed after the character object that is displayed in duplicate.

In one aspect, when the display request processing unit 21f receives a display request for a specific decorative object from the viewing user, the display request processing unit 21f adds the decorative object for which the display is requested to the candidate list 23d based on the display request. The display request for the decorative object is an example of the first display request. For example, the display request processing unit 21f stores the identification information (object ID) for identifying a specific decorative object requested to be displayed by the viewing user in the candidate list 23d in association with the user ID of the viewing user. (See Fig. 4). When a plurality of display requests for the decoration object are made, the user ID of the viewing user who made the display request and the decoration object ID of the decoration object whose display request is requested by the viewing user are used for each of the plurality of display requests. Are associated with each other and stored in the candidate list 23d.

In one aspect, the decoration object selection unit 21g displays the selected decoration object on the moving image display image 70 in response to the selection of one or more of the decoration objects included in the candidate list 23d. Perform the process to make it. In the present specification, the decoration object selected from the candidate list 23d may be referred to as a "selected decoration object".

The selection of the decorative object from the candidate list 23d is made, for example, by the supporter B1 and the supporter B2 operating the supporter computer 40. In one aspect, the supporter computer 40 displays a decorative object selection screen. FIG. 8 shows an example of the decorative object selection screen 80 in one aspect. The decorative object selection screen 80 is displayed, for example, on the display of the supporter computer 40. The decoration object selection screen 80 displays each of the plurality of decoration objects included in the candidate list 23d in a table format, for example. As shown in the figure, in the decoration object selection screen 80 in one embodiment, the first column 81 for displaying the type of the decoration object, the second column 82 for displaying the image of the decoration object, and the character object to which the decoration object is associated. It has a third column 83, and a third column 83, which displays the parts of the above. Further, on the decoration object selection screen 80, selection buttons 84a to 84c for selecting each decoration object are displayed. In this way, the decoration object selection screen 80 displays the decoration objects that are candidates for the selection decoration object.

Supporter B1 and supporter B2 can select one or more of the decorative objects included in the decorative object selection screen 80. For example, the supporter B1 and the supporter B2 can select the Katyusha by selecting the selection button 84a. When it is detected by the decoration object selection unit 21g that the cutie is selected, the display request processing unit 21f displays the selection decoration object 75 indicating the selected cutie on the moving image display screen 70 as shown in FIG. Let me. The selection decoration object 75 is displayed on the display image 70 in association with a specific part of the character object. The selection decoration object 75 may be displayed so as to be in contact with the specific part of the character object. For example, since the selection 75 indicating the Katyusha is associated with the head of the character object, it is attached to the head of the character object 71A as shown in FIG. The decorative object may be displayed on the display screen 70 so as to move along with the movement of the specific portion of the character object. For example, when the head of the character object 71A equipped with the Katyusha moves, the selective decoration object 75 indicating the Katyusha is also attached to the head of the character object 71A as if the Katyusha is attached to the head of the character object 71A. Moves.

The selection decoration object 75 may be displayed on the display screen 70 in association with the character object 71B instead of the character object 71A. The selection decoration object 75 may be displayed on the display screen 70 in association with the character object 71A and the character object 71B.

In one embodiment, the decoration object selection screen 80 may be configured not to include information that can identify the user who owns each decoration object or the user who has made a display request for each decoration object. This makes it possible to prevent the chooser from giving preferential treatment to a specific user when selecting a decorative object.

In one embodiment, the decoration object selection screen 80 may display information about the user who owns each decoration object or, for each decoration object, the user who has made a display request for the decoration object. For example, for each decoration object, the number of times the user who made the display request of the decoration object has made the display request of the decoration object and the number of times selected (for example, the number of times the decoration object display request has been made 5 times so far). Information indicating that two of them have been selected), the number of times the user has viewed at least one of the moving images of the character object 71A and the character object B, and the moving image (character object 71A and the character) in the moving image distribution system 1 by the user. Number of views (regardless of whether object B appears), amount of purchase of gift object by the user, number of purchases of the object by the user, points that can be used in the video distribution system 1 owned by the user, video distribution of the user Various information about the level in the system 1 and the user who requested the display of each decoration object other than these may be displayed. According to the embodiment, the decoration object can be selected based on the behavior and viewing history of the user who has requested the display of the decoration object in the video distribution system 1 so far.

In one embodiment, the display of decorative objects may be constrained to eliminate duplication. For example, with respect to the character object 71A, if the decoration object associated with the specific part is already selected, the selection of another decoration object associated with the specific part may be prohibited. As shown in the embodiment of FIG. 7, if the Katyusha associated with the "head" of the character object 71B is already selected, the other decorations associated with the "head". The object (for example, the decorative object representing the "hat" associated with the head) is not displayed on the decorative object selection screen 80, or the selection button for selecting the decorative object representing the hat is selected on the selection screen 80. It may be made unselectable in. According to the embodiment, it is possible to prevent the decorative object from being duplicated in a specific part of the character object.

The decorative object selection screen 80 may be displayed on another device in place of the supporter computer 40 or in addition to the supporter computer 40. For example, the decorative object selection screen 80 may be displayed on at least one of the display 39 and the screen S in the studio room R. In this case, the actor A1 and the actor A2 can select a desired decorative object based on the decorative object selection screen 80 displayed on the display 39 or the screen S. The selection of the decorative object by the actors A1 and A2 can be made, for example, by operating the controller 33a, the controller 33b, the controller 34a, or the controller 34b.

In one aspect, the object purchase processing unit 21h receives the purchase information of each of the plurality of gift objects that can be purchased in connection with the video in response to the request from the viewing user of the video, and the client device (for example, the viewing user) of the viewing user. It is transmitted to the client device 10a). The purchase information for each gift object includes the type of gift object (effect object, regular object, or decoration object), the image of the gift object, the price of the gift object, and the information required to purchase other gift objects. Can be included. The viewing user can select the gift object to be purchased based on the purchase information of the gift object displayed on the client device 10a. The gift object to be purchased may be selected by operating the client device 10a. When the gift object to be purchased is selected by the viewing user, a purchase request for the gift object is transmitted to the server device 20. The object purchase processing unit 21h performs payment processing based on the purchase request. When this payment process is completed, the purchased gift object is held by the viewing user. In this case, the object ID of the purchased gift object is stored in the possession list 23c in association with the user ID of the purchased viewing user.

The gift objects that can be purchased may vary from video to video. The gift objects that can be purchased may be available for purchase in a plurality of videos. That is, the gift objects that can be purchased may include a unique gift object that is unique to each video and a common gift object that can be purchased in a plurality of videos. For example, the effect object indicating confetti may be a common gift object that can be purchased in a plurality of videos.

In one aspect, when an effect object is purchased while watching a predetermined video, the effect object to be purchased becomes the video being viewed in response to the completion of the payment process for purchasing the effect object. It may be displayed automatically. If a normal object is purchased while watching a predetermined video, the normal gift object to be purchased is the video being watched in response to the completion of the payment process for purchasing the normal object, as described above. May be displayed automatically.

In another aspect, a payment completion notification is transmitted to the client device 10a in response to the completion of the payment process in the object purchase processing unit 21h for the effect object to be purchased, and the effect object purchased in the client device 10a. A confirmation screen may be displayed for confirming with the viewing user whether or not to request the display of. When the viewing user chooses to make a display request for the purchased effect object, the display request requesting the display of the purchased effect object is sent from the viewing user's client device to the display request processing unit 21f. The display request processing unit 21f may perform a process of displaying the effect object to be purchased on the moving image 70. Even when the purchase target is a normal object, a confirmation screen for confirming with the viewing user whether or not to request the display of the purchased normal object may be displayed on the client device 10a in the same manner as described above.

Next, the supporter computer 40 in one embodiment will be described. In one embodiment, the supporter computer 40 includes a computer processor 41, a communication I / F 42, a storage 43, a display 44, and an input interface 45.

The computer processor 41 is various arithmetic units such as a CPU, like the computer processor 21. The communication I / F 42, like the communication I / F 22, is various drivers, software, or a combination thereof for communicating with other devices. Similar to the storage 23, the storage 43 is various storage devices capable of storing data such as a magnetic disk. The display 44 is a liquid crystal display, an organic EL display, an inorganic EL display, or any display device capable of displaying an image other than these. The input interface 45 is an arbitrary input interface that accepts input from supporters such as a mouse and a keyboard. The display 44 is an example of the second display device.

FIG. 9 shows an example of an image displayed on the display 44 of the supporter computer 40. In the illustrated embodiment, the display image 46 displayed on the display 44 has a first display area 47a for displaying a moving image display screen on a first type client device (for example, a personal computer) and a second type. A second display area 47b for displaying a moving image display screen on the client device (for example, a personal computer), and a plurality of operation icons 48 are included.

By monitoring the video displayed in the first display area 47a, the supporter can confirm whether the video delivered to the first type client device is normally displayed, and similarly, the first display area 47a. By monitoring the moving image displayed in the 2 display area 47b, it is possible to confirm whether or not the moving image delivered to the second type client device is normally displayed.

In the illustrated embodiment, the operation icon 48 includes a first operation icon 48a, a second operation icon 48b, a third operation icon 48c, a fourth operation icon 48d, and a fifth operation icon 48e. .. The supporter can select a desired operation icon via the input interface 45. When the supporter selects the first operation icon 48a, the second operation icon 48b, the third operation icon 48c, the fourth operation icon 48d, and the fifth operation icon 48e, the supporter computer 40 performs the first operation. Input, second operation input, third operation input, fourth operation input, and fifth operation input are accepted, respectively.

Next, the functions realized by the computer processor 41 will be described more specifically. The computer processor 41 functions as an additional information display unit 41a and a distribution management unit 41b by executing a computer-readable instruction included in a predetermined program. At least a part of the functions realized by the computer processor 41 may be realized by a computer processor other than the computer processor 41 of the moving image distribution system 1. At least a part of the functions realized by the computer processor 41 may be realized by, for example, the computer processor 21.

The additional information display unit 41a is configured to add various additional information to the distributed moving image in response to various operation inputs from the supporter via the input interface 45.

In one embodiment, when the supporter computer 40 receives the first operation input by selecting the first operation icon 48a, the additional information display unit 41a displays the moving image on the display 39 based on the first operation input. Is configured to display the message 101 while not displaying the message 101 on the client device. Message 101 is an example of the first additional information.

FIG. 10 shows an example of a display image of the moving image displayed on the display 39. The display image 100 shown in FIG. 10 includes a message 101. The display image 100 is the same as the display image 70 (see FIG. 5) of the moving image distributed and displayed on the client device of the viewing user at the same timing except that the message 101 is included. That is, the display image 100 displayed on the display 39 is configured by adding the message 101 to the delivered moving image.

The message 101 is, for example, a text-format message input by the supporter. The message 101 may include image information in addition to the text format information or in addition to the text format information. The message 101 can include instructions regarding the performance of the actor, instructions regarding the remarks of the actor, instructions regarding the standing position of the actor, and various instructions and communication matters related to the video being distributed other than the above.

In one embodiment, when the supporter selects the first operation icon 48a on the supporter computer 40, a window for inputting a message is displayed on the display 44 of the supporter computer 40. The supporter can input a message in the message input area in the window by operating the input interface 45. The input message is transmitted from the supporter computer 40 to the display 39. The moving image distributed by the moving image distribution unit 21e is displayed on the display 39 and the client devices 10a to 10c. As shown in FIG. 10, the message received from the supporter computer 40 is displayed as a message 101 on the display 39 in a predetermined area of the moving image distributed from the moving image distribution unit 21e. On the other hand, the moving image displayed on the client devices 10a to 10c does not display the message 101 or the information corresponding thereto. According to the embodiment, it is related to the video being distributed to the distribution side staff who can see the actor or other display 39 without affecting the video viewed by the viewing user of the client device. Can convey instructions and communications.

Subsequently, the display of the second additional information will be described with reference to FIGS. 11a and 11b. In one embodiment, when the supporter computer 40 accepts the second operation input by selecting the second operation icon 48b, the additional information display unit 41a is displayed on the display 39 based on the second operation input. It is configured to display an interrupt image on the moving image and the moving image displayed on the client device. The interrupt image is an example of the second additional information. The second operation input may be input to the server device 20 or the supporter computer 40 in response to the operation of the controller 33a or 33b of the actor A1 or in response to the operation of the controller 34a or 34b of the actor A2.

FIG. 11a shows an example of the display image 110a displayed on the client device 10a when the second operation input is received, and FIG. 11b shows the display 39 when the second operation input is received. An example of the display image 110b displayed in is shown. Both the display image 110a and the display image 110b include the interrupt image 111. The interrupt image 111 is set in the upper layer of the moving image being distributed. As a result, the interrupt image 111 is displayed on the screen of the client device and the display 39 during playback of the moving image. There may be a slight visual difference between the interrupt image 111 displayed on the client device and the interrupt image 111 displayed on the display 39, depending on the performance of the displayed device. It is a substantially the same image.

In the illustrated embodiment, the interrupt image 111 is an image for emergency display for displaying in place of the normal virtual space and the moving image including the character object when an unexpected situation occurs during the moving image distribution. The supporter can select the second operation icon 48b on the supporter computer 40, for example, when the motion control of the character object is hindered due to a failure of the equipment used in the studio room R. The additional information display unit 41a performs a process for displaying the interrupt image 111 on each client device and the display 39 in response to the selection of the second operation icon 48b.

When the first operation input is received by selecting the first operation icon 48a while the interrupt image 111 is displayed on the display 39, the additional information display unit 41a superimposes the message on the interrupt image 111 on the display 39. 101 may be displayed.

According to the above embodiment, when an unexpected situation occurs during the distribution of the moving image, the interrupt image 111 for emergency display is delivered to the client device and the display 39 instead of continuing to deliver the normal moving image as it is. Can be done. The distributor side can try to control the situation while the interrupt image 111 is displayed.

Subsequently, the display of the third additional information and the generation of the modified character object will be described with reference to FIGS. 12a and 12b. In one embodiment, when the supporter computer 40 receives the third operation input by selecting the third operation icon 48c, the additional information display unit 41a generates a modified character object and distributes a moving image including the modified character object. It is configured to perform processing to do so. This third operation input may be input to the server device 20 or the supporter computer 40 in response to the operation of the controller 33a or 33b of the actor A1 or in response to the operation of the controller 34a or 34b of the actor A2.

FIG. 12a shows another example of the display image 120a displayed on the client device 10a, and FIG. 12b shows an example of the display image 120b displayed on the client device 10a. The display image 120a of FIG. 12a includes a character object 71B, and the display image 120a of FIG. 12b includes a modified character object 171B described later. As described above, the facial expression of the character object 71B is controlled to change in synchronization with the movement of the facial expression of the actor A2 based on the face motion data of the actor A2. In the present specification, all or a part of the character object that changes based on the face motion data may be referred to as a "face portion". In the illustrated embodiment, the motion of the eyes of the character object 71B is controlled so as to move in synchronization with the movement of the eyes of the actor A2, so that the face portion 121 is set at a position including both eyes of the character object 71B. Has been done. The face portion 121 may be set to include the entire face of the character object 71B.

In the display image 120a shown in FIG. 12a, the eye image of the character object 71B displayed in the face portion 121 is generated by applying the face motion data to the model data. On the other hand, in the display image 120b shown in FIG. 12b, the eye image displayed in the face portion 121 is an inset image 122b prepared in advance before the start of moving image distribution. The fitted image 122b may be stored in the storage 23. The inset image 122b is an example of the third additional information.

In one embodiment, the additional information display unit 41a is configured to display the fitted image 122b on the face unit 121 of the character object 71B when the third operation input is received. For example, when the third operation input is received, the additional information display unit 41a transmits a correction instruction to the animation generation unit 21c, and the animation generation unit 21c is based on the correction instruction from the additional information display unit 41a. By fitting the fitted image 122b instead of the image generated based on the face motion data into the face portion 121 of the character object 71B, the animation of the modified character object 171B is generated. In the present specification, the character object in which the fitted image 122b is fitted to the face portion 121 instead of the image generated based on the face motion data may be referred to as a modified character object. The modified character object is an object in which a part of the character object generated by the animation generation unit 21c is modified. In the above example, the animation of the modified character object 171B is generated by fitting and displaying the fitted image 122b in the face portion 121 of the character object 71B generated by the animation generation unit 21c.

The process of applying face motion data to model data to change the facial expression of a character object is a heavy processing load for the processor. Therefore, the facial expression of the character object may not follow the facial expression of the actor in a timely manner. Since movements other than voice and facial expressions can timely follow the utterances and body movements of the actor, if the facial expressions of the character object cannot follow the facial expressions of the actor in a timely manner, the viewer will feel uncomfortable. According to the above embodiment, when the facial expression of the character object cannot follow the facial expression of the actor in a timely manner, a modified character object incorporating the inset image 122b prepared in advance is generated, and a moving image including the modified character object is generated. By delivering, it is possible to prevent deterioration of the quality of the moving image due to the facial expression of the character object not being able to follow the facial expression of the actor.

A plurality of different types of images may be stored in the storage 23 as candidates for the fitted image 122b. Candidates for the fitted image 122b are displayed on the display 44 of the supporter computer 40, and an image selected by the supporter from these candidates may be used as the fitted image 122b.

Subsequently, the display of the blind object will be described with reference to FIG. In one embodiment, when the fourth operation input is received by selecting the fourth operation icon 48d, the additional information display unit 41a displays a blind object for hiding at least a part of the character object to which the decoration object is attached. It is configured as follows. This fourth operation input may be input to the server device 20 or the supporter computer 40 in response to the operation of the controller 33a or 33b of the actor A1 or in response to the operation of the controller 34a or 34b of the actor A2.

In the embodiment of FIG. 13, it is assumed that the T-shirt is selected as the decorative object to be attached to the character object 71B on the selection screen 80. The additional information display unit 41a can display the blind object so as to hide the portion of the character object 71B associated with the selected decorative object. In the display image 130 of FIG. 13, the blind object 131 of the character object 71B is displayed so that at least the body portion associated with the T-shirt, which is the selected decoration object, is hidden.

Subsequently, the process of prohibiting the distribution of the moving image to the distribution prohibited user will be described. In one embodiment, the distribution management unit 41b performs a process for prohibiting distribution of a moving image to a specific viewing user or a specific client device when the fifth operation input is received by selecting the fifth operation icon 48e. Configured to do. In one embodiment, when the fifth operation icon 48e is selected during the distribution of the moving image, a list of users who are watching the moving image is displayed. The distribution management unit 41b performs a process for canceling the distribution of the moving image to the user selected by the supporter or the actor from this list. For example, the distribution management unit 41b can write a flag for identifying the selected user (distribution prohibited user) in the distribution destination list of the moving image. The video distribution unit 21e can distribute the video to the users included in the video distribution destination list that are not given the flag for identifying the distribution prohibited user. As a result, the distribution of the video to the distribution prohibited user is stopped. In another embodiment, the distribution management unit 41b cannot use some of the functions normally available to the user selected by the supporter or actor from the list of users who are watching the video being distributed. It may be configured to. For example, the distribution management unit 41b may be configured to prohibit posting of a message to a user who has posted an inappropriate message. A user who is prohibited from posting a message can continue to watch the video even after the prohibition, but cannot post a message related to the video.

Next, with reference to FIGS. 14 to 19, the moving image distribution process according to one aspect will be described. FIG. 14 is a flow chart showing a flow of video distribution processing in one embodiment, FIG. 15 is a flow chart showing a flow of processing for displaying the first additional information in one embodiment, and FIG. 16 is a flow chart showing one. FIG. 17 is a flow chart showing a flow of processing for displaying the second additional information in the embodiment, FIG. 17 is a flow chart showing the flow of processing for displaying the third additional information in one embodiment, and FIG. 18 is a flow chart showing one. FIG. 19 is a flow chart showing a flow of processing for displaying a blind object in the embodiment, and FIG. 19 is a flow chart showing a flow of processing for prohibiting distribution of a moving image to a distribution prohibited user in one embodiment.

First, with reference to FIG. 14, the moving image distribution process in one embodiment will be described. In the video distribution process, it is assumed that actors A1 and A2 are performing in the studio room R. First, in step S11, body motion data which is a digital representation of the body movements of the actors A1 and A2 and face motion data which is a digital representation of the facial movements (facial expressions) of the actors A1 and A2 are generated. NS. The body motion data is generated by, for example, the body motion data generation unit 21a, and the face motion data is generated by the face motion data generation unit 21b, for example.

Next, in step S12, by applying the body motion data and face motion data of the actor A1 to the model data for the actor A1, the animation of the first character object that moves in synchronization with the movement of the body and facial expression of the actor A1. Is generated. Similarly, by applying the body motion data and face motion data of the actor A2 to the model data for the actor A2, an animation of the second character object that moves in synchronization with the movement of the body and facial expression of the actor A2 is generated. .. This animation is generated, for example, by the animation generation unit 21c described above.

Next, in step S13, a moving image including the animation of the first character object corresponding to the actor A1 and the animation of the second character object corresponding to the actor A2 is generated. The audio of actor A1 and actor A2 may be combined with this moving image. The animation of the first character object and the animation of the second character object may be arranged in the virtual space. This moving image is generated, for example, by the moving image generation unit 21d described above.

Next, the process proceeds to step S14, and the moving image generated in step S13 is distributed. This moving image is distributed to the client devices 10a to 10c and other client devices via the network 50. This moving image may be distributed to the supporter computer 40 and projected on the screen S in the studio room R. The video is continuously distributed over a predetermined distribution period. The video distribution period can be set to, for example, 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, 60 minutes, 120 minutes, and any other time.

Next, the process proceeds to step S15, and it is determined whether or not the termination condition for terminating the distribution of the moving image is satisfied. The end condition is, for example, that the end time of the distribution has been reached, that the supporter computer 40 has given an instruction to end the distribution, or other conditions. If the end condition is not satisfied, the processes of steps S11 to S14 are repeatedly executed, and the distribution of the moving image including the animation synchronized with the movements of the actors A1 and A2 is continued. If it is determined that the end condition for the video is satisfied, the distribution process of the video is terminated.

Next, with reference to FIG. 15, the display processing of the first additional information performed during the distribution of the moving image will be described. During the distribution of the moving image, it is determined in step S21 whether or not the first operation input has been made. For example, when the first operation icon 48a is selected in the supporter computer 40, it is determined that the first operation input has been made. When the first operation input is made, the display process of the first additional information proceeds to step S22. In step S22, a message 101, which is an example of the first additional information, is displayed on the moving image displayed on the display 39, and the display process of the first additional information ends. At this time, the message 101 is not displayed on the client device.

Next, with reference to FIG. 16, the display processing of the second additional information performed during the distribution of the moving image will be described. During the distribution of the moving image, it is determined in step S31 whether or not the second operation input has been made. For example, when the second operation icon 48b is selected in the supporter computer 40, it is determined that the second operation input has been made. When the second operation input is made, the display process of the second additional information proceeds to step S32. In step S32, the interrupt image 111, which is an example of the second additional information, is displayed on the client device and the display 39, respectively, and the display process of the second additional information ends.

Next, with reference to FIG. 17, a third additional information display process performed during the distribution of the moving image will be described. During the distribution of the moving image, it is determined in step S41 whether or not the third operation input has been made. For example, when the third operation icon 48c is selected on the supporter computer 40, it is determined that the third operation input has been made. When the third operation input is made, the display process of the third additional information proceeds to step S42. In step S42, the fitted image 122b, which is an example of the third additional information, is displayed on the face portion of the character object, and the display process of the third additional information ends.

Next, with reference to FIG. 18, a blind object display process performed during the distribution of the moving image will be described. During the distribution of the moving image, it is determined in step S51 whether or not the decoration object to be attached to the character object appearing in the moving image has been selected. As described above, the decoration object attached to the character object can be selected from the decoration object candidates included in the candidate list 23d. The decoration object selection process is executed by, for example, the decoration object selection unit 21g described above.

If it is determined in step S51 that the decorative object is selected, the process proceeds to step S52. In step S52, it is determined whether or not the fourth operation input has been made. For example, when the fourth operation icon 48d is selected in the supporter computer 40, it is determined that the fourth operation input has been made. When the fourth operation input is made, the blind object display process proceeds to step S53. In another embodiment, it may be determined that the fourth operation input has been made when the decoration object is selected from the candidate list 23d. In this case, step S52 is omitted, and the blind object display process proceeds from step S51 to step S53.

In step S53, the blind object 131 is displayed so as to hide at least a part of the character object. For example, when the decorative object selected in step S51 is a T-shirt, a blind object 131 is added to the moving image so as to hide the body of the character object associated with the T-shirt.

Next, with reference to FIG. 19, a distribution prohibition process for prohibiting the distribution of the moving image to a specific user will be described. During the distribution of the moving image, it is determined in step S61 whether or not the fifth operation input has been made. For example, when the fifth operation icon 48e is selected in the supporter computer 40, it is determined that the fifth operation input has been made.

When the fifth operation input is made, the distribution prohibition process proceeds to step S62. In step S62, a user (delivery prohibited user) who prohibits the distribution of the moving image is designated. For example, a list of users who are watching the moving image is displayed on the display 44 of the supporter computer 40, and a user selected by the supporter from this list is designated as a distribution prohibited user. The process for designating the distribution prohibited user is executed, for example, by the distribution management unit 41b described above.

Next, the distribution prohibition process proceeds to step S63. In step S63, the video is distributed only to the users included in the video distribution destination list that are not designated as the distribution prohibited users. This moving image distribution process is executed by, for example, the above-mentioned moving image distribution unit 21e.

According to the video distribution system 1 in the above embodiment, it becomes easy to deal with an unexpected situation that occurs during live distribution. For example, by displaying the message 101 on the display 39 without displaying it on the client device, the staff on the distribution side who can see the actor or other display 39 without affecting the moving image viewed by the viewing user. It is possible to convey instructions and communication matters related to the video being distributed to the client.

According to another embodiment, when an unexpected situation occurs during the distribution of the video, the interrupt image 111 for emergency display is distributed to the client device and the display 39 instead of continuing to distribute the normal video as it is. be able to.

According to another embodiment, even when the facial expression of the character object cannot follow the facial expression of the actor in a timely manner, a modified character object incorporating the inset image 122b prepared in advance is generated, and the modified character is generated. By distributing the moving image including the object, it is possible to prevent the deterioration of the quality of the moving image due to the facial expression of the character object not being able to follow the facial expression of the actor.

According to another embodiment, even if some viewing users take inappropriate actions such as posting an inappropriate message, the distribution of the moving image to the viewing user can be stopped.

The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist thereof. For example, the delivered moving image may be shot and generated in a place other than the studio room R. For example, the shooting for generating the delivered video may be performed at the home of the actor or supporter.

In the processing procedure described in the present specification, particularly the processing procedure described using the flow chart, a part of the steps (steps) constituting the processing procedure is omitted, as a step constituting the processing procedure. It is possible to add steps that are not specified and / or to change the order of the steps, and the processing procedure in which such omissions, additions, and changes in order are made does not deviate from the gist of the present invention. It is included in the scope of the invention.

The inventions described in the claims at the time of filing the original application of the present application are added below.
[1]
A distribution server device configured to live stream a video containing an animation of a character object generated based on the movement of an actor to a first client device used by the first user.
A first display device arranged at a position visible to the actor and configured to display the moving image, and a first display device.
A supporter configured to display the second additional information on the moving image displayed on the first display device and the moving image displayed on the first client device based on the second operation input from the actor. With a computer
Video distribution system equipped with.
[2]
The supporter computer is configured to generate a modified character object in which at least a part of the character object is modified based on a third operation input, and include the modified character object in the moving image.
The video distribution system described in [1].
[3]
The character object has a face portion generated to move based on face motion data indicating the movement of the face of the actor.
The distribution server device is configured to display a third additional information on the face portion based on the third operation input.
The video distribution system described in [2].
[4]
It also has storage for storing decorative objects that are displayed in association with the character object.
The supporter computer is configured to display a blind object for hiding at least a part of the character object when displaying the decorative object in the moving image based on the fourth operation input.
The video distribution system according to any one of [1] to [3].
[5]
The distribution server device is configured to live-distribute the moving image to a second client device used by a second user in addition to the first client device.
The first client device is a first type device.
The second client device is a second type device different from the first type device, and is a device of the second type.
The supporter computer includes a display image of the moving image displayed on the first client device and a second display device displaying the displayed image of the moving image displayed on the second client device.
The video distribution system according to any one of [1] to [4].
[6]
The supporter computer is configured to prohibit the distribution of the moving image to the first user based on the fifth operation input.
The video distribution system according to any one of [1] to [5].
[7]
A video distribution method executed by one or more computer processors executing computer-readable instructions.
A process of live-streaming a video including an animation of a character object generated based on the movement of an actor to a first client device used by the first user, and
A step of displaying the moving image on the first display device arranged at a position visible from the actor, and
A step of displaying the second additional information on the moving image displayed on the first display device and the moving image displayed on the first client device based on the second operation input from the actor.
Video distribution method with.
[8]
For one or more computer processors
A process of live-streaming a video including an animation of a character object generated based on the movement of an actor to a first client device used by the first user, and
A step of displaying the moving image on the first display device arranged at a position visible from the actor, and
A step of displaying the second additional information on the moving image displayed on the first display device and the moving image displayed on the first client device based on the second operation input from the actor.
Delivery program to execute.

1 Video distribution system

Claims (11)

A distribution server device configured to live stream a video containing an animation of a character object generated based on the movement of an actor to a first client device used by the first user.
A supporter computer configured to prohibit posting of a message to the video to the first user based on the fifth operation input, and
Video distribution system equipped with. The supporter computer displays a distribution destination list indicating a user who is watching the moving image in response to the fifth operation input.
When the first user is selected from the delivery destination list, posting of a message to the video to the first user is prohibited.
The video distribution system according to claim 1. The first user can watch the video after the posting of a message to the video is prohibited.
The video distribution system according to claim 1 or 2. The supporter computer displays a distribution destination list indicating a user who is watching the moving image in response to the fifth operation input.
When the first user is selected from the distribution destination list, the distribution of the moving image to the first user is stopped.
The video distribution system according to claim 1 or 2. The supporter computer is configured to generate a modified character object in which at least a part of the character object is modified based on a third operation input, and include the modified character object in the moving image.
The video distribution system according to any one of claims 1 to 4. The character object has a face portion generated to move based on face motion data indicating the movement of the face of the actor.
The distribution server device is configured to display a third additional information on the face portion based on the third operation input.
The video distribution system according to claim 5. It also has storage for storing decorative objects that are displayed in association with the character object.
The supporter computer is configured to display a blind object for hiding at least a part of the character object when displaying the decorative object in the moving image based on the fourth operation input.
The video distribution system according to any one of claims 1 to 6. The distribution server device is configured to live-distribute the moving image to a second client device used by a second user in addition to the first client device.
The first client device is a first type device.
The second client device is a second type device different from the first type device, and is a device of the second type.
The supporter computer includes a display image of the moving image displayed on the first client device and a second display device displaying the displayed image of the moving image displayed on the second client device.
The video distribution system according to any one of claims 1 to 7. The supporter computer is configured to prohibit the distribution of the moving image to the first user based on the fifth operation input.
The video distribution system according to any one of claims 1 to 8. A video distribution method executed by one or more computer processors executing computer-readable instructions.
A process of live-streaming a video including an animation of a character object generated based on the movement of an actor to a first client device used by the first user, and
A process of prohibiting the posting of a message to the video to the first user based on the fifth operation input to the supporter computer, and
Video distribution method with. For one or more computer processors
A process of live-streaming a video including an animation of a character object generated based on the movement of an actor to a first client device used by the first user, and
A process of prohibiting the posting of a message to the video to the first user based on the fifth operation input to the supporter computer, and
Delivery program to execute.

Images