JP6548802B1 - A video distribution system that delivers live video including animation of character objects generated based on the movement of actors - Google Patents

Abstract

The present invention provides a moving image distribution system that makes it easy to cope with an unexpected situation that occurs during live distribution. A moving image distribution system is configured to distribute a moving image including animation of a character object generated based on an actor's movement to a first display device and a first client device used by a first user. A distribution server device and a supporter computer configured to display the first additional information on the moving image displayed on the first display device while not displaying the first additional information on the first client device. And comprising. [Selection] Figure 1

Description

  The disclosure in this specification relates to a video distribution system for live distribution of a video including animation of a character object generated based on the movement of an actor.

  There is known a video distribution system which generates animation of a character object based on the movement of an actor and distributes live a video including the animation of the character object. Such a moving image delivery system is disclosed, for example, in Japanese Patent Application Laid-Open No. 2015-184689 (Patent Document 1).

JP, 2015-184689, A

  During live distribution of moving images, unexpected situations such as a failure of a device used to generate animation may occur. In the above-described conventional moving image distribution system, live distribution is performed only by the actor and the operator of the camera. If the actor or the camera operator copes with an unexpected situation, the movement for coping may be reflected in the movement of the character object, or the camera work may become inappropriate. As a result, the quality of the distributed video may be adversely affected.

  The purpose of the present disclosure is to provide technical improvements that solve or alleviate at least some of the problems of the prior art described above. One of the more specific objects of the present disclosure is to provide a video distribution system that facilitates coping with contingencies that occur during live distribution.

  According to an aspect of the present invention, there is provided an animation distribution system according to an aspect, wherein an animation including a character object animation generated based on a movement of an actor is live on a first display device and a first client device used by a first user. A distribution server device configured to distribute, and not displaying the first additional information on the first client device while displaying the first additional information on the moving image displayed on the first display device And a supporter computer configured in The first display may be used by the actor. The first display may display an object selectable by the actor.

  In one aspect, the supporter computer displays second additional information on the moving image displayed on the first display device and the moving image displayed on the first client device based on a second operation input. Configured

  In one aspect, the supporter computer is configured to generate a corrected character object in which at least a part of the character object is corrected based on a third operation input, and to include the corrected character object in the moving image.

  In one aspect, the character object has a face unit that is generated to move based on face motion data that indicates the movement of the face of the actor, and the distribution server device generates the third operation input based on the third operation input. The third additional information is displayed on the face unit.

  In one aspect, the computer system further includes a storage for storing a decoration object displayed in association with the character object, and the supporter computer is configured to display the decoration object in the moving image based on a fourth operation input. It is configured to display a blind object for hiding at least a part.

  In one aspect, the distribution server device is configured to live distribute the moving image to a second client device used by a second user in addition to the first client device, the first client device being of a first type The second client device is a second type device different from the first type device, and the supporter computer is a display image of the moving image displayed on the first client device, and A second display device is provided that displays a display image of the moving image displayed on the second client device.

  In one aspect, the supporter computer is configured to prohibit distribution of the moving image to the first user based on a fifth operation input.

  One aspect relates to a video delivery method performed by one or more computer processors executing computer readable instructions. The moving image distribution method comprises the steps of live distributing a moving image including animation of a character object generated based on the movement of an actor to a first display device and a first client device used by a first user, and the first display device And displaying the first additional information on the moving image displayed on the screen, while not displaying the first additional information on the first client device.

  A distribution program according to one aspect performs live distribution to one or more computer processors on a first display device and a first client device used by a first user, a moving image including animation of a character object generated based on the motion of an actor. And causing the first client device to display no first additional information while displaying the first additional information on the moving image being displayed on the first display device.

  According to an embodiment of the present invention, it is possible to provide a video distribution system that facilitates handling of unexpected situations that occur during live distribution.

FIG. 1 is a block diagram illustrating a video distribution system according to one embodiment. It is a schematic diagram which shows typically the installation of the studio which produces the moving image delivered by the moving image delivery system of FIG. It is a figure explaining the possession list | wrist memorize | stored in the moving image delivery system of FIG. It is a figure explaining the candidate list memorize | stored in the moving image delivery system of FIG. It is a figure which shows the example of the moving image displayed on the client apparatus 10a in one Embodiment. FIG. 5 includes animation of a character object. It is a figure which shows the example of the moving image displayed on the client apparatus 10a in one Embodiment. A normal object is included in FIG. It is a figure which shows the example of the moving image displayed on the client apparatus 10a in one Embodiment. FIG. 7 includes a decoration object. It is a schematic diagram which shows the example of the decoration object selection screen for selecting a desired selection decoration object out of decoration objects contained in a candidate list | wrist. It is a figure which shows the example of the image displayed on the display 44 of the supporter computer 40 in one Embodiment. It is a figure which shows the example of the moving image displayed on the display 39 in one Embodiment. An example of the first additional information is shown in FIG. It is a figure which shows the example of the moving image displayed on the client apparatus 10a in one Embodiment. The second additional information is included in FIG. 11a. It is a figure which shows the example of the moving image displayed on the client apparatus 10a in one Embodiment. FIG. 11 b includes second additional information. It is a figure which shows the example of the moving image displayed on the client apparatus 10a in one Embodiment. FIG. 12a contains third additional information. It is a figure which shows the example of the moving image displayed on the client apparatus 10a in one Embodiment. FIG. 12 b includes third additional information. It is a figure which shows the example of the moving image displayed on the client apparatus 10a in one Embodiment. FIG. 13 includes blind objects. It is a flow figure showing the flow of the animation distribution processing in one embodiment. It is a flow figure showing the flow of processing which displays the 1st additional information in one embodiment. It is a flow figure showing the flow of processing which displays the 2nd additional information in one embodiment. It is a flow figure showing the flow of processing which displays the 3rd addition information in one embodiment. It is a flow figure showing the flow of processing which displays a blind object in one embodiment. It is a flow figure showing the flow of processing which forbids animation distribution to a delivery prohibition user in one embodiment.

  Hereinafter, various embodiments of the present invention will be described with reference to the drawings as appropriate. The same reference symbols are attached to the same or similar components in multiple drawings.

  A video distribution system according to an embodiment will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram showing a moving picture distribution system 1 according to an embodiment, and FIG. 2 is a schematic view schematically showing a studio facility where production of a moving picture distributed by the moving picture distribution system 1 is performed. 3 and 4 are diagrams for explaining information stored in the moving picture distribution system 1.

  The video distribution system 1 includes client devices 10 a to 10 c, a server device 20, a studio unit 30, and a storage 60. The client devices 10a to 10c, the server device 20, and the storage 60 are communicably connected to each other via the network 50. The server device 20 is configured to deliver a moving image including an animation of a character, as described later. The characters included in this moving image may be motion controlled in the virtual space.

  This moving image can be distributed from the server device 20 to each of the client devices 10a to 10c. The first viewing user who is the user of the client device 10a, the second viewing user who is the user of the client device 10b, and the third viewing user who is the user of the client device 10c view the distributed moving image by each client device be able to. The video distribution system 1 may include less than three client devices, and may include more than three client devices. In the present specification, when it is not necessary to distinguish the client devices 10a to 10c and other client devices from each other, these may be generically referred to as "client devices".

  The client devices 10a to 10c are information processing devices such as smartphones. The client devices 10a to 10c may be mobile phones, tablet terminals, personal computers, electronic book readers, wearable computers, game consoles, and various information processing devices capable of reproducing moving images other than these, in addition to smartphones. Each of the client devices 10a to 10c includes a computer processor, a memory, a communication I / F, a display, a sensor unit having various sensors such as a gyro sensor, a sound collecting device such as a microphone, and a storage for storing various information. It is also good.

  The viewing user can input a message regarding the distributed moving image via the input interface of the client devices 10a to 10c, and post the message to the server device 20. The message posted from each viewing user may be displayed superimposed on the moving image. Thereby, exchange between viewing users is achieved.

  In the illustrated embodiment, the server device 20 includes a computer processor 21, a communication I / F 22, and a storage 23.

The computer processor 21 is an arithmetic device that loads an operating system and various programs for realizing various functions from the storage 23 or other storages into the memory and executes instructions included in the loaded programs. The computer processor 21 is, for example, a CPU, an MPU, a DSP, a GPU, various arithmetic devices other than these, or a combination thereof. The computer processor 21 may be realized by an integrated circuit such as an ASIC, a PLD, an FPGA, or an MCU. Although computer processor 21 is illustrated as a single component in FIG. 1, computer processor 21 may be a collection of multiple physically separate computer processors. In the present specification, a program described as being executed by the computer processor 21 or an instruction included in the program may be executed by a single computer processor or may be executed by a plurality of computer processors in a distributed manner. It is also good. Also, a program executed by the computer processor 21 or an instruction included in the program may be executed by a plurality of virtual computer processors.

  The communication I / F 22 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination thereof. The server device 20 can transmit and receive data with other devices via the communication I / F 22.

  The storage 23 is a storage device accessed by the computer processor 21. The storage 23 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various other storage devices capable of storing data. Various programs can be stored in the storage 23. The programs that may be stored in the storage 23 and at least a portion of various data may be stored in a storage (for example, the storage 60) that is physically separate from the server device 20.

  Many of the components of the studio unit 30 are arranged, for example, in the studio room R shown in FIG. As illustrated, in the studio room R, an actor A1 and an actor A2 perform a performance. The studio unit 30 is configured to detect movements and expressions of the actor A1 and the actor A2, and to output the detected information to the server device 20.

  Both the actor A1 and the actor A2 are objects whose movements and expressions are captured by a sensor group described later provided in the studio unit 30. The actor A1 and the actor A2 are, for example, humans, animals or moving objects that perform a performance. The actor A1 and the actor A2 may be, for example, robots capable of autonomous movement. The number of actors in the studio room R may be one or three or more.

  The studio unit 30 includes six motion sensors 31a to 31f attached to the actor A1, a controller 33a held by the left hand of the actor A1, a controller 33b held by the right hand of the actor A1, and a mounting tool 37b. And a camera 37a attached to the head of the actor A1. The studio unit 30 also includes six motion sensors 32a to 32f attached to the actor A2, a controller 34a held by the left hand of the actor A2, a controller 34b held by the right hand of the actor A2, and an attachment 38b. And a camera 38a attached to the head of the actor A2. A microphone for acquiring audio data may be attached to each of the attachment 37 b and the attachment 38 b. This microphone can acquire the speech of the actor A1 and the actor A2 as voice data. The microphone may be a wearable microphone attached to the actor A1 and the actor A2 via the attachment 37b and the attachment 38b, and may be installed on the floor, wall or ceiling of the studio room R. It may be The studio unit 30 includes, in addition to the components described above, a base station 35a, a base station 35b, a tracking sensor 36a, a tracking sensor 36b, and a display 39. A supporter computer 40 is installed in the studio room R and the next room separated from the glass window. The server device 20 may be installed in the same room as the room in which the supporter computer 40 is installed.

  The motion sensors 31a to 31f and the motion sensors 32a to 32f cooperate with the base station 35a and the base station 35b to detect their position and orientation. In one embodiment, base station 35a and base station 35b are multi-axis laser emitters. The base station 35a scans laser light around, for example, the vertical axis after emitting synchronization light. The base station 35 b scans laser light around, for example, a horizontal axis. Each of the motion sensors 31a to 31f and the motion sensors 32a to 32f may include a plurality of optical sensors that detect the incidence of the blinking light and the laser light from the base station 35a and the base station 35b. The motion sensors 31a to 31f and the motion sensors 32a to 32f respectively have a time difference between the incident timing of the blinking light and the incident timing of the laser light, the light reception time of each optical sensor, the incident angle of the laser light detected by each optical sensor, Other information may be detected as necessary. The motion sensors 31a to 31f and the motion sensors 32a to 32f may be, for example, Vive Trackers provided by HTC CORPORATION. The base station 35a and the base station 35b may be, for example, base stations provided by HTC CORPORATION. Three or more base stations may be provided. The position of the base station may be changed as appropriate. For example, in addition to or instead of the base station disposed in the upper corner of the space to be detected of the tracking sensor shown in FIG. 2, the upper position and the lower position close to the floor are paired. A base station arranged may be provided.

  Detection information (hereinafter sometimes referred to as “tracking information”) detected in each of the motion sensors 31 a to 31 f and the motion sensors 32 a to 32 f is transmitted to the server device 20. The detection information may be wirelessly transmitted to the server device 20 from each of the motion sensors 31a to 31f and the motion sensors 32a to 32f. Since the base station 35a and the base station 35b emit flashing light and scan laser light at regular intervals, the tracking information of each motion sensor is updated at each interval. The position and orientation of each of the motion sensors 31a to 31f and the motion sensors 32a to 32f can be calculated based on the tracking information detected by the motion sensors 31a to 31f and the motion sensors 32a to 32f. The position and orientation of each of the motion sensors 31a to 31f and the motion sensors 32a to 32f may be calculated based on the tracking information in the server device 20, for example.

  In the illustrated embodiment, the actor A1 mounts six motion sensors 31a to 31f. The motion sensors 31a, 31b, 31c, 31d, 31e, and 31f are attached to the left wrist, the right wrist, the left instep, the right instep, the hip, and the top of the actor A1, respectively. The motion sensors 31a to 31f may be attached to the actor A1 via an attachment. The actor A2 is equipped with six motion sensors 32a to 32f. The motion sensors 32a to 32f may be attached to the actor A2 at the same position as the motion sensors 31a to 31f. Motion sensors 31 a to 31 f and motion sensors 32 a to 32 f shown in FIG. 2 are examples. The motion sensors 31a to 31f may be attached to various parts of the actor A1, and the motion sensors 32a to 32f may be attached to various parts of the actor A2. The number of motion sensors attached to the actor A1 and the actor A2 may be five or less or seven or more. As described above, the body motion of the actor A1 and the actor A2 is detected by detecting the position and the orientation of the motion sensors 31a to 31f and the motion sensors 32a to 32f attached to each part of the body of the actor A1 and the actor A2. be able to.

  In one embodiment, a plurality of infrared LEDs are mounted on each of the plurality of motion sensors attached to the actor A1 and the actor A2, and the light from the infrared LEDs is provided on the floor or wall of the studio room R. The position and orientation of each of the motion sensors may be detected by detecting with an infrared camera. A visible light LED may be used in place of the infrared LED, and the position and orientation of each of the motion sensors may be detected by detecting light from the visible light LED with a visible light camera. As described above, a light emitting unit (for example, an infrared LED or a visible light LED) is provided for each of the plurality of motion sensors attached to the actor, and light from the light emitting unit is provided in the studio room R For example, the position and orientation of each of the motion sensors may be detected by detecting it with an infrared camera or a visible light camera.

  In one embodiment, a plurality of reflection markers can be used instead of the motion sensors 31a to 31f and the motion sensors 32a to 32f. The reflection marker is attached to each of the actor A1 and the actor A2 with an adhesive tape or the like. The position and the orientation of the reflection marker can be detected by photographing the actor A1 and the actor A2 to which the reflection marker is attached as described above to generate photographing data and performing image processing on the photographing data.

  The controller 33a and the controller 33b output control signals to the server device 20 according to the operation of the actor A1. Similarly, the controller 34 a and the controller 34 b output control signals to the server device 20 according to the operation of the actor A 2.

  The tracking sensor 36 a and the tracking sensor 36 b generate tracking information for defining setting information of a virtual camera for constructing a virtual space included in the moving image. Tracking information of the tracking sensor 36 a and the tracking sensor 36 b is calculated as the position in each three-dimensional orthogonal coordinate system and the angle around each axis. The position and orientation of the tracking sensor 36a may be changed according to the operation of the operator. The tracking sensor 36 a transmits tracking information indicating the position and the orientation to the tracking information server device 20. Similarly, the position and orientation of the tracking sensor 36b are set according to the operation of the operator. The tracking sensor 36 b transmits tracking information indicating the position and the orientation to the tracking information server device 20. The tracking sensors 36a, 36b may be supported by gimbals or stabilizers. The gimbal may be configured to be graspable by an actor or a supporter. By supporting the tracking sensors 36a and 36b with gimbals or stabilizers, it is possible to suppress camera shake.

  The camera 37a is attached to the head of the actor A1, as described above. For example, the camera 37a is arranged to be able to capture the face of the actor A1. The camera 37a continuously captures the face of the actor A1 and acquires imaging data of the face of the actor A1. Similarly, the camera 38a is attached to the head of the actor A2. The camera 38a is arranged so as to be able to capture the face of the actor A2, and continuously captures the face of the actor A2 and acquires imaging data of the face of the actor A2. The camera 37a transmits imaging data of the face of the actor A1 to the server device 20, and the camera 38a transmits imaging data of the face of the actor A1 to the server device 20. The camera 37a and the camera 38a may be a 3D camera capable of detecting the depth of the face of a person.

  The display 39 is configured to display the information received from the supporter computer 40. The information transmitted from the supporter computer 40 to the display 39 may include, for example, text information, image information, and various other information. The display 39 is disposed at a position visible by the actor A1 and the actor A2. The display 39 is an example of a first display device.

  In the illustrated embodiment, the supporter computer 40 is installed in the next room of the studio room R. Since the room in which the supporter computer 40 is installed and the studio room R are separated by the glass window, the operator of the supporter computer 40 (sometimes referred to as "supporter" in this specification) is the actor A1 and the actor A2. You can see In the illustrated embodiment, a supporter B1 and a supporter B2 are present as operators of the supporter computer 40. In the present specification, the supporter B1 and the supporter B2 may be collectively referred to as the supporter B1 and the supporter B2 "supporter" when it is not necessary to distinguish them from each other.

  The supporter computer 40 may be configured to be able to change the setting of the component of the studio unit 30 according to the operation of the supporter B1 and the supporter B2. The supporter computer 40 can change, for example, the setting of the scanning interval by the base station 35a and the base station 35b, the setting of the position or the orientation of the tracking sensor 36a and the tracking sensor 36b, and various settings of other various devices. . At least one of the supporter B1 and the supporter B2 can input a message to the supporter computer 40, and can display the input message on the display 39.

  The components and functions of studio unit 30 shown in FIG. 2 are exemplary. The studio unit 30 applicable to the present invention may comprise various components not shown. For example, studio unit 30 may include a projector. The projector can project a moving image distributed to the client device 10 a or another client device on the screen S.

  Next, in one aspect, the information stored in the storage 23 will be described. In the illustrated embodiment, the storage 23 stores the model data 23a, the object data 23b, the possession list 23c, the candidate list 23d, and various information necessary for generating and distributing the distribution moving picture other than the above.

  The model data 23a is model data for generating animation of a character. The model data 23a may be three-dimensional model data for generating three-dimensional animation, or may be two-dimensional model data for generating two-dimensional model data. The model data 23a includes, for example, rig data (sometimes called "skeleton data") indicating the skeleton of the character and surface data indicating the shape and texture of the surface of the character. The model data 23a can include a plurality of different model data. The plurality of model data may have different rig data, or may have the same rig data. The plurality of model data may have surface data different from each other or may have the same surface data. In the illustrated embodiment, in order to generate a character object corresponding to the actor A1 and a character object corresponding to the actor A2, the model data 23a has at least two types of model data different from each other. The model data for the character object corresponding to the actor A1 and the model data for the character object corresponding to the actor A2 may have, for example, the same rig data but different surface data.

  The object data 23 b includes asset data for constructing a virtual space constituting a moving image. The object data 23 b is data for drawing the background of the virtual space constituting the moving image, data for drawing various objects displayed in the moving image, and for drawing various objects displayed in the moving image other than these. Contains data. The object data 23a may include object position information indicating the position of the object in the virtual space.

  In addition to the above, the object data 23b may include a gift object displayed in a moving image based on a display request from viewing users of the client devices 10a to 10c. The gift objects may include effect objects, regular objects, and decoration objects. The viewing user can purchase a desired gift object. An upper limit may be set on the number of purchasable objects or the purchase price of the objects.

  The effect object is an object that affects the impression of the entire viewing screen of the distribution moving image, and is, for example, an object that simulates confetti. An object simulating a confetti may be displayed on the entire viewing screen, which can change the impression of the entire viewing screen before and after the display. The effect object may be displayed so as to overlap with the character object, but differs from the decorative object in that the display associated with a specific part of the character object is not made.

  The normal object is an object to be a digital gift from the viewing user to the actor (for example, the actor A1 or the actor A2), and is, for example, an object resembling a stuffed toy or a bouquet. In one aspect, the normal object is displayed on the display screen of the moving image so as not to be in contact with the character object. In one aspect, the normal object is displayed on the display screen of the moving image so as not to overlap with the character object. The normal object may be displayed in the virtual space so as to overlap with an object other than the character object. The normal object may be displayed so as to overlap with the character object, but differs from the decorative object in that the display associated with a specific part of the character object is not made. In one aspect, when the normal object is displayed overlapping with the character object, the normal object overlaps a portion other than the head including the face of the character object and does not overlap the head of the character object. It will be displayed as

  The decoration object is an object displayed on the display screen in association with a specific part of the character object. In one aspect, the decorative object displayed on the display screen in association with the specific part of the character object is displayed on the display screen so as to touch the specific part of the character object. In one aspect, the decorative object displayed on the display screen in association with the specific part of the character object is displayed on the display screen so as to cover a part or all of the specific part of the character object.

  The decorative object is, for example, an accessory (such as a headband, a necklace, an earring, etc.) attached to a character object, clothes (such as a T-shirt), a costume, and an object that can be attached to other character objects. The object data 23b corresponding to the decoration object may include mounting position information indicating which part of the character object the decoration object is associated with. The mounting position information of a certain decoration object can indicate to which part of the character object the decoration object is to be mounted. For example, when the decorative object is a headband, the mounting position information of the decorative object may indicate that the decorative object is mounted on the "head" of the character object. If the decorative object is a T-shirt, the mounting position information of the decorative object may indicate that the decorative object is mounted on the “body” of the character object.

  The display time according to the kind may be set to each of the gift objects. In one aspect, the display time of the decoration object may be set longer than the display time of the effect object and the display time of the normal object. For example, the display time of the decoration object may be set to 60 seconds, the display time of the effect object may be set to 5 seconds, and the display time of the normal object may be set to 10 seconds.

  The possession list 23c is a list showing gift objects possessed by the viewing user of the moving image. An example of the possession list 23c is shown in FIG. As illustrated, in the possession list 23c, an object ID for specifying a gift object possessed by the viewing user is stored in association with account information of the viewing user (for example, a user ID of the viewing user). The viewing users include, for example, first to third viewing users of the client devices 10a to 10c.

  The candidate list 23d is a list of decoration objects for which a display request has been made from the viewing user. As described later, the viewing user who holds the decoration object can make a display request of the holding decoration object. In the candidate list 23d, an object ID for specifying the decoration object is stored in association with the account information of the viewing user who has made the display request for the decoration object. The candidate list 23d may be created for each distributor. The candidate list 23d may be stored, for example, in association with distributor identification information that specifies a distributor (the actor A1, the actor A2, the supporter B1, and / or the supporter B2).

  Next, functions implemented by the computer processor 21 will be described more specifically. The computer processor 21 executes a computer-readable instruction included in the distribution program to obtain a body motion data generation unit 21a, a face motion data generation unit 21b, an animation generation unit 21c, a moving image generation unit 21d, a moving image distribution unit 21e, It functions as a display request processing unit 21f, a decoration object selection unit 21g, and an object purchase processing unit 21h. At least a part of the functions implemented by the computer processor 21 may be implemented by a computer processor other than the computer processor 21 of the video distribution system 1. At least a part of the functions realized by the computer processor 21 may be realized, for example, by a computer processor mounted on the supporter computer 40.

  The body motion data generation unit 21a generates first body motion data of each part of the body of the actor A1 based on the tracking information detected by each of the motion sensors 31a to 31f, and the motion sensors 32a to 32f. Second body motion data, which is a digital representation of the position and orientation of each part of the body of the actor A2, is generated based on the tracking information detected by each. In the present specification, the first body motion data and the second body motion data may be collectively referred to simply as "body motion data". Body motion data is generated as needed over time. For example, body motion data may be generated at predetermined sampling time intervals. Thus, the body motion data can represent the movement of the body of the actor A1 and the actor A2 in time series as digital data. In the illustrated embodiment, the motion sensors 31a to 31f and the motion sensors 32a to 32f are attached to the left and right limbs, the waist, and the head of the actor A1 and the actor A2, respectively. Based on the tracking information detected by the motion sensors 32a to 32f, it is possible to digitally represent generally the positions and orientations of the whole body of the actor A1 and the actor A2 over time. The body motion data can define, for example, the position and rotation angle of bones constituting the rig data included in the model data 23a.

  The face motion data generation unit 21b generates first face motion data, which is a digital representation of the movement of the face of the actor A1, based on the imaging data of the camera 37a, and based on the imaging data of each of the cameras 38a, Second face motion data, which is a digital representation of the facial movement of actor A2, is generated. In the present specification, the first face motion data and the second face motion data may be collectively referred to simply as “face motion data”. Face motion data is generated as needed over time. For example, face motion data may be generated at predetermined sampling time intervals. Thus, the face motion data can digitally represent the movement (change in expression) of the face of the actor A2 and the actor A2 in time series.

  The animation generation unit 21c applies the body motion data generated by the body motion data generation unit 21a and the face motion data generated by the face motion data generation unit 21b to predetermined model data included in the model data 23a. It is configured to generate animation of a character object that moves in a virtual space and whose expression changes. Specifically, the animation generation unit 21c generates an animation of a character object moving in synchronization with the movement of the body and expression of the actor A1, based on the first body motion data and the first face motion data regarding the actor A1. Based on the second body motion data and the second face motion data related to the actor A2, an animation of a character object moving in synchronization with the movement of the body and expression of the actor A2 can be generated. In this specification, a character object generated based on the movement and expression of the actor A1 is referred to as a "first character object", and a character object generated based on the movement and expression of the actor A2 is referred to as a "second character object". There is a thing called ".

  The moving image generation unit 21d constructs a virtual space using the object data 23b, and this virtual space, the animation of the first character object corresponding to the actor A1, and the animation of the second character object corresponding to the actor A2 Generate a movie that contains The first character object is disposed in the virtual space so as to match the position of the actor A1 with respect to the tracking sensor 36a, and the second character object is disposed in the virtual space so as to coincide with the position of the actor A2 with respect to the tracking sensor 36a. Be done. Therefore, the position and the orientation of the first character object and the second character object in the virtual space can be changed by changing the position or the orientation of the tracking sensor 36a.

  In one aspect, the moving image generation unit 21d constructs a virtual space based on the tracking information of the tracking sensor 36a. For example, the moving image generation unit 21d determines setting information (a position in the virtual space, a gaze position, a gaze direction, and an angle of view) of the virtual camera based on the tracking information of the tracking sensor 36a, and based on the setting information of the virtual camera Then, a drawing area of the entire virtual space is determined, and moving image information for displaying the drawing area of the virtual space is generated.

  The moving image generation unit 21 d is configured to use the first character object and the second character in the virtual space based on the tracking information of the tracking sensor 36 b instead of the tracking information of the tracking sensor 36 a or in addition to the tracking information of the tracking sensor 36 a. The position and orientation of the object, and setting information of the virtual camera may be determined.

  The moving image generation unit 21d can synthesize the voice of the actor A1 and the actor A2 acquired from the microphone of the studio unit 30 with the generated moving image.

  As described above, the moving image generation unit 21d performs the animation of the first character object moving in synchronization with the movement of the body and expression of the actor A1, and the second character moving in synchronization with the movement of the body and expression of the actor A2. An animation of an object can be generated, and an animation for delivery, in which the voices of the actor A1 and the actor A2 are synthesized, can be generated.

  The moving image distribution unit 21e distributes the moving image generated by the moving image generation unit 21d. This moving image is distributed to the client devices 10a to 10c and other client devices via the network 50. The moving image distribution unit 21e refers to the list of users who have requested distribution of the moving image (distribution destination list), and distributes the moving image to the client device of the user included in the list. As will be described later, there are cases where the distribution from the specific moving image to the specific user is prohibited by the instruction from the supporter computer 40. In this case, the moving image is distributed to users other than the user whose distribution of the moving image is prohibited. The received moving image is reproduced by the client device 10a to the client device 10c.

  This moving image may be distributed to a client device (not shown) installed in the studio room R, and may be projected onto the screen S from the client device via the short focus projector. Also, this moving image may be distributed to the supporter computer 40. Thereby, supporter B1 and supporter B2 can confirm the viewing screen of the animation distributed.

  A display example of a moving image distributed from the server device 20 to the client device 10a and reproduced by the client device 10a is shown in FIG. As illustrated, the display image 70 of the moving image distributed from the server device 20 is displayed on the display of the client device 10a. The display image 70 displayed on the client device 10a includes, in the virtual space, a character object 71A corresponding to the actor A1, a character object 71B corresponding to the actor A2, and an object 72 indicating a table. There is. The object 72 is not a gift object, but one of objects for constructing a virtual space included in the object data 23 b. The character object 71A is generated by applying the first body motion data and the first face motion data of the actor A1 to the model data for the actor A1 included in the model data 23a. The character object 71A is motion controlled based on the first body motion data and the first face motion data. The character object 71B is generated by applying the second body motion data and the second face motion data of the actor A2 to the model data for the actor A2 included in the model data 23a. The motion control of the character object 71B is performed based on the second body motion data and the second face motion data. Therefore, the character object 71A is controlled to move in the screen in synchronization with the movement of the body and expression of the actor A1, and the character object 71B is moved in the screen in synchronization with the movement of the body and expression of the actor A2. Controlled by

  As described above, the distribution video from the server device 20 may be distributed to the supporter computer 40. The moving image distributed to the supporter computer 40 is displayed on the supporter computer 40 as in FIG. The supporter B1 and the supporter B2 can change the setting of the components of the studio unit 30 by viewing the moving image reproduced by the supporter computer 40. In one aspect, when the supporter B1 and the supporter B2 want to change the angle of the character object 71A and the character object 71B in the moving image being distributed, the supporter computer 40 tracks an instruction signal for instructing the change of the direction of the tracking sensor 36a. It can be sent to the sensor 36a. The tracking sensor 36a can change its direction in accordance with the instruction signal. For example, the tracking sensor 36a is pivotably attached to the stand via a pivoting mechanism having an actuator built-in around its axis, and when the tracking sensor 36a receives an instruction signal instructing to change its direction, The actuator of the rotation mechanism may be driven based on the instruction signal, and the tracking sensor 36a may be rotated by an angle corresponding to the instruction signal. In one aspect, the supporter B1 and the supporter B2 transmit an instruction to use the tracking information from the tracking sensor 36b from the supporter computer 40 to the tracking sensor 36a and the tracking sensor 36b instead of the tracking information from the tracking sensor 36a. It is also good. The tracking sensor 36a and the tracking sensor 36b may be configured and installed so as to be movable by an actor or a supporter. Thereby, the actor or the supporter can hold and move the tracking sensor 36a and the tracking sensor 36b.

  In one embodiment, when the supporter B1 and the supporter B2 determine that the instruction for the actor A1 or the actor A2 is necessary by viewing the moving image reproduced by the supporter computer 40, the supporter computer 40 sends a message indicating the instruction to the supporter computer 40. This message can be input and output to the display 39. For example, the supporter B1 and the supporter B2 can instruct the actor A1 or the actor A2 to change the standing position through the message displayed on the display 39.

  The display request processing unit 21 f receives a display request for a gift object from the client device of the viewing user, and performs processing according to the display request. Each viewing user can transmit a gift object display request to the server device 20 by operating his / her client device. For example, the first viewing user can transmit a display request for a gift object to the server device 20 by operating the client device 10a. The display request for the gift object may include the user ID of the viewing user and identification information (object ID) for specifying an object for which display is requested.

  As mentioned above, gift objects may include effect objects, regular objects, and decorative objects. The effect object and the normal object are examples of the first object. In addition, a display request that requests display of an effect object or a normal object is an example of a second display request.

  In one aspect, when the display request processing unit 21 f receives a display request for a specific effect object from the viewing user, the display request processing unit 21 f causes the display image 70 of the moving image to display the effect object for which display is requested based on the display request. Do the processing. For example, when the display request of the effect object corresponding to the confetti is made, the display request processing unit 21 f displays the effect object 73 indicating the confetti based on the display request as shown in FIG. Display on.

  In one aspect, when the display request processing unit 21 f receives a display request for a specific normal object from the viewing user, the display request processing unit 21 f performs processing for causing the moving image 70 to display the normal object whose display is requested based on the display request. . For example, when a display request for a normal object indicating a stuffed bear is made, the display request processing unit 21 f displays a normal object 74 indicating a stuffed bear on the basis of the display request, as shown in FIG. 6. Display on 70

  The display request for the normal object 74 may include a display position specification parameter for specifying the display position of the normal object 74 in the virtual space. In this case, the display request processing unit 21 f can display the normal object 74 at the position designated by the display position designation parameter in the virtual space. For example, the display position specification parameter can specify the display position of the normal object 74 above the object 72 indicating the table. The viewing user designates the position where the normal object is to be displayed by viewing the display of the character object 71A, the character object 71B, the gift object, and other objects included in the moving image 70 by using the display position specification parameter. can do.

In one aspect, the normal object 74 may be displayed to move within the display image 70 of the moving image. For example, the normal object 74 may be displayed to fall from the top to the bottom of the screen. In this case, the normal object 74 may be displayed in the display image 70 while falling to the floor of the virtual space constituting the moving image 70 from the start of falling, and may disappear from the display image 70 after falling to the floor . The viewing user can view the falling normal object 74 from the drop start time to the drop end time. The moving direction of the normal object 74 in the screen can be set arbitrarily. For example, the normal object 74 may be displayed in the display image 70 so as to move from the left to the right, the right to the left, the upper left to the lower left, or other directions of the moving image 70. The normal object 74 can move on various trajectories. For example, the normal object 74 can move on a linear orbit, a circular orbit, an elliptical orbit, a helical orbit, or any other orbit. The viewing user specifies the movement direction parameter for specifying the movement direction of the normal object 74 and the trajectory for the normal object 74 to move, in addition to or in place of the display position specification parameter in the normal object display request. At least one of the orbital parameters can be included. In one embodiment, an effect object and a normal object whose size in the virtual space is smaller than a reference size (for example, a piece of paper of confetti of the effect object 73) are included in the character object 71A and the character object 71B. Some or all may be displayed overlapping. In one embodiment, an effect object and a normal object whose size in the virtual space is larger than a reference size (for example, a normal object 74 (bear's plush toy)) are displayed at a position not overlapping the character object. . In one embodiment, an effect object and a normal object whose size in the virtual space is larger than a reference size (for example, a normal object 74 (a plush toy)) are at least one of the character object 71A and the character object 71B. When it overlaps with, it is displayed behind the character object displayed in the overlapping.

  In one aspect, when receiving a display request for a specific decoration object from the viewing user, the display request processing unit 21 f adds the decoration object whose display is requested to the candidate list 23 d based on the display request. The display request for the decoration object is an example of the first display request. For example, the display request processing unit 21 f stores identification information (object ID) identifying a specific decoration object requested to be displayed by the viewing user in the candidate list 23 d in association with the user ID of the viewing user. (See Figure 4). When a plurality of display requests for decoration objects are made, for each of the plurality of display requests, the user ID of the viewing user who made the display request, and the decoration object ID of the decoration object requested to be displayed by the viewing user. Are associated with each other and stored in the candidate list 23d.

  In one aspect, the decoration object selection unit 21g displays the selected decoration object on the display image 70 of the moving image in response to one or more of the decoration objects included in the candidate list 23d being selected. Perform processing to In the present specification, a decoration object selected from the candidate list 23 d may be referred to as a “selected decoration object”.

  Selection of the decoration object from the candidate list 23 d is made by, for example, the supporter B 1 and the supporter B 2 operating the supporter computer 40. In one aspect, the supporter computer 40 displays a decoration object selection screen. FIG. 8 shows an example of the decoration object selection screen 80 in one aspect. The decoration object selection screen 80 is displayed on the display of the supporter computer 40, for example. The decoration object selection screen 80 displays, for example, each of the plurality of decoration objects included in the candidate list 23d in a tabular form. As illustrated, the decoration object selection screen 80 according to one aspect includes a first column 81 for displaying the type of decoration object, a second column 82 for displaying an image of the decoration object, and a character object associated with the decoration object. And a third column 83 for displaying the site of Further, on the decoration object selection screen 80, selection buttons 84a to 84c for selecting each decoration object are displayed. Thus, the decoration object selection screen 80 displays decoration objects as candidates for the selected decoration object.

  The supporter B1 and the supporter B2 can select one or more of the decoration objects included in the decoration object selection screen 80. For example, the supporter B1 and the supporter B2 can select the headband by selecting the selection button 84a. When it is detected by the decoration object selection unit 21g that the headgear is selected, the display request processing unit 21f displays the selected decoration object 75 indicating the selected headgear on the display screen 70 of the moving image, as shown in FIG. Let The selected decoration object 75 is displayed in the display image 70 in association with a specific part of the character object. The selected decoration object 75 may be displayed in contact with the specific part of the character object. For example, since the selection 75 indicating the headband is associated with the head of the character object, it is worn on the head of the character object 71A as shown in FIG. The decorative object may be displayed on the display screen 70 so as to move along with the movement of the specific part of the character object. For example, when the head of the character object 71A attached with the headband moves, the selected decoration object 75 indicating the headband is attached to the head of the character object 71A as if the headpiece is attached to the head of the character object 71A. Move.

  The selected decoration object 75 may be displayed on the display screen 70 in association with the character object 71B instead of the character object 71A. The selected decoration object 75 may be displayed on the display screen 70 in association with the character object 71A and the character object 71B.

  In one embodiment, the decoration object selection screen 80 may be configured not to include information that can identify a user who holds each decoration object or a user who has made a display request for each decoration object. This makes it possible to prevent the selector from giving preference to a particular user when selecting a decoration object.

  In one embodiment, the decoration object selection screen 80 may display information regarding the user who holds each decoration object or the user who made a display request for the decoration object for each decoration object. For example, for each decoration object, the number of times the user who has made the display request of the decoration object has made the display request of the decoration object so far and the number of times selected (for example, the display request of the decoration object has been made 5 times) Information indicating that the user has selected at least one of the character object 71A and the character object B, the number of viewings by the user, the moving image (character object 71A and characters in the moving image distribution system 1) The number of times the object B appears or not, the purchase amount of the gift object by the user, the number of purchases of the object by the user, points usable in the moving image distribution system 1 held by the user, moving images of the user system At level, and it may display various information about the user who made the request for display of the ornamental objects other than these. According to the embodiment, it is possible to select the decoration object based on the behavior and the viewing history in the moving image distribution system 1 of the user who has made the display request of the decoration object so far.

  In one embodiment, the display of decorative objects may be constrained to eliminate duplicates. For example, with regard to the character object 71A, if a decoration object associated with the specific part is already selected, selection of other decoration objects associated with the specific part may be prohibited. As shown in the embodiment of FIG. 7, if the headband associated with the "head" of the character object 71B is already selected, the other decorations associated with the "head" are already selected. An object (for example, a decoration object representing a "hat" associated with the head) is not displayed on the decoration object selection screen 80, or a selection button for selecting a decoration object representing the hat is a selection screen 80. May not be selected. According to this embodiment, it is possible to prevent the decoration object from being displayed redundantly on a specific part of the character object.

  The decoration object selection screen 80 may be displayed on another device instead of or in addition to the supporter computer 40. For example, the decoration object selection screen 80 may be displayed on at least one of the display 39 and the screen S in the studio room R. In this case, the actor A1 and the actor A2 can select a desired decoration object based on the decoration object selection screen 80 displayed on the display 39 or the screen S. The selection of the decoration object by the actor A1 and the actor A2 can be made, for example, by operating the controller 33a, the controller 33b, the controller 34a, or the controller 34b.

  In one aspect, the object purchase processing unit 21 h responds to a request from the viewing user of the moving image to purchase information of each of a plurality of gift objects that can be purchased in relation to the moving image as a client device of the viewing user Send to the client device 10a). The purchase information of each gift object includes the type of the gift object (effect object, normal object, or decoration object), the image of the gift object, the price of the gift object, and information necessary to purchase other gift objects. May be included. The viewing user can select a gift object to be purchased based on the gift object purchase information displayed on the client device 10a. Selection of a gift object to be purchased may be performed by the operation of the client device 10a. When a gift object to be purchased is selected by the viewing user, a purchase request for the gift object is transmitted to the server device 20. The object purchase processing unit 21 h performs settlement processing based on the purchase request. When the settlement process is completed, the purchased gift object is held by the viewing user. In this case, in the possession list 23c, the object ID of the purchased gift object is stored in association with the user ID of the purchased viewing user.

  Gift objects that can be purchased may be different for each video. A purchasable gift object may be purchasable in a plurality of moving pictures. That is, the purchasable gift objects may include a unique gift object unique to each video and a common gift object that can be purchased in a plurality of videos. For example, the effect object indicating confetti may be a common gift object that can be purchased in a plurality of animations.

  In one aspect, when the effect object is purchased while watching a predetermined moving image, the effect object to be purchased is the moving image being viewed according to the completion of the settlement process for purchasing the effect object. It may be displayed automatically. When a normal object is purchased while watching a predetermined moving image, the normal gift object to be purchased is the moving image being viewed according to the completion of the settlement process for purchasing the normal object, as described above. May be displayed automatically.

  In another aspect, in response to the completion of the settlement process in the object purchase processing unit 21h for the effect object to be purchased, a settlement completion notification is sent to the client device 10a, and the effect object purchased in the client device 10a A confirmation screen may be displayed to confirm to the viewing user whether or not to make a display request of When the viewing user selects to request display of the purchased effect object, the display request for requesting display of the purchased effect object is displayed on the display request processing unit 21 f from the client device of the viewing user. And the display request processing unit 21 f may display the effect object to be purchased on the moving image 70. Even when the purchase target is a normal object, a confirmation screen may be displayed on the client device 10a for confirming with the viewing user whether to request the display of the purchased normal object, as described above.

  Next, the supporter computer 40 in one embodiment will be described. In one embodiment, the supporter computer 40 includes a computer processor 41, a communication I / F 42, a storage 43, a display 44, and an input interface 45.

  The computer processor 41 is, like the computer processor 21, various arithmetic devices such as a CPU. The communication I / F 42 is, like the communication I / F 22, various drivers, software, or a combination thereof for communicating with other devices. The storage 43 is, like the storage 23, various storage devices capable of storing data such as a magnetic disk. The display 44 is a liquid crystal display, an organic EL display, an inorganic EL display, or any other display device capable of displaying an image. The input input interface 45 is an arbitrary input interface that receives input from a supporter such as a mouse and a keyboard. The display 44 is an example of a second display device.

  FIG. 9 shows an example of an image displayed on the display 44 of the supporter computer 40. In the illustrated embodiment, the display image 46 displayed on the display 44 is a first display area 47a for displaying a display screen of a moving image in a first type client device (for example, a personal computer), and a second type And a second display area 47 b for displaying a display screen of a moving image in a client device (for example, a personal computer) of

  By monitoring the moving image displayed in the first display area 47a, the supporter can confirm whether the moving image delivered to the first type client device is normally displayed or not. By monitoring the moving image displayed in the second display area 47b, it can be confirmed whether the moving image distributed to the second type of client device is normally displayed.

  In the illustrated embodiment, the operation icon 48 includes a first operation icon 48a, a second operation icon 48b, a third operation icon 48c, a fourth operation icon 48d, and a fifth operation icon 48e. . The supporter can select a desired operation icon via the input interface 45. When the first operation icon 48a, the second operation icon 48b, the third operation icon 48c, the fourth operation icon 48d, and the fifth operation icon 48e are selected by the supporter, the supporter computer 40 performs the first operation. The input, the second operation input, the third operation input, the fourth operation input, and the fifth operation input are received.

  Next, functions implemented by the computer processor 41 will be more specifically described. The computer processor 41 functions as an additional information display unit 41a and a distribution management unit 41b by executing computer-readable instructions included in a predetermined program. At least a part of the functions implemented by the computer processor 41 may be implemented by a computer processor other than the computer processor 41 of the video distribution system 1. At least a part of the functions implemented by the computer processor 41 may be implemented by the computer processor 21, for example.

  The additional information display unit 41a is configured to add various additional information to the distribution moving image in response to various operation inputs from the supporter via the input interface 45.

  In one embodiment, when the first operation input is accepted in the supporter computer 40 by the selection of the first operation icon 48a, the additional information display unit 41a displays the moving image displayed on the display 39 based on the first operation input. , And the client apparatus is configured not to display the message 101. The message 101 is an example of first additional information.

  An example of the display image of the moving image displayed on the display 39 is shown in FIG. The display image 100 shown in FIG. 10 includes a message 101. The display image 100 is the same as the display image 70 (see FIG. 5) of the moving image distributed and displayed on the client device of the viewing user at the same timing except that the message 101 is included. That is, the display image 100 displayed on the display 39 is configured by adding the message 101 to the distributed moving image.

  The message 101 is, for example, a text message input by a supporter. The message 101 may include image information in addition to the textual information or in addition to the textual information. The message 101 may include an instruction on the actor's performance, an instruction on the actor's statement, an instruction on the actor's position, and various other instructions or contacts related to the moving image being delivered.

In one embodiment, when the supporter selects the first operation icon 48 a on the supporter computer 40, a message input window is displayed on the display 44 of the supporter computer 40. The supporter can input a message in a message input area in the window by operating the input interface 45. The input message is transmitted from the supporter computer 40 to the display 39. The display 39 and the client apparatuses 10a to 10c display the moving image distributed by the moving image distribution unit 21e. On the display 39, as shown in FIG. 10, a message received from the supporter computer 40 is displayed as a message 101 in a predetermined area of the moving image distributed from the moving image distribution unit 21e. On the other hand,
The message 101 or the information corresponding thereto is not displayed in the moving image displayed on the client devices 10a to 10c. According to this embodiment, it is related to the moving image being distributed to the staff on the distribution side who can view the display 39 other than the actor without affecting the moving image to be viewed by the viewing user of the client device. Can communicate instructions and communications.

  Subsequently, display of the second additional information will be described with reference to FIGS. 11a and 11b. In one embodiment, the additional information display unit 41a is displayed on the display 39 based on the second operation input when the second operation input is accepted in the supporter computer 40 by the selection of the second operation icon 48b. The interruption image is configured to be displayed on the moving image and the moving image displayed on the client device. The interrupt image is an example of the second additional information. The second operation input may be input to the server device 20 or the supporter computer 40 according to the operation of the controller 33a or 33b of the actor A1, or according to the operation of the controller 34a or 34b of the actor A2. Supporter computer 40

  FIG. 11a shows an example of the display image 110a displayed on the client device 10a when the second operation input is accepted, and FIG. 11b shows the display 39 when the second operation input is accepted. An example of the display image 110 b displayed on the screen is shown. Both the display image 110 a and the display image 110 b include the interrupt image 111. The interrupt image 111 is set to the top layer of the moving image being delivered. As a result, the interruption image 111 is displayed on the screen of the client device and the display 39 which are reproducing the moving image. Although there may be a slight visual difference between the interrupt image 111 displayed on the client device and the interrupt image 111 displayed on the display 39 depending on the performance of the device to be displayed, both are It is substantially the same image.

  In the illustrated embodiment, the interrupt image 111 is an image for emergency display to be displayed in place of a moving image including a normal virtual space and a character object when an unexpected situation occurs during moving image distribution. The supporter can select the second operation icon 48b in the supporter computer 40, for example, when trouble occurs in motion control of a character object due to a failure of a device used in the studio room R. The additional information display unit 41 a performs processing for displaying the interrupt image 111 on each client device and the display 39 in response to the selection of the second operation icon 48 b.

  When the first operation input is accepted by the selection of the first operation icon 48a while the interrupt image 111 is displayed on the display 39, the additional information display unit 41a causes the display 39 to superimpose on the interrupt image 111 to be a message. 101 may be displayed.

  According to the above embodiment, when an unexpected situation occurs during the delivery of a moving image, the interruption image 111 for emergency display is delivered to the client device and the display 39 instead of continuing to deliver the normal moving image as it is. Can. The distributor can collect the situation while the interrupt image 111 is displayed.

  Subsequently, display of third additional information and generation of a corrected character object will be described with reference to FIGS. 12a and 12b. In one embodiment, when the third operation input is accepted in the supporter computer 40 by the selection of the third operation icon 48c, the additional information display unit 41a generates a correction character object, and delivers a moving image including the correction character object. Configured to perform processing to This third operation input may be input to the server device 20 or the supporter computer 40 according to the operation of the controller 33a or 33b of the actor A1, or according to the operation of the controller 34a or 34b of the actor A2.

  12a shows another example of the display image 120a displayed on the client device 10a, and FIG. 12b shows an example of the display image 120b displayed on the client device 10a. The display image 120a of FIG. 12a includes the character object 71B, and the display image 120a of FIG. 12b includes the correction character object 171B described later. As described above, the expression of the character object 71B is controlled to change in synchronization with the movement of the expression of the actor A2 based on the face motion data of the actor A2. In the present specification, all or part of a portion of a character object that changes based on face motion data may be referred to as a “face portion”. In the illustrated embodiment, motion control is performed so that the eye movement of the character object 71B moves in synchronization with the eye movement of the actor A2, so the face unit 121 sets the position including both eyes of the character object 71B. It is done. The face unit 121 may be set to include the entire face of the character object 71B.

  In the display image 120a shown in FIG. 12a, the eye image of the character object 71B displayed in the face portion 121 is generated by applying face motion data to model data. On the other hand, in the display image 120b shown in FIG. 12b, the eye image displayed in the face portion 121 is the fitted image 122b prepared in advance before the start of moving image distribution. The inset image 122 b may be stored in the storage 23. The inset image 122 b is an example of third additional information.

  In one embodiment, the additional information display unit 41a is configured to display the fitted image 122b on the face unit 121 of the character object 71B when the third operation input is received. For example, when the third operation input is received, the additional information display unit 41a transmits a correction instruction to the animation generation unit 21c, and the animation generation unit 21c, based on the correction instruction from the additional information display unit 41a, An animation of the corrected character object 171B is generated by inserting the fitted image 122b instead of the image generated based on the face motion data into the face portion 121 of the character object 71B. In the present specification, a character object into which the inset image 122b is embedded may be referred to as a correction character object, not an image generated based on face motion data in the face section 121. The correction character object is an object in which a part of the character object generated by the animation generation unit 21c is corrected. In the above example, the animation of the corrected character object 171B is generated by fitting and displaying the fitting image 122b in the face portion 121 of the character object 71B generated by the animation generation unit 21c.

  The process of applying face motion data to model data to change the expression of a character object has a high processing load on the processor. Therefore, the expression of the character object may not be able to follow the expression of the actor timely. Motions other than voice and expression can follow the actor's vocalization and body movement in a timely manner, so if the expression of the character object can not follow the actor's expression in a timely manner, the viewing user may feel uncomfortable. According to the above embodiment, when the expression of the character object can not follow the expression of the actor timely, the correction character object incorporating the fitted image 122b prepared in advance is generated, and the moving image including the correction character object is generated. By distributing, it is possible to prevent the deterioration of the quality of the moving image due to the inability of the expression of the character object to follow the expression of the actor.

  The storage 23 may store a plurality of different types of images as candidates for the fitted image 122 b. A candidate of the fitted image 122b is displayed on the display 44 of the supporter computer 40, and an image selected by the supporter from among the candidates may be used as the fitted image 122b.

  Subsequently, display of a blind object will be described with reference to FIG. In one embodiment, when the fourth operation input is accepted by the selection of the fourth operation icon 48d, the additional information display unit 41a displays a blind object for hiding at least a part of the character object on which the decoration object is attached. Configured as. The fourth operation input may be input to the server device 20 or the supporter computer 40 according to the operation of the controller 33a or 33b of the actor A1, or according to the operation of the controller 34a or 34b of the actor A2.

  In the embodiment of FIG. 13, it is assumed that a T-shirt is selected as a decoration object to be attached to the character object 71B in the selection screen 80. The additional information display unit 41a can display a blind object so as to hide a portion of the character object 71B that is associated with the selected decoration object. In the display image 130 of FIG. 13, the blind object 131 is displayed so that at least the trunk associated with the T-shirt, which is the selected decoration object, of the character object 71B is hidden.

  Subsequently, the process of prohibiting the delivery of a moving image to the user who is not permitted to deliver will be described. In one embodiment, when the fifth operation input is accepted by the selection of the fifth operation icon 48e, the distribution management unit 41b prohibits distribution of moving images to a specific viewing user or a specific client device. Configured to do. In one embodiment, when the fifth operation icon 48 e is selected during distribution of a moving image, a list of users viewing the moving image is displayed. The distribution management unit 41b performs processing for canceling the distribution of the moving image to the user selected from the list by the supporter or the actor. For example, the distribution management unit 41b can write a flag for identifying the selected user (distribution prohibited user) in the distribution destination list of the moving image. The moving image distribution unit 21e can distribute the moving image to the users not included in the distribution destination list of the moving image and to which the flag for identifying the distribution prohibited user is not attached. As a result, the distribution of the moving image to the distribution prohibited user is discontinued. In another embodiment, the distribution management unit 41b can not use a part of functions normally available to the user for the user selected by the supporter or the actor from the list of users viewing the moving image being distributed. It may be configured to For example, the distribution management unit 41b may be configured to prohibit the posting of a message to the user who has posted an inappropriate message. A user who is prohibited from posting a message can continue viewing the moving image even after the prohibition, but can not post a message regarding the moving image.

  Next, with reference to FIG. 14 to FIG. 19, a moving image distribution process according to an aspect will be described. FIG. 14 is a flow chart showing a flow of moving picture delivery processing in one embodiment, and FIG. 15 is a flow chart showing a flow of processing for displaying first additional information in one embodiment, and FIG. FIG. 17 is a flow chart showing a flow of processing for displaying second additional information in the embodiment, FIG. 17 is a flow chart showing a flow of processing for displaying third additional information in one embodiment, and FIG. FIG. 19 is a flow chart showing a flow of processing for displaying a blind object in the embodiment, and FIG. 19 is a flow chart showing a flow of processing for prohibiting moving picture distribution to a distribution prohibited user in one embodiment.

  First, with reference to FIG. 14, a moving image delivery process according to an embodiment will be described. In the video distribution process, it is assumed that the actor A1 and the actor A2 perform in the studio room R. First, in step S11, body motion data that is a digital representation of the body movement of the actor A1 and the actor A2 and face motion data that is a digital representation of the face movement (expression) of the actor A1 and the actor A2 are generated Ru. The generation of body motion data is performed, for example, by the body motion data generation unit 21a described above, and the generation of face motion data is performed, for example, by the face motion data generation unit 21b described above.

  Next, in step S12, the body motion data and face motion data of the actor A1 are applied to the model data for the actor A1 to animate the first character object moving in synchronization with the movement of the body and expression of the actor A1. Is generated. Similarly, by applying the body motion data and face motion data of the actor A2 to the model data for the actor A2, an animation of the second character object moving in synchronization with the movement of the body and expression of the actor A2 is generated. . The generation of this animation is performed, for example, by the above-mentioned animation generation unit 21c.

  Next, in step S13, a moving image including an animation of the first character object corresponding to the actor A1 and an animation of the second character object corresponding to the actor A2 is generated. The voice of the actor A1 and the actor A2 may be synthesized with this moving image. The animation of the first character object and the animation of the second character object may be arranged in a virtual space. The generation of the moving image is performed, for example, by the above-described moving image generating unit 21 d.

  Next, the process proceeds to step S14, and the moving image generated in step S13 is distributed. This moving image is distributed to the client devices 10a to 10c and other client devices via the network 50. This moving image may be distributed to the supporter computer 40 and may be projected on the screen S in the studio room R. The video is distributed continuously over a predetermined distribution period. The distribution period of the moving image may be set to, for example, 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, 60 minutes, 120 minutes, and any other time.

  Next, the process proceeds to step S15, and it is determined whether the termination condition for terminating the distribution of the moving image is satisfied. The end condition is, for example, that the end time of the distribution has been reached, that the supporter computer 40 has instructed to end the distribution, or other conditions. If the end condition is not satisfied, the processes of steps S11 to S14 are repeatedly executed, and the distribution of the moving image including the animation synchronized with the movement of the actor A1 and the actor A2 is continued. If it is determined that the ending condition is satisfied for the moving image, the distribution processing of the moving image is ended.

  Next, the display processing of the first additional information performed during delivery of the moving image will be described with further reference to FIG. During distribution of a moving image, it is determined in step S21 whether or not the first operation input has been made. For example, when the first operation icon 48a is selected in the supporter computer 40, it is determined that the first operation input has been made. When the first operation input is made, the display process of the first additional information proceeds to step S22. In step S22, the message 101, which is an example of the first additional information, is displayed on the moving image displayed on the display 39, and the display process of the first additional information ends. At this time, the message 101 is not displayed on the client device.

  Next, with reference to FIG. 16 further, display processing of second additional information performed during distribution of a moving image will be described. During distribution of the moving image, it is determined in step S31 whether or not the second operation input has been made. For example, when the second operation icon 48b is selected in the supporter computer 40, it is determined that the second operation input has been made. When the second operation input is performed, the process of displaying the second additional information proceeds to step S32. In step S32, the interrupt image 111, which is an example of the second additional information, is displayed on the client device and the display 39, and the display process of the second additional information ends.

  Next, the display processing of the third additional information performed during delivery of the moving image will be described with further reference to FIG. During distribution of the moving image, it is determined in step S41 whether or not the third operation input has been made. For example, when the third operation icon 48c is selected in the supporter computer 40, it is determined that the third operation input has been made. When the third operation input is performed, the process of displaying the third additional information proceeds to step S42. In step S42, the fitted image 122b, which is an example of the third additional information, is displayed on the face portion of the character object, and the display process of the third additional information ends.

  Next, with reference to FIG. 18, the display process of the blind object performed during distribution of the moving image will be described. During distribution of the moving image, in step S51, it is determined whether or not a decoration object to be attached to the character object appearing in the moving image is selected. As described above, the decoration object attached to the character object may be selected from among the decoration object candidates included in the candidate list 23d. The decoration object selection process is performed, for example, by the above-mentioned decoration object selection unit 21g.

  If it is determined in step S51 that a decoration object is selected, the process proceeds to step S52. In step S52, it is determined whether the fourth operation input has been made. For example, when the fourth operation icon 48d is selected in the supporter computer 40, it is determined that the fourth operation input has been made. When the fourth operation input is made, the display processing of the blind object proceeds to step S53. In another embodiment, it may be determined that the fourth operation input has been made on the selection of the decoration object from the candidate list 23d. In this case, step S52 is omitted, and the display processing of the blind object proceeds from step S51 to step S53.

  In step S53, the blind object 131 is displayed to hide at least a part of the character object. For example, if the decorative object selected in step S51 is a T-shirt, a blind object 131 is added to the moving image so as to hide the trunk of the character object associated with this T-shirt.

  Next, with reference to FIG. 19 further, a distribution prohibition process for prohibiting distribution of a moving image to a specific user will be described. While the moving image is being distributed, it is determined in step S61 whether or not the fifth operation input has been made. For example, when the fifth operation icon 48e is selected in the supporter computer 40, it is determined that the fifth operation input has been made.

  When the fifth operation input is made, the distribution prohibiting process proceeds to step S62. In step S62, a user (distribution prohibited user) who prohibits distribution of the moving image is designated. For example, a list of users who are watching the moving image is displayed on the display 44 of the supporter computer 40, and a user selected by the supporter from this list is designated as a distribution prohibited user. The process for specifying the distribution prohibited user is executed, for example, by the distribution management unit 41b.

  Next, the distribution prohibiting process proceeds to step S63. In step S63, the moving image is distributed only to the users not specified as the distribution prohibited users among the users included in the distribution destination list of the moving image. The moving image distribution process is performed by, for example, the above-described moving image distribution unit 21e.

  According to the moving image distribution system 1 in the above embodiment, it is easy to cope with an unexpected situation that occurs during live distribution. For example, by displaying the message 101 on the display 39 without displaying the message 101 on the client device, staff on the distribution side who can view the actor or the other display 39 without affecting the moving image watched by the viewing user Can communicate instructions and information related to the video being delivered.

  According to another embodiment, when an unexpected situation occurs during distribution of a moving image, the interruption image 111 for emergency display is distributed to the client device and the display 39, instead of continuing distribution of the normal moving image as it is. be able to.

  According to another embodiment, even if the expression of the character object can not follow the expression of the actor timely, a corrected character object incorporating the fitted image 122b prepared in advance is generated, and this corrected character By distributing the moving image including the object, it is possible to prevent the deterioration of the moving image quality due to the inability of the expression of the character object to follow the expression of the actor.

  According to another embodiment, even when some viewing users take an inappropriate action such as posting an inappropriate message, it is possible to stop delivery of the moving image to the viewing users.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention. For example, the video to be distributed may be taken and generated outside the studio room R. For example, shooting for generating a distributed moving image may be performed at the actor or supporter's home.

  In the processing procedure described in the present specification, in particular, the processing procedure described using the flow chart, part of the steps (steps) constituting the processing procedure may be omitted, and as the steps constituting the processing procedure. It is possible to add a process that is not specified and / or to change the order of the process, and the processing procedure in which such omission, addition, or change of order is made is not limited to the present invention unless it deviates from the spirit of the present invention. Included in the scope of the invention.

In the following, the invention described in the original claim of the original application of the present application is appended.
[1]
A distribution server device configured to live distribute a moving image including animation of a character object generated based on the motion of an actor to a first client device used by a first user;
A first display device disposed at a position visible from the actor and configured to display the moving image;
A supporter configured to display first additional information on the moving image displayed on the first display device based on a first operation input, but not to display the first additional information on the first client device With a computer
Video distribution system with
[2]
The supporter computer is configured to display second additional information on the moving image displayed on the first display device and the moving image displayed on the first client device based on a second operation input. ,
The video delivery system described in [1].
[3]
The supporter computer is configured to generate a corrected character object obtained by correcting at least a part of the character object based on a third operation input, and to include the corrected character object in the moving image.
The video delivery system according to [1] or [2].
[4]
The character object has a face unit which is generated to move based on face motion data indicating movement of the face of the actor.
The distribution server apparatus is configured to display third additional information on the face unit based on the third operation input.
The video delivery system described in [3].
[5]
The storage device further includes a storage for storing a decoration object displayed in association with the character object,
The supporter computer is configured to display a blind object for hiding at least a part of the character object when displaying the decoration object in the moving image based on a fourth operation input.
The video delivery system according to any one of [1] to [4].
[6]
The distribution server device is configured to live distribute the moving image to a second client device used by a second user in addition to the first client device;
The first client device is a first type of device,
The second client device is a second type of device different from the first type of device,
The supporter computer includes a second display device for displaying a display image of the moving image displayed on the first client device and a display image of the moving image displayed on the second client device.
The video delivery system according to any one of [1] to [5].
[7]
The supporter computer is configured to prohibit distribution of the moving image to the first user based on a fifth operation input.
The video delivery system according to any one of [1] to [6].
[8]
A video distribution method performed by one or more computer processors executing computer readable instructions, comprising:
Live distributing a moving image including animation of a character object generated based on movement of an actor to a first client device used by a first user;
Displaying the moving image on a first display device arranged at a position visible from the actor;
Displaying first additional information on the moving image displayed on the first display device based on a first operation input, while not displaying the first additional information on the first client device;
A video delivery method comprising:
[9]
One or more computer processors,
Live distributing a moving image including animation of a character object generated based on movement of an actor to a first client device used by a first user;
Displaying the moving image on a first display device arranged at a position visible from the actor;
Displaying first additional information on the moving image displayed on the first display device based on a first operation input, while not displaying the first additional information on the first client device;
A delivery program that runs

  1 Video distribution system

Claims (10)

A distribution server device configured to live distribute a moving image including animation of a character object generated based on the movement of an actor to a first display device and a first client device used by a first user;
A supporter computer configured to display first additional information on the moving image displayed on the first display device while not displaying the first additional information on the first client device;
A storage for storing a decoration object displayed in association with the character object;
Equipped with
The supporter computer is configured to display a blind object for hiding at least a part of the character object when displaying the decoration object in the moving image based on a fourth operation input.
Video distribution system. The supporter computer is configured to display second additional information on the moving image displayed on the first display device and the moving image displayed on the first client device based on a second operation input. ,
The video distribution system according to claim 1. The supporter computer is configured to generate a corrected character object obtained by correcting at least a part of the character object based on a third operation input, and to include the corrected character object in the moving image.
The moving image distribution system according to claim 1 or 2. The character object has a face unit which is generated to move based on face motion data indicating movement of the face of the actor.
The distribution server apparatus is configured to display third additional information on the face unit based on the third operation input.
The video delivery system according to claim 3. The distribution server device is configured to live distribute the moving image to a second client device used by a second user in addition to the first client device;
The first client device is a first type of device,
The second client device is a second type of device different from the first type of device,
The supporter computer includes a second display device for displaying a display image of the moving image displayed on the first client device and a display image of the moving image displayed on the second client device.
The moving image distribution system according to any one of claims 1 to 4 . The supporter computer is configured to prohibit distribution of the moving image to the first user based on a fifth operation input.
The moving image distribution system according to any one of claims 1 to 5 . The moving image distribution system according to any one of claims 1 to 6 , wherein the first display device is used by the actor. The moving image distribution system according to any one of claims 1 to 7 , wherein the first display device displays an object selectable by the actor. A video distribution method performed by one or more computer processors executing computer readable instructions, comprising:
Live distributing a moving image including animation of a character object generated based on the motion of an actor to a first display device and a first client device used by a first user;
Displaying first additional information on the moving image displayed on the first display device while not displaying the first additional information on the first client device;
Displaying a blind object for hiding at least a part of the character object when displaying a decoration object displayed in association with the character object on the moving image based on a fourth operation input;
A video delivery method comprising: One or more computer processors,
Live distributing a moving image including animation of a character object generated based on the motion of an actor to a first display device and a first client device used by a first user;
Displaying first additional information on the moving image displayed on the first display device while not displaying the first additional information on the first client device;
Displaying a blind object for hiding at least a part of the character object when displaying a decoration object displayed in association with the character object on the moving image based on a fourth operation input;
A delivery program that runs

Images