JP7296846B2 - Live communication system with characters - Google Patents

Description

The present invention relates to visual communication technology over communication networks.

Communication via SNS (Social Networking Service) has become popular with the spread of smartphones. On SNS, not only text but also various contents such as photos and videos can be easily shared. SNS is a tool that makes people feel a strong bond (connection) between people.

On the other hand, in the field of computer games, there are games in which the user plays while conversing with a character (computer). Users often have a strong attachment to characters that do not exist in reality.

JP 2012-231281 A

A character is a virtual entity created by a game designer and does not have a sense of reality like a human being. Characters, on the other hand, have a unique charm due to their unreality. The inventors of the present invention thought that it would be possible to propose a new communication method by incorporating both the character-specific charm and the "real connection" that is specific to humans.

Specifically, a conversation between a distributor (human) and a viewer (human) is assumed. The distributor becomes a spokesperson for the character just like a voice actor in TV animation, and converses with viewers through the character. Assuming such communication, if the viewer can have a relationship with the character that goes beyond words, it is thought that the viewer's affection for the character can be further strengthened.

The present invention is an invention completed based on the above recognition, and its main purpose is to provide a technique for deepening non-verbal communication between characters and viewers, particularly in communication via characters. be.

A live communication system in one aspect of the present invention includes a distribution device operated by a distributor and a viewing device operated by a viewer.
The distribution device includes an imaging unit that captures an image of the distributor, a motion information generation unit that generates motion information indicating characteristics of the motion of the distributor from the captured image of the distributor, and a first message that receives a first message from the distributor. An acquisition unit and a transmission unit that transmits the operation information and the first message.
The viewing device includes a receiving unit for receiving action information and a first message, a viewing character display unit for displaying a moving image of a character on a screen of the viewing device and causing the character to move according to the action information, and a third message in accordance with the action of the character. a first message output unit that outputs one message; a food operation unit that receives a moving operation for a food object from a viewer; a food display unit that moves and displays the food object according to an operation by the viewer; Prepare.
The viewing character display unit displays a moving image of a character eating and drinking a food object when a predetermined eating and drinking condition is satisfied.

A live communication system in another aspect of the present invention includes a distribution device operated by a distributor and a viewing device operated by a viewer.
The distribution device includes an imaging unit that captures an image of the distributor, a motion information generation unit that generates motion information indicating characteristics of the motion of the distributor from the captured image of the distributor, and displays a moving image of the character on the screen of the distribution device, a distribution character display unit that moves a character according to movement information; a mouth identification unit that identifies the position coordinates of the character's mouth; A 1-message acquisition unit and a transmission unit that transmits the operation information and the first message.
The transmitting unit transmits the eating and drinking instruction as part of the operation information when the eating and drinking condition is satisfied.
The viewing device includes a receiving unit for receiving action information and a first message, a viewing character display unit for displaying a moving image of a character on a screen of the viewing device and causing the character to move according to the action information, and a third message in accordance with the action of the character. a first message output unit that outputs one message; a food operation unit that receives a moving operation for a food object from a viewer; a food display unit that moves and displays the food object according to an operation by the viewer; A food/drink identification unit that identifies the positional coordinates of the food/drink object, and a transmission unit that transmits the positional/coordinates of the food/drink object.
The eating/drinking determination unit of the delivery device determines that the eating/drinking condition is established when the distance between the position coordinates of the mouth of the character and the position coordinates of the food object is equal to or less than a first threshold value.
When transmitting action information, the transmitting unit of the distribution device does not transmit the moving image data of the character to the viewing device.
The viewing character display unit of the viewing device displays a moving image of the character eating and drinking the food object when the eating and drinking instruction is received.

According to the present invention, it becomes easier to realize rich communication through characters.

1 is a hardware configuration diagram of a live communication system; FIG. FIG. 2 is a schematic diagram for explaining a conversation service by a live communication system; FIG. 3 is a functional block diagram of a distribution device and a server; FIG. It is a functional block diagram of a viewing device. It is a screen figure of the viewer conversation screen displayed on a viewing-and-listening apparatus. FIG. 11 is a screen diagram of a distributor conversation screen displayed on the distribution device; FIG. 10 is a sequence diagram of processing for displaying a character on both the distribution device and the viewing device; FIG. 10 is a screen diagram showing a character when the distributor leans his or her body; FIG. 10 is a screen diagram showing a character trying to eat a food object on the viewing device; It is a schematic diagram for demonstrating eating-and-drinking conditions. It is a screen figure which shows the eating-and-drink animation in a viewing-and-listening apparatus. It is a schematic diagram which shows the image change in the case of a drink. FIG. 10 is a sequence diagram of processing for displaying characters eating and drinking on both the distribution device and the viewing device; It is a screen figure when adding a sign object to a food and drink object. FIG. 10 is a screen diagram when a character eats a food object to which a sign object is added; FIG. 10 is a schematic diagram showing how a part of the food object is divided and given. FIG. 10 is a screen diagram when a character catches a food object thrown by a viewer;

FIG. 1 is a hardware configuration diagram of a live communication system 100. As shown in FIG.
In the live communication system 100, a plurality of distribution devices 104a, 104b, . 108a, 108b, . A smartphone is assumed as the distribution device 104 and the viewing device 108, but a general-purpose computer such as a laptop PC may also be used.

A user who uses the distribution device 104 is called a "distributor". A user using the viewing device 108 is called a "viewer". The distributor becomes the service provider, and the viewer becomes the service beneficiary.

FIG. 2 is a schematic diagram for explaining a conversation service provided by the live communication system 100. As shown in FIG.
In this embodiment, a large number of viewing devices 108 can be connected simultaneously to one distribution device 104 . The following description assumes one-to-many communication. In FIG. 2, three viewers 112a, 112b, and 112c (hereinafter collectively referred to as “viewers 112” when not distinguished) are having a group conversation with one distributor 110. We assume a situation where

A character is associated with the distributor 110 in advance. A distribution character 114 a associated with the distributor 110 is displayed on the distribution device 104 . A viewing character 114 b is also displayed on the viewing device 108 . The delivery character 114a and the viewing character 114b are exactly the same character. Hereinafter, when the delivery character 114a and the viewing character 114b are not particularly distinguished, they are simply referred to as "character 114".

When the distributor 110 speaks in front of the distribution device 104 , the audio data is broadcast to the viewing device 108 . The distribution device 104 detects the movement of the distributor's mouth and also moves the mouth of the distribution character 114a. In conjunction with this, the viewing device 108 also moves the mouth of the viewing character 114b. To the viewer 112, it seems as if the viewing character 114b is speaking of his/her own will, not the distributor. In other words, the distributor 110 becomes a spokesperson for the character 114 as a "voice actor". Hereinafter, the voice message input by the distributor 110 will be referred to as "first message".

The viewer 112 inputs a message (hereinafter referred to as a "second message") to the viewing device 108 in text. The second message is broadcast to distribution device 104 and viewing device 108 . The viewer 112 converses with the character 114 (distributor 110) by inputting the second message while listening to the viewing character 114b.

The individuality of the character 114 is formed by the basic settings of the character 114 (such as the appearance of the character 114 and the richness of facial expression changes) and the voice quality and conversational ability of the distributor 110 .

FIG. 3 is a functional block diagram of distribution device 104 and server 102 .
Each component of the distribution device 104 and the server 102 includes computing units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, hardware including wired or wireless communication lines connecting them, It is implemented by software that is stored in a storage device and that supplies processing instructions to the calculator. A computer program may consist of a device driver, an operating system, various application programs located in their higher layers, and a library that provides common functions to these programs. Each block described below represents a functional block rather than a hardware configuration.
The same is true for the viewing device 108 .

(Distribution device 104)
Distribution device 104 includes user interface processing unit 116 , imaging unit 118 , data processing unit 120 , communication unit 122 and data storage unit 124 .
The communication unit 122 is in charge of communication processing. The imaging unit 118 images the distributor 110 . The user interface processing unit 116 receives operations from the distributor 110, and is in charge of user interface processing such as image display and audio output. The data storage unit 124 stores various data including image data of the character 114 . The data processing unit 120 executes various processes based on the data acquired by the communication unit 122 , the user interface processing unit 116 and the imaging unit 118 and the data stored in the data storage unit 124 . The data processing unit 120 also functions as an interface for the user interface processing unit 116 , imaging unit 118 , communication unit 122 and data storage unit 124 .

Communication unit 122 includes a transmitting unit 148 and a receiving unit 150 .

User interface processing unit 116 includes an input unit 126 that receives input from distributor 110 and an output unit 128 that outputs various types of information to distributor 110 .
Input unit 126 includes motion detection unit 130 , first message acquisition unit 132 , delivery sign input unit 134 and food/drink selection unit 136 . The motion detection unit 130 detects facial expressions and body motions of the distributor 110 via the imaging unit 118 . The first message acquisition unit 132 acquires a first message (voice message). The distribution signature input unit 134 receives a signature input from the distributor 110 for the "food and drink object" (described later). Food and drink objects will be described in detail later, but any image that represents food or drink, or at least a two-dimensional or three-dimensional image that represents what is commonly ingested and digested by humans, can be used. The food/drink selection unit 136 receives selections of a plurality of food/drink objects provided by a plurality of viewers 112 .

Output unit 128 includes delivery character display unit 138 , second message output unit 140 and food display unit 142 .
The delivery character display unit 138 causes the delivery device 104 to display the delivery character 114a. The second message output unit 140 causes the distribution device 104 to display a second message (text message) from the viewer 112 . The food display unit 142 displays the food object provided by the viewer 112 on the screen of the distribution device 104 .

Data processing unit 120 includes motion information generating unit 144 , eating/drinking determining unit 220 and mouth identifying unit 146 .
The motion information generation unit 144 generates “motion information” from the detection result of the motion detection unit 130 . The motion information is a feature vector indicating the facial expression of the distributor 110 (eg, a numerical value indicating the angle of the eyebrows, the degree of opening of the eyes and mouth, and the relative positional relationship (front, back, left, and right) between the distributor 110 and the distribution device 104). This is numerical data indicating the operating state of the distributor 110. The eating/drinking determination unit 220 determines whether or not the “eating/drinking condition” is satisfied, because it is assumed that the character 114 has ingested the food/drink object. In the present embodiment, it is assumed that the food and drink condition is satisfied when the food and drink object comes sufficiently close to the mouth of the character 114 (described later with reference to FIG. 9). detects the position coordinates of the mouth of the character 114 (hereinafter referred to as "mouth coordinates") The distribution device 104 and the viewing device 108 share the same screen coordinate system (two-dimensional coordinate system). The position coordinates of the mouth in the screen coordinate system (xy coordinate system) are shown.

(Server 102)
Server 102 includes communication unit 152 , live control unit 154 and data storage unit 156 .
The communication unit 152 is in charge of communication processing. The data storage unit 156 stores various data. The live control unit 154 mediates and controls communication between the distribution device 104 and the viewing device 108 based on the data acquired by the communication unit 152 and the data stored in the data storage unit 156 . Live control unit 154 also functions as an interface for communication unit 152 and data storage unit 156 .

Data storage portion 156 includes a character storage portion 158 that stores data of a plurality of characters 114 . The data storage unit 156 also stores image data of food objects. A food object is uniquely identified by a food ID.

FIG. 4 is a functional block diagram of the viewing device 108. As shown in FIG.
The viewing device 108 includes a user interface processing section 162 , a data processing section 166 , a communication section 164 and a data storage section 168 .
The communication unit 164 is in charge of communication processing. The user interface processing unit 162 receives operations from the viewer 112, and is in charge of user interface processing such as image display and audio output. A data storage unit 168 stores various data including image data of the character 114 . Both the distribution device 104 and the viewing device 108 each store data for the character 114 . The data processing unit 166 executes various processes based on the data acquired by the communication unit 164 and the user interface processing unit 162 and the data stored in the data storage unit 168 . Data processing unit 166 also functions as an interface for user interface processing unit 162 , communication unit 164 and data storage unit 168 .

Communication unit 122 includes a transmitting unit 186 and a receiving unit 190 .

The user interface processing unit 162 includes an input unit 170 that receives input from the viewer 112 and an output unit 172 that outputs various information to the viewer 112 .
Input unit 170 includes second message acquisition unit 174 , character selection unit 176 , food operation unit 178 and viewing sign input unit 222 . The second message acquisition unit 174 acquires a second message (text message). Character selection unit 176 accepts selection of character 114 to be a conversation target. The food/drink operation unit 178 receives selection of a food/drink object to be presented to the character 114 and operation of moving the food/drink object 226 . The viewing signature input unit 222 receives a signature input from the viewer 112 for the food object (described later).

Output unit 172 includes viewing character display unit 180 , first message output unit 182 , second message output unit 184 and food display unit 224 .
The viewing character display unit 180 causes the viewing device 108 to display the viewing character 114b. The first message output unit 182 outputs the first message from the distributor 110 by voice from the viewing device 108 . The second message output unit 184 causes the viewing device 108 to display the second message from the viewer 112 on the screen. The food display unit 224 displays food objects on the screen of the viewing device 108 .

The data processing unit 166 includes a food/drink identification unit 227 . The food/drink identification unit 227 detects the position coordinates of the food/drink object (hereinafter referred to as "food/drink coordinates"). The food coordinates indicate the position coordinates of the food object in the same screen coordinate system (xy coordinate system) as the mouth coordinates.

At the start of the live communication service, the live control unit 154 of the server 102 causes the viewing device 108 to display a character selection screen (not shown). The viewer 112 selects a viewing character 114b (distributor 110) with which he/she wishes to converse on the character selection screen displayed on the viewing device 108. FIG. The live control unit 154 presents the character 114 corresponding to the distributor 110 in a conversationable state (standby state) on the character selection screen as a conversationable character. The character selection unit 176 accepts selection of the viewing character 114b.
In the following description, it is assumed that the viewer 112 enjoys conversation with the viewing character 114b1 (see FIG. 5).

FIG. 5 is a screen diagram of a viewer conversation screen 196 displayed on the viewing device 108. As shown in FIG.
The viewing character display unit 180 of the viewing device 108 displays a moving image of the viewing character 114b1 selected as the chat partner. The viewer 112 enjoys conversation with the viewing character 114b1. The viewer 112 cannot see the distributor 110 . The first message output unit 182 of the viewing device 108 outputs the voice (first message) from the distributor 110 in charge of the viewing character 114b1. The viewer 112 inputs a message (second message) to the viewing character 114b1 by text. Server 102 causes the first message and the second message to be shared between distribution device 104 and viewing device 108 .

The second message output unit 184 causes the message area 198 to display the second message as text. Since one broadcaster 110 converses with multiple viewers 112 concurrently, the server 102 broadcasts the second message of each viewer 112 to the broadcast device 104 and the multiple viewers 112 . Viewers 112 can view not only their own secondary messages but also the secondary messages of other viewers 112 through message area 198 . The second message output unit 184 arranges and displays the second messages in order of reception.

The viewer 112 can provide a food object to the viewing character 114b1 by selecting the food button 200 (details will be described later).

FIG. 6 is a screen diagram of a distributor conversation screen 206 displayed on the distribution device 104. As shown in FIG.
The distributor conversation screen 206 shown in FIG. 6 is displayed on the distribution device 104 of the distributor 110 in charge of the distribution character 114a1 (corresponding to the viewing character 114b1). The delivery character display unit 138 displays the delivery character 114a1 as a moving image. The viewing character 114b1 and the delivery character 114a are the same character and have the same movement. The distributor 110 converses with a plurality of viewers 112 (the viewers 112 who have selected the viewing character 114b1 as a conversation partner) while confirming the movement of the distribution character 114a1, which is their alter ego, on the distribution device 104. The distributor 110 inputs the first message by voice to the distribution device 104 . The transmitter 148 transmits the first message to the server 102 , and the live controller 154 of the server 102 distributes the first message to the plurality of viewing devices 108 .

Second message output unit 140 displays second messages from viewer 112 in message area 198 in chronological order. The distributor 110 speaks to an unspecified number of viewers 112 while watching the second messages displayed one after another in the message area 198 . Camera 208 (imaging unit 118 ) of distribution device 104 captures image of distributor 110 . The motion detection unit 130 detects the expression and motion of the distributor 110 from the captured image, and the motion information generation unit 144 generates motion information. The delivery character display unit 138 determines the motion of the delivery character 114a1 based on the motion information. The viewing character 114b1 is also displayed as a moving image on the viewing device 108 based on the same motion information, although the details will be described later.

FIG. 7 is a sequence diagram of processing for displaying character 114 on both distribution device 104 and viewing device 108 .
The motion detector 130 of the distribution device 104 detects the motion of the distributor 110 (S10). The motion information generator 144 generates motion information (S12). The motion information is numerical data indicating the facial expression of the distributor 110, the angle of the body, the perspective of the distributor 110 from the imaging unit 118, and the like (S12). The action information also includes "eating and drinking instructions", which will be described later.

The transmission unit 148 of the distribution device 104 transmits the motion information to each viewing device 108 via the server 102 (S14). The distribution character display unit 138 generates a moving image of the distribution character 114a1 based on the action information (S16), and causes the distribution device 104 to display the distribution character 114a1 as a moving image (S18).

On the other hand, the viewing character display unit 180 of the viewing device 108 also generates a moving image of the viewing character 114b1 based on the received action information (S20), and causes the viewing device 108 to display the viewing character 114b1 as a moving image (S22).

Since the delivery character 114a1 and the viewing character 114b1 are the same character and their motions are determined based on the same action information, the delivery character 114a1 and the viewing character 114b have the same appearance in each of the delivery device 104 and the viewing device 108. Do the same.

The distribution device 104 does not transmit the moving image (character image) of the character 114 itself to the viewing device 108 , but transmits only action information defining the action of the character 114 . Since the amount of data to be transmitted is smaller than that of transmitting the moving image itself, it is easy to quickly link the movements of the distribution character 114a1 and the viewing character 114b1. In other words, since the difference between the time it takes for the action information to be reflected as the movement of the distribution character 114a1 and the time it takes for the same action information to be reflected as the movement of the viewing character 114b1 is small, the viewer 112 will not be able to see the character 114 (distribution character 114). You can enjoy a comfortable conversation with the person 110).

The first message (audio message) and the action information may be integrated and transmitted from the distribution device 104 to the viewing device 108 . In this case, the viewing character display unit 180 makes it difficult for the motion of the viewing character 114b1 to deviate from the voice message. Also, transmitting the motion information for determining the moving image instead of the moving image itself is effective in suppressing the communication load on the Internet 106 .

FIG. 8 is a screen diagram showing the character 114 when the distributor 110 leans his body.
When the distributor 110 tilts his body to the left or right, the motion detection unit 130 detects the body tilt. The motion information generator 144 includes the tilt angle and the tilt direction in the motion information. The viewing character display unit 180 tilts the viewing character 114b1 according to the action information. The delivery character display unit 138 also tilts the delivery character 114a1 according to the same action information. In this way, when the distributor 110 tilts his body to the left or right, the characters 114 on both the distributor side and the viewer side tilt their bodies to match the distributor 110 . When the distributor 110 tilts his body back and forth, the character 114 similarly tilts his body back and forth. The motion detection unit 130 may recognize forward tilting and backward tilting of the distributor 110 from the size of the distributor 110 appearing in the captured image.

Generally, it is said that human psychology appears in the inclination of the body. For example, it is said that when people are interested in conversation, they tend to lean forward (the posture of listening), and when they are upset, their bodies sway from side to side. By reflecting the body movement of the distributor 110 on the character 114, the viewer 112 can sense the psychology of the character 114 (distributor 110).

Also, the distributor 110 can change the position of the mouth of the character 114 by moving the body. The mouth identification unit 146 of the delivery device 104 appropriately detects the mouth coordinates of the delivery character 114a1. The success or failure of the eating and drinking condition is determined based on the mouth coordinates of the delivery character 114a1, and the details will be described later.

FIG. 9 is a screen diagram showing the character 114 about to eat the food object 226 on the viewing device 108. As shown in FIG.
The viewer 112 selects one food object 226 from a plurality of types of food objects 226 by touching the food button 200 (see FIG. 5). The food operation unit 178 accepts selection of the food object 226 . In FIG. 9, a food object 226 imitating the chocolate candy Pocky (registered trademark) is selected.

The food display unit 224 causes the viewing device 108 to display the food object 226 on the screen. The viewing device 108 transmits the food ID of the food object 226 to the distribution device 104 via the server 102 . The food display unit 142 of the distribution device 104 also causes the distribution device 104 to display the food object 226 on the screen. As a result, the same food object 226 is displayed on both the distribution device 104 and the viewing device 108 . The viewing device 108 does not transmit the image data of the food object 226 itself to the distribution device 104, but transmits the food ID, thereby suppressing the communication load.

The viewer 112 can freely move the food object 226 two-dimensionally on the screen by touching the food object 226 on the viewing device 108 . The food operation unit 178 of the viewing device 108 receives the viewer's 112 movement operation for the food object 226 . The food/drink identification unit 227 identifies food coordinates indicating the position of the food/drink object 226 .

The transmission unit 186 of the viewing device 108 transmits the food coordinates to the distribution device 104 . The food display unit 142 of the distribution device 104 determines the display position of the food object 226 according to the received food coordinates. Therefore, the same food object 226 is displayed at the same position (mouth coordinates) on both the distribution device 104 and the viewing device 108 . When the viewer 112 brings the tip of the food object 226 close to the mouth of the viewing character 114b1, the eating and drinking condition is satisfied, and the viewing character 114b1 (distribution character 114a1) eats and drinks the food object 226. ”) is displayed. The eating and drinking moving image is prepared in advance in both the distribution device 104 and the viewing device 108 as "moving image in which the character 114 holds the food object 226 in his mouth, bites it, and moves his mouth."

FIG. 10 is a schematic diagram for explaining eating and drinking conditions.
The food identification unit 227 of the viewing device 108 identifies the coordinates of the tip of the food object 226 on the screen as the food coordinates P2. The transmission unit 186 of the viewing device 108 transmits the food and drink coordinates P2 to the distribution device 104 as appropriate. On the other hand, the mouth identification unit 146 of the delivery device 104 appropriately detects the mouth coordinates P1 of the delivery character 114a1. When the distance d1 between the mouth coordinates P1 and the food coordinates P2 is within a predetermined threshold, the food/drink determination unit 220 of the distribution device 104 determines that the tip of the food/drink object 226 and the mouth of the character 114 are sufficiently close to each other. , it is determined that the eating and drinking conditions are met.

When the eating and drinking condition is satisfied, the distribution character display unit 138 of the distribution device 104 displays a food and drink moving image, which is a moving image of the food and drink object 226 being eaten by the distribution character 114a1. On the other hand, the transmitting unit 148 of the distribution device 104 transmits to the viewing device 108, as part of the operation information, a “drinking instruction” indicating that the eating and drinking condition is satisfied. When the viewing character display unit 180 of the viewing device 108 receives the eating and drinking instruction, the viewing character 114b1 displays a food and drink moving image, which is a moving image of eating and drinking the food object 226. FIG. The eating and drinking video data itself is not transmitted from the distribution device 104 to the viewing device 108 .

FIG. 11 is a screen diagram showing a eating and drinking video on the viewing device 108. As shown in FIG.
When the eating and drinking conditions are satisfied, the distribution device 104 transmits a eating and drinking instruction to the viewing device 108 . At this time, the viewing character 114b1 opens its mouth, stuffs part of the food object 226 into its mouth, and eats the food object 226. In this way, the viewing character display unit 180 of the viewing device 108 displays the eating and drinking animation of the viewing character 114b1 when receiving the eating and drinking instruction.

The food display unit 224 of the viewing device 108 changes the image of the food object 226 . In FIG. 11, half of the food object 226 is missing and half remains. With such an expression method, when the viewer 112 brings the food object 226 close to the mouth of the character 114, the viewer 112 feels that the character 114 has eaten the food object 226. You can have a realistic feeling that you are doing the act of eating and drinking after accepting a gift from.

FIG. 12 is a schematic diagram showing image changes in the case of a drink.
The food object 226 is not limited to food and may be a drink. The food object 226 shown in FIG. 12 represents a drink modeled after tapioca milk tea (registered trademark). Tapioca milk tea (registered trademark) includes milk 232 (liquid) and tapioca balls 230 (solid), and is eaten by sucking it up through a straw 228 . The food/drink identification unit 227 identifies the tip of the straw 228 as the food/drink coordinates P2. The eating and drinking condition is met when the tip of the straw 228 is brought close to the mouth of the viewing character 114b1.

When the eating and drinking conditions are satisfied, the character 114 drinks tapioca milk tea (registered trademark) little by little. The food display portion 142 gradually lowers the water level of the milk 232 over time to represent the appearance of the character 114 drinking tapioca milk tea (registered trademark) (food object 226). Tapioca dumplings 230 are also sucked up from the straw 228 as appropriate.

In this way, the viewer 112 can feel that the character 114 is drinking the drink presented by the viewer 112 by providing the food object 226 as a drink. In the live communication system 100, the viewer 112 not only enjoys conversation with the character 114, but also realizes the actions of "giving food and drink" and "having food and drink (accepted)", so that the viewer 112 can interact with the character 114. It heightens the sense of reality.

FIG. 13 is a sequence diagram of processing for displaying the character 114 eating and drinking on both the distribution device 104 and the viewing device 108 .
The viewer 112 touches the food button 200 (see FIG. 5) to select the food object 226 (S30). The transmission unit 186 of the viewing device 108 transmits the food ID of the selected food object 226 to the server 102 (S32). The server 102 associates and registers the food ID and the image of the food object 226 in advance, and transmits the image information of the food object 226 (hereinafter also referred to as "food image") to the distribution device 104 ( S34).
Note that when the distribution device 104 already has food and drink images, the server 102 may transmit food and drink IDs to the distribution device 104 instead of the food and drink images.

Viewer 112 can move food object 226 . The food object 226 of the viewing device 108 detects food coordinates, and the transmission unit 186 transmits the food coordinates to the server 102 (S36). The communication unit 152 of the server 102 transmits the food coordinates to the distribution device 104 (S38).

The food display unit 142 of the distribution device 104 displays the food object 226 at the designated food coordinates (S40). Similarly, the food display unit 224 of the viewing device 108 also displays the food object 226 (S41). The food identification unit 227 of the viewing device 108 periodically detects the food coordinates, and the transmission unit 186 also periodically transmits the food coordinates to the server 102 (S42, S44).

The eating/drinking determination unit 220 of the distribution device 104 determines whether or not the eating/drinking conditions are satisfied each time the food/drink coordinates are received. When the eating and drinking condition is satisfied (S46), the viewing device 108 of the distribution device 104 transmits a eating and drinking instruction to the server 102 (S48). The server 102 transfers the eating and drinking instruction to the viewing device 108 (S49).

The delivery character display unit 138 of the delivery device 104 displays the eating and drinking animation of the delivery character 114a1 (S50). Further, the food display unit 142 of the distribution device 104 displays the food object 226 partially lost due to eating and drinking (S52). Similarly, the viewing character display unit 180 of the viewing device 108 displays the eating and drinking animation of the viewing character 114b1 (S54). The food/drink display unit 224 of the viewing device 108 also displays the food/drink object 226 partially lost due to eating/drinking (S56).

In FIG. 13, the server 102 transmits the food image to the distribution device 104, and the distribution device 104 moves and displays the food object 226 indicated by the food image based on the food coordinates. Once the distribution device 104 can acquire the food and drink image, it does not need to acquire the food and drink image again. After obtaining the food image, the distribution device 104 independently moves and displays the food object 226 according to the food coordinates received as appropriate. Since the food image of the food object 226 is not always transmitted from the viewing device 108 to the distribution device 104 , the communication load is small, and the food object 226 can be smoothly moved and displayed on the distribution device 104 .

Multiple viewers 112 may seek to provide multiple food objects 226 to character 114 at the same time. In this case, the delivery device 104 receives multiple food IDs. The distributor 110 selects and eats a food object 226 by bringing the mouth of the character 114 close to the food object 226 that the distributor 110 wants to eat from among the plurality of food objects 226 corresponding to the plurality of food IDs.

Alternatively, the distributor 110 may tap the food object 226 to be eaten from the plurality of food objects 226 to select it. In this case, the food selection unit 136 of the distribution device 104 receives selection of the food object 226 by the distributor 110 . The food display unit 142 of the distribution device 104 displays only the selected food object 226 on the distribution device 104 .

For example, suppose viewer 112a provided food object 226a and viewer 112b provided food object 226b. A food object 226 a and a food object 226 b are displayed on the delivery device 104 . Here, assume that the distributor 110 selects the food object 226a. The transmission unit 148 of the distribution device 104 transmits the food ID of the selected food object 226 a to all the viewing devices 108 . The food coordinates of food object 226 a are distributed from distribution device 104 to all viewing devices 108 . Therefore, the same food object 226a is displayed at the same position on all viewing devices 108 . When the eating and drinking conditions are met, the same eating and drinking video is displayed on all the viewing devices 108 .

The distributor 110 selects one of the food objects 226 provided by the viewers 112 . In the above example, when the distributor 110 selects the food object 226a, all the viewing devices 108 display a moving image of the food object 226a being eaten by the character 114. FIG. Therefore, the viewer 112a who is provided with the food object 226a can have a special feeling of being "chosen".

FIG. 14 is a screen diagram when adding the sign object 236 to the food object 226. As shown in FIG.
Viewer 112 can also manipulate food object 226 using tool object 234 . In FIG. 14, a sign object 236 indicating the message "Congratulations" is drawn with a tool object 234 imitating ketchup for a food object 226 imitating an omelet.

After selecting the food object 226, the viewer 112 instructs execution of the processing mode. The food/drink operation unit 178 receives an input of a processing mode. In the processing mode, the food display unit 224 enlarges and displays the food object 226 . In processing mode, viewer 112 can select tool object 234 (ketchup) and move tool object 234 freehand to write a message on food object 226 (omelet). The message is integrated with food object 226 as sign object 236 . Here, viewer 112 is about to present character 114 with an omelet with "Congratulations" painted on it in ketchup to celebrate his favorite character 114's birthday.

The transmission unit 186 of the viewing device 108 transmits the food ID of the food object 226 and the image of the sun object sign object 236 (hereinafter referred to as “signature image”) to the distribution device 104 via the server 102 . The delivery device 104 acquires the food object 226 and the signature image and generates an image showing the signed omelet. The distribution device 104 also sends the new food object 226 representing the signed omelet to the other viewing devices 108 . With such a control method, all the participants of the live communication system 100 share the "autographed omelet image".

FIG. 15 is a screen diagram when the character 114 eats the food object 226 to which the sign object 236 is added.
When the viewer 112 brings the food object 226 close to the mouth of the viewing character 114b1, the food and drink condition is established, and a food and drink animation in which the viewing character 114b1 eats the food object 226 is displayed. At this time, the food display unit 224 causes the food object 226 to partially disappear. In addition, the food display unit 224 also causes a part of the sign object 236 to be lost in accordance with the loss of part of the food object 226 . The same applies to the distribution device 104 and other viewing devices 108 .

The viewer 112 can create a highly original food object 226 by using the tool object 234 . In addition, the viewer 112 can further feel a bond and a special relationship with the character 114 when the character 114 eats the food object 226 with his/her own feelings through the signature object 236 . The viewer 112 may save the food object 226 to which the sign object 236 is added or the scene where the character 114 eats the food object 226 as a moving image or a still image.

FIG. 16 is a schematic diagram showing how part of the food object 226 is divided and given.
Food objects 226 shown in FIG. 16 are three ice creams 238 (ice cream 238 a , ice cream 238 b and ice cream 238 c ) piled up in a glass container 242 . Of the three ice creams 238, the viewer 112 can separate the ice cream 238c (first partial object) from the food object 226 by the spoon object 240 (dividing operation). The viewer 112 moves the spoon object 240 with a finger so that the tip of the spoon object 240 overlaps the ice cream 238c. At this time, the food display unit 224 of the viewing device 108 displays an image of the ice cream 238 c being scooped from the food object 226 by the spoon object 240 . Similarly, the position coordinates of the spoon object 240 are also transmitted to the distribution device 104, and the food display unit 142 of the distribution device 104 independently displays an image of the ice cream 238c scooped by the spoon object 240. . The same applies to other viewing devices 108 as well.

Viewer 112 moves ice cream 238c to character 114's mouth. The food object 226 of the viewing device 108 transmits the position coordinates of the ice cream 238c to the distribution device 104 as the food coordinates, and the food and drink determination unit 220 of the distribution device 104 determines whether the food and drink conditions are satisfied based on the food and drink coordinates. do. When the eating and drinking condition is satisfied, a eating and drinking animation is displayed in which the character 114 receives the ice cream 238c from the spoon object 240 with his mouth and eats it. Ice cream 238c is erased from the screen.

Food object 226 includes ice cream 238a, ice cream 238b, and glass container 242, resulting in an image missing ice cream 238c (residual object). The viewer 112 can also scoop the ice cream 238a (the second partial object) with the spoon object 240 (split operation) and similarly feed the ice cream 238a to the character 114. FIG. At this time, a food and drink animation in which the character 114 eats the ice cream 238a is displayed. Food object 226 includes ice cream 238b and glass container 242, resulting in an image that also lacks ice cream 238a. Ice cream 238a is erased from the screen.

According to such a control method, the viewer 112 can enjoy the act of scooping a portion of the food (such as the ice cream 238c) and making the character 114 eat it. Distributor 110 also moves his or her body to bring the mouth of character 114 closer to ice cream 238c, thereby causing character 114 to receive ice cream 238c. In this way, by realizing the action of "taking out" and "passing" a part of the food object 226, the viewer 112 can easily have a feeling of interacting with the character 114 as if they were in contact with their lover. Become.

FIG. 17 is a screen diagram when the character 114 catches the food object 226 thrown by the viewer 112 .
The food object 226 shown in FIG. 17 imitates a marshmallow. The viewer 112 can throw the food object 226 by placing a finger on the food object 226 and performing a flick operation (flicking operation with the finger). The food operation unit 178 detects a flick operation on the food object 226 and detects the direction and speed of the flick. The food display unit 224 displays on the screen how the food object 226 flies in a parabola based on the speed and direction of the flick. At this time, the food object 226 of the viewing device 108 appropriately detects the food coordinates, and the transmission unit 186 transmits the food coordinates to the distribution device 104 .

The eating/drinking determination unit 220 of the distribution device 104 calculates the distance between the food coordinates of the food object 226 that changes as appropriate and the mouth coordinates of the character 114, and when the distance between the two coordinates is within the threshold, the eating/drinking condition is established. It is determined that The character 114 can eat the food object 226 when the mouth of the character 114 and the food object 226 are sufficiently close. According to such a control method, the so-called “marshmallow catch” can be enjoyed with the character 114 .

<Summary>
The live communication system 100 has been described above based on the embodiment.
According to the present embodiment, the viewer 112 can not only enjoy conversation with the character 114, but also can have the character 114 eat and drink. can be enhanced. The eating and drinking condition is met when the food and drink object 226 comes sufficiently close to the mouth of the character 114 . The viewer 112 brings the food object 226 close to the mouth of the character 114 , and the character 114 (distributor 110 ) brings the mouth close to the food object 226 . A certain degree of communication is required between the viewer 112 and the character 114 (distributor 110) in order to establish the eating and drinking conditions. When the character 114 eats and drinks the food object 226, the viewer 112 can have a unique feeling as if the character 114 and the viewer 112 are in harmony or communicate with each other.

When humans become good friends, they often go out to eat together. The joint act of "eating together" has an important meaning in enhancing intimacy between humans. Also, buying souvenirs at a travel destination is an important act for increasing intimacy and confirmation. Focusing on such psychological meanings related to food and drink, the viewer 112 presents the food and drink object 226 to the character 114, and the character 114 eats or drinks the food and drink object 226. The live communication system 100 realizes the feeling of being together. Not only is the food object 226 given, but also the food object 226 is brought close to the mouth of the character 114, and the character 114 (distributor 110) brings the mouth close to the food object 226 to establish the food and drink condition. As a result, the viewer 112 can have a feeling that the character 114 has understood his sincerity.

Furthermore, when the character 114 eats the food object 226, the food object 226 may be lost (see FIG. 11), the image of the food object 226 may be changed (see FIG. 12), or the food object 226 may be changed. By distributing a portion and displaying the remainder (see FIG. 16), it is possible to enhance the rendering effect as if the character 114 really ingested the food object 226 .

It should be noted that the present invention is not limited to the above-described embodiments and modifications, and can be embodied by modifying constituent elements without departing from the scope of the invention. Various inventions may be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments and modifications. Also, some components may be deleted from all the components shown in the above embodiments and modifications.

[Modification]
In the present embodiment, it has been described that the first message is voice and the second message is text. Alternatively, the first message may be text and the second message may be voice. Also, a message may be conveyed by an icon image such as a stamp image or a pictogram.

In the present embodiment, it has been described that the delivery device 104 determines whether the eating and drinking conditions are met. As a modified example, the viewing device 108 may include the eating/drinking determination unit 220 and the mouth specifying unit 146 instead of the distribution device 104 . In this case, the mouth identifying unit 146 of the viewing device 108 detects the mouth coordinates, and the eating/drinking determination unit 220 of the viewing device 108 determines the eating/drinking condition based on the mouth coordinates and the food/drink coordinates. When the eating/drinking condition is satisfied, the viewing device 108 transmits an eating/drinking instruction to the distribution device 104 . Server 102 also transmits eating and drinking instructions to other viewing devices 108 . The distribution device 104 and the viewing device 108 may each display the eating and drinking video when the eating and drinking condition is satisfied.

Food object 226 may be provided for a fee. The viewer 112 may purchase the food object 226 for a fee and provide it to the character 114 . Moreover, the food operation unit 178 may allow the viewer 112 to obtain the food object 226 for free only for a predetermined period, for example, from 19:00 to 20:00.

The eating/drinking determination unit 220 has been described as determining whether the eating/drinking conditions are satisfied based on the food/drink coordinates and the mouth coordinates. Alternatively, the eating and drinking condition may be satisfied when the viewer 112 brings the food and drink object 226 close to the character 114 . Also, the output unit 128 of the distribution device 104 may display the food and drink button on the screen when the food and drink object 226 is provided. The character 114 may eat and drink the food object 226 when the distributor 110 touches the food and drink button.

In the present embodiment, it has been described that the food object 226 is moved by the viewer 112 touching and moving the food object 226 . As a modification, a camera may be used to detect the hand movement or body movement of the viewer 112, and the position of the food object 226 (food coordinates) may be changed accordingly.

Character 114 (distributor 110) may eat multiple food objects 226 in the order in which they are served. For example, viewing device 108a may provide food object 226a, and viewing device 108b may provide food object 226b a short time later. The food display unit 142 of the distribution device 104 may first display only the food object 226a and, on condition that the food object 226a is eaten, display the next food object 226b.

A satiety level may be set for the character 114 . The eating/drinking determination unit 220 may establish the eating/drinking condition on the condition that the character's 114 satiety level is equal to or less than a predetermined value. When the character 114 eats the food object 226, the food determination unit 220 increases the degree of fullness. In addition, the eating/drinking determination unit 220 reduces the satiety level of the character 114 over time. According to such a control method, the viewer 112 needs to provide the food object 226 to the character 114 when the character 114 is hungry. Depending on the character 114, the rate of increase or decrease of the degree of satiety may be varied.

In this embodiment, viewer 112 designs sign object 236 freehand by using tool object 234 . As a modification, the food object 226 may be decorated with a tool object 234 imitating a stamp, or the food object 226 may be decorated with items such as small flags and powder.

The viewer 112 may give the character 114 not only the food object 226 but also decoration objects such as hats and clothes. At this time, a signature or stamp may be added to the decoration object. According to such a control method, a highly original decoration object can be presented to the character 114 .

Not only the viewer 112 but also the distributor 110 may decorate the food object 226 with the sign object 236 . For example, the distributor 110 designates a processing mode for the food object 226 provided by the viewer 112 . In processing mode, the distributor 110 adds the signature object 236 to the food object 226 . The distribution device 104 transmits the food image of the food object 226 to which the signature object 236 is added to the viewing device 108 . As a result, the food image of the signed food object 226 is shared with all participants. The character 114 who provided the food object 226 moves the food object 226 , and the character 114 eats the food object 226 .

According to such a control method, the distributor 110 (character 114) can return an original message such as "thank you" to the food object 226 provided by the viewer 112. FIG. The viewer 112 may photograph the original food object 226 processed by the distributor 110 (character 114).

The distributor 110 (character 114 ) may sign goods (objects) such as colored paper presented by the viewer 112 .

The viewer 112 may create an original food object 226 by cooking. For example, the viewer 112 selects several material objects from multiple material objects. The viewing device 108 includes a food and drink creator (not shown) that generates various food and drink objects 226 based on the selected material objects. According to such a control method, the viewer 112 can enjoy the pleasure of cooking and the pleasure of having the food eaten.

100 live communication system, 102 server, 104 distribution device, 106 Internet, 108 viewing device, 110 distributor, 112 viewer, 114 character, 114a distribution character, 114b viewing character, 116 user interface processing unit, 118 imaging unit, 120 data processing unit, 122 communication unit, 124 data storage unit, 126 input unit, 128 output unit, 130 motion detection unit, 132 first message acquisition unit, 134 delivery sign input unit, 136 food and drink selection unit, 138 delivery character display unit, 140 second message output unit, 142 food and drink display unit, 144 action information generation unit, 146 mouth identification unit, 148 transmission unit, 150 reception unit, 152 communication unit, 154 live control unit, 156 data storage unit, 158 character storage unit , 162 user interface processing unit, 164 communication unit, 166 data processing unit, 168 data storage unit, 170 input unit, 172 output unit, 174 second message acquisition unit, 176 character selection unit, 178 food and drink operation unit, 180 viewing character display unit 182 first message output unit 184 second message output unit 186 transmission unit 190 reception unit 196 viewer conversation screen 198 message area 200 food and drink button 206 broadcaster conversation screen 208 camera 220 Food/drink determination unit 222 Viewing sign input unit 224 Food/drink display unit 226 Food/drink object 227 Food/drink identification unit 228 Straw 230 Tapioca dumpling 232 Milk 234 Tool object 236 Sign object 238 Ice cream 240 spoon object, 242 glass container

Claims (10)

Including a distribution device operated by a distributor and a viewing device operated by a viewer,
The distribution device
an imaging unit that captures an image of a distributor;
a motion information generation unit that generates motion information indicating features of the motion of the distributor from the captured image of the distributor;
a first message obtaining unit that receives a first message from a distributor;
a transmission unit that transmits the operation information and the first message;
The viewing device is
a receiver that receives the motion information and the first message;
a viewing character display unit that displays a moving image of a character on the screen of the viewing device and moves the character according to the motion information;
a first message output unit that outputs the first message in accordance with the movement of the character;
a food operation unit that receives a movement operation for a food object from a viewer;
a food display unit that moves and displays the food object according to an operation by a viewer;
The live communication system, wherein the viewing character display unit displays a moving image of the character eating and drinking the food object when a predetermined eating and drinking condition is satisfied. The food display unit of the viewing device changes the position coordinates of the food object in accordance with the position of the viewer's hand,
In the viewing character display unit, when the distance between the position coordinates of the food object and the position coordinates of the character's mouth is equal to or less than a first threshold value, the eating condition is satisfied and the character eats and drinks the food object. 2. The live communication system according to claim 1, wherein moving images are displayed. 3. The viewing device according to claim 1, wherein the viewing character display unit of the viewing device displays an image in which a part of the food object is lost when the character eats or drinks the food object. live communication system. The viewing device is
further comprising a viewing signature input unit that receives an input of a signature for the food object from a viewer,
4. The live communication system according to any one of claims 1 to 3, wherein the food display unit synthesizes the sign with the food object and displays it. The distribution device
further comprising a delivery signature input unit that accepts an input of a signature for the food and drink object from a delivery person;
The transmission unit further transmits an image of the signature,
The receiving unit of the viewing device receives the image of the signature,
5. The live communication system according to any one of claims 1 to 4, wherein the food display unit synthesizes the image of the sign with the food object and displays it. The distribution device is simultaneously connected to a plurality of the viewing devices,
The viewing device further comprises a transmitting unit that transmits the ID of the food object designated by the viewer,
The distribution device further comprises a food selection unit that receives a distributor's selection of a food object specified in one or more viewing devices,
The transmission unit of the distribution device simultaneously transmits the selected food and drink objects to the plurality of viewing devices,
6. The viewing character display unit of each of the plurality of viewing devices displays a moving image of the character eating and drinking for the selected food object. live communication system. The food operation unit of the viewing device further receives a division operation from a viewer for the food object,
The food display unit separates the food object into a first partial object and a remaining object when the division operation is accepted, and moves and displays the first partial object according to an operation by the viewer. ,
When the distance between the position coordinates of the first partial object and the position coordinates of the character's mouth is equal to or less than a second threshold value, the viewing character display unit is configured such that the eating and drinking condition is satisfied and the character is displayed in the first portion. displaying an animation of eating and drinking an object and erasing the first partial object;
The food and drink display unit separates the remaining object into a second partial object and a new remaining object when an operation to divide the remaining object is accepted, and divides the remaining object into a second partial object and a new remaining object, and divides the second partial object according to an operation by the viewer. to move and display
When the distance between the position coordinates of the second partial object and the position coordinates of the character's mouth is equal to or less than a third threshold value, the viewing character display unit is configured such that when the distance between the position coordinates of the second partial object and the position coordinates of the character's mouth is equal to or less than a third threshold value, the eating and drinking condition is established again, and the character is displayed on the second partial object. 7. The live communication system according to any one of claims 1 to 6, wherein a moving image of eating and drinking of the partial object is displayed and the second partial object is erased. The food display unit of the viewing device moves and displays the food object along a predetermined trajectory when a viewer performs an operation of throwing the food object,
The viewing character display unit is configured such that when the distance between the position coordinates of the food object to be moved and displayed and the position coordinates of the mouth of the character is within a fourth threshold value, the food and drink condition is established, and the character displays the food and drink. 8. The live communication system according to any one of claims 1 to 7, wherein a moving image of eating and drinking an object is displayed. Including a distribution device operated by a distributor and a viewing device operated by a viewer,
The distribution device
an imaging unit that captures an image of a distributor;
a motion information generation unit that generates motion information indicating features of the motion of the distributor from the captured image of the distributor;
a distribution character display unit that displays a moving image of a character on the screen of the distribution device and moves the character according to the movement information;
a mouth identifying unit that identifies position coordinates of the character's mouth;
an eating/drinking determination unit that determines whether the eating/drinking condition is successful;
a first message obtaining unit that receives a first message from a distributor;
a transmission unit that transmits the operation information and the first message;
The transmission unit transmits an eating and drinking instruction as part of the operation information when the eating and drinking condition is satisfied,
The viewing device is
a receiver that receives the motion information and the first message;
a viewing character display unit that displays a moving image of a character on the screen of the viewing device and moves the character according to the motion information;
a first message output unit that outputs the first message in accordance with the movement of the character;
a food operation unit that receives a movement operation for a food object from a viewer;
a food display unit for moving and displaying the food object according to an operation by a viewer;
a food and drink identification unit that identifies position coordinates of the food and drink object;
a transmission unit that transmits the position coordinates of the food object,
The eating/drinking determining unit of the distribution device determines that the eating/drinking condition is established when a distance between the position coordinates of the mouth of the character and the position coordinates of the food object is equal to or less than a first threshold, wherein, when transmitting the action information, the transmission unit does not transmit moving image data of the character to the viewing device;
The live communication system, wherein the viewing character display unit of the viewing device displays a moving image of the character eating the food object when the eating instruction is received. A function to image the distributor,
A function of generating motion information indicating characteristics of the motion of the distributor from the captured image of the distributor;
a function of accepting a first message from a distributor;
a first program that causes a distribution device to perform a function of transmitting the operation information and the first message;
a function of receiving said operational information and said first message;
a function of displaying a moving image of a character on a screen of a viewing device and moving the character according to the movement information;
a function of outputting the first message in accordance with the movement of the character;
a function of receiving a movement operation for a food object from a viewer;
a function of moving and displaying the food object according to an operation by a viewer;
A live communication program characterized by comprising: a function of displaying a video of the character eating and drinking the food object when a predetermined eating and drinking condition is established;

Images