JP2023103317A - Live communication system using character - Google Patents

Abstract

To achieve, in particular, rich and smooth non-verbal communication in communication via characters.SOLUTION: A live communication system including a distribution terminal operated by a distributor and a viewing terminal operated by a viewer is assumed. The distribution terminal takes an image of a distributor and generates movement information indicating a characteristic of movement of the distributor. Also, the distribution terminal receives a voice message from the distributor and transmits the movement information and the voice message to the viewing terminal. The viewing terminal displays a character as a moving image, moves the character according to the movement information, and outputs the voice message in tune with the movement of the character.SELECTED DRAWING: Figure 7

Description

Patent Law Article 30, Paragraph 2 application filed On September 18, 2018, Apple Inc. at TestFlight (https://developer.apple.com/jp/testflight/).

Patent Law Article 30, Paragraph 2 application filed On October 4, 2018, https://itunes. apple. com/jp/app/iriam-%E3%82%A4%E3%83%AA%E3%82%A2%E3%83%A0/id1358057296? Published at mt=8.

Patent Act Article 30, Paragraph 2 application filed On October 4, 2018, https://play. google. com/store/apps/details? id=jp. co. duo7. vr. Published in iriam.

Patent Act Article 30, Paragraph 2 application filed On October 4, 2018, https://iriam. com/ and https://twitter.com/. Published at com/iriam_official.

The present invention relates to visual communication technology over communication networks.

Communication via SNS (Social Networking Service) has become popular with the spread of smartphones. On SNS, not only text but also various contents such as photos and videos can be easily shared. SNS is a tool that makes people feel a strong bond (connection) between people.

On the other hand, in the field of computer games, there are many games in which the user plays while conversing with a character (computer). Users often have a strong attachment to characters that do not exist in reality.

JP 2012-231281 A

A character is a virtual entity created by a game designer and does not have a sense of reality like a human being. On the other hand, characters have a unique charm due to their unreality. The inventors of the present invention thought that it would be possible to propose a new communication method by incorporating both character-specific charm and human-specific "real connection".

Specifically, a conversation between a distributor (human) and a viewer (human) is assumed. The distributor becomes a spokesperson for the character just like a voice actor in TV animation, and converses with viewers through the character. Assuming such communication, it is desirable that the character can express not only the words of the distributor but also the feelings and attitudes of the distributor in a rich and smooth manner.

The present invention is an invention completed based on the above recognition, and its main purpose is to provide a technique for realizing rich and smooth non-verbal communication, especially in communication via characters.

A live communication system in one aspect of the present invention includes a distribution terminal operated by a distributor and a viewing terminal operated by a viewer.
The distribution terminal includes an imaging unit that captures an image of the distributor, an action information generation unit that generates action information indicating characteristics of the action of the distributor from the captured image of the distributor, a first message acquisition unit that receives a first message from the distributor, and a transmission unit that transmits the action information and the first message.
The viewing terminal includes a receiving section for receiving motion information and a first message, a viewing character display section for displaying a moving image of a character on the screen of the viewing terminal and causing the character to move according to the motion information, and a first message output section for outputting the first message in accordance with the motion of the character.

A distribution terminal in one aspect of the present invention is connected to a plurality of viewing terminals operated by a plurality of viewers. The distribution terminal includes an imaging unit that captures an image of the distributor, an action information generation unit that generates action information indicating characteristics of the action of the distributor from the captured image of the distributor, a first message acquisition unit that receives a first message from the distributor, a transmission unit that transmits the action information and the first message, a reception unit that receives second messages from a plurality of viewing terminals, and a second message output unit that outputs the second message to the distributor.
The second message output unit sequentially displays the plurality of second messages received from the plurality of viewing terminals to the distributor.

According to the present invention, it becomes easier to realize rich communication through characters.

1 is a hardware configuration diagram of a live communication system; FIG. FIG. 2 is a schematic diagram for explaining a conversation service by a live communication system; FIG. 3 is a functional block diagram of a delivery terminal and server; FIG. 3 is a functional block diagram of a viewing terminal; FIG. It is a screen figure of a character selection screen. FIG. 10 is a screen diagram of a viewer conversation screen displayed on the viewing terminal; FIG. 11 is a screen diagram of a distributor conversation screen displayed on the distribution terminal; FIG. 10 is a sequence diagram of processing for displaying a character on both the delivery terminal and the viewing terminal; FIG. 10 is a screen diagram showing a character when the distributor leans his or her body; FIG. 11 is a screen diagram showing a character when the distributor changes the facial expression; FIG. 11 is a screen diagram showing a character when making a specific facial expression; FIG. 11 is a screen diagram showing a character waving his hand; FIG. 10 is a screen diagram when a viewer selects a decoration item; 4 is a data structure diagram of an item storage unit; FIG. FIG. 10 is a screen diagram when a character wears a decoration item; FIG. 10 is a sequence diagram showing a process of giving a decoration item to a character and wearing the decoration item by the character.

FIG. 1 is a hardware configuration diagram of a live communication system 100. As shown in FIG.
In the live communication system 100, a server 102 is connected to a plurality of distribution terminals 104a, 104b, . A smartphone is assumed as the distribution terminal 104 and the viewing terminal 108, but a general-purpose computer such as a laptop PC may also be used.

A user who uses the distribution terminal 104 is called a "distributor". A user using the viewing terminal 108 is called a "viewer". The distributor becomes the service provider, and the viewer becomes the service beneficiary.

FIG. 2 is a schematic diagram for explaining a conversation service provided by the live communication system 100. As shown in FIG.
In this embodiment, a large number of viewing terminals 108 can be simultaneously connected to one distribution terminal 104 . The following description assumes one-to-many communication. FIG. 2 assumes a situation in which three viewers 112a, 112b, and 112c (hereinafter collectively referred to as “viewers 112” when collectively referred to or when not distinguished) are having a group conversation with one distributor 110.

A character is associated with the distributor 110 in advance. A distribution character 114 a associated with the distributor 110 is displayed on the distribution terminal 104 . A viewing character 114 b is also displayed on the viewing terminal 108 . The delivery character 114a and the viewing character 114b are exactly the same character. Hereinafter, when the delivery character 114a and the viewing character 114b are not particularly distinguished, they are simply referred to as "character 114".

When the distributor 110 speaks in front of the distribution terminal 104 , the voice data is broadcast to the viewing terminal 108 . The distribution terminal 104 detects the movement of the distributor's mouth, and also moves the mouth of the distribution character 114a. In conjunction with this, the viewing terminal 108 also moves the mouth of the viewing character 114b. To the viewer 112, it seems as if the viewing character 114b is speaking of his/her own will, not the distributor. In other words, the distributor 110 becomes a spokesperson for the character 114 as a "voice actor". Hereinafter, the voice message input by the distributor 110 will be referred to as "first message".

The viewer 112 inputs a message (hereinafter referred to as a "second message") to the viewing terminal 108 in text. The second message is broadcast to distribution terminal 104 and viewing terminal 108 . The viewer 112 converses with the character 114 (distributor 110) by inputting the second message while listening to the viewing character 114b.

The individuality of the character 114 is formed by the basic settings of the character 114 (such as the appearance of the character 114 and the richness of facial expression changes) and the voice quality and conversational ability of the distributor 110 .

FIG. 3 is a functional block diagram of distribution terminal 104 and server 102 .
Each component of the distribution terminal 104 and the server 102 is implemented by hardware including computing units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, wired or wireless communication lines connecting them, and software stored in the storage device and supplying processing instructions to the computing units. A computer program may consist of a device driver, an operating system, various application programs located in their higher layers, and a library that provides common functions to these programs. Each block described below represents a functional block rather than a hardware configuration.
The same is true for the viewing terminal 108 .

(Distribution terminal 104)
Distribution terminal 104 includes user interface processing unit 116 , imaging unit 118 , data processing unit 120 , communication unit 122 and data storage unit 124 .
The communication unit 122 is in charge of communication processing. The imaging unit 118 images the distributor 110 . The user interface processing unit 116 receives operations from the distributor 110, and is in charge of user interface processing such as image display and audio output. The data storage unit 124 stores various data including image data of the character 114 . The data processing unit 120 executes various processes based on the data acquired by the communication unit 122 , the user interface processing unit 116 and the imaging unit 118 and the data stored in the data storage unit 124 . The data processing unit 120 also functions as an interface for the user interface processing unit 116 , imaging unit 118 , communication unit 122 and data storage unit 124 .

Communication unit 122 includes a transmitting unit 148 and a receiving unit 150 .

User interface processing unit 116 includes an input unit 126 that receives input from distributor 110 and an output unit 128 that outputs various types of information to distributor 110 .
Input unit 126 includes motion detection unit 130 , first message acquisition unit 132 , button selection unit 134 and wearing item selection unit 136 . The motion detection unit 130 detects facial expressions and body motions of the distributor 110 via the imaging unit 118 . The first message acquisition unit 132 acquires a first message (voice message). Button selection unit 134 accepts selection of an “expression button (described later)”. Wearing item selection unit 136 receives selection of a “decorative item (described later)” to be worn by character 114 .

Output unit 128 includes distribution character display unit 138 , second message output unit 140 and button display unit 142 .
The distribution character display unit 138 causes the distribution terminal 104 to display the distribution character 114a. The second message output unit 140 causes the distribution terminal 104 to display a second message (text message) from the viewer 112 . The button display unit 142 displays facial expression buttons.

Data processing unit 120 includes motion information generating unit 144 and specific facial expression determining unit 146 .
The motion information generation unit 144 generates “motion information” from the detection result of the motion detection unit 130 . The action information is numerical data indicating the motion state of the distributor 110, such as feature vectors indicating the facial expression of the distributor 110 (eg, eyebrow angle, numerical values indicating the degree of opening of the eyes and mouth, numerical values indicating the relative positional relationship between the distributor 110 and the distribution terminal 104 (front, back, left, and right). The specific facial expression determination unit 146 determines a "specific facial expression" that the character 114 should be given. .

(server 102)
Server 102 includes communication unit 152 , live control unit 154 and data storage unit 156 .
The communication unit 152 is in charge of communication processing. The data storage unit 156 stores various data. The live control unit 154 mediates and controls communication between the distribution terminal 104 and the viewing terminal 108 based on the data acquired by the communication unit 152 and the data stored in the data storage unit 156 . Live control unit 154 also functions as an interface for communication unit 152 and data storage unit 156 .

The data storage unit 156 includes a character storage unit 158 storing data of a plurality of characters 114 and an item storage unit 160 storing decoration items possessed by the characters 114 . The data structure of the item store 160 will be described later with reference to FIG.

FIG. 4 is a functional block diagram of the viewing terminal 108. As shown in FIG.
The viewing terminal 108 includes a user interface processing section 162 , a data processing section 166 , a communication section 164 and a data storage section 168 .
The communication unit 164 is in charge of communication processing. The user interface processing unit 162 receives operations from the viewer 112, and is in charge of user interface processing such as image display and audio output. A data storage unit 168 stores various data including image data of the character 114 . Both the distribution terminal 104 and the viewing terminal 108 store data of the character 114 respectively. The data processing unit 166 executes various processes based on the data acquired by the communication unit 164 and the user interface processing unit 162 and the data stored in the data storage unit 168 . Data processing unit 166 also functions as an interface for user interface processing unit 162 , communication unit 164 and data storage unit 168 .

Communication unit 122 includes a transmitting unit 186 and a receiving unit 190 .

The user interface processing unit 162 includes an input unit 170 that receives input from the viewer 112 and an output unit 172 that outputs various information to the viewer 112 .
Input unit 170 includes second message acquisition unit 174 , character selection unit 176 and provided item selection unit 178 . The second message acquisition unit 174 acquires a second message (text message). Character selection unit 176 accepts selection of character 114 to be a conversation target. The provided item selection unit 178 accepts selection of a decoration item to be presented to the character 114 .

Output unit 172 includes viewing character display unit 180 , first message output unit 182 and second message output unit 184 .
The viewing character display unit 180 causes the viewing terminal 108 to display the viewing character 114b. The first message output unit 182 outputs the first message from the distributor 110 by voice from the viewing terminal 108 . The second message output unit 184 causes the viewing terminal 108 to display the second message from the viewer 112 on the screen.

FIG. 5 is a screen diagram of the character selection screen 194. As shown in FIG.
At the start of the live communication service, the live control unit 154 of the server 102 causes the viewing terminal 108 to display the character selection screen 194 . The viewer 112 selects the viewing character 114 b (distributor 110 ) with which he/she wishes to converse on the character selection screen 194 displayed on the viewing terminal 108 . The live control unit 154 presents the character 114 corresponding to the distributor 110 in a conversationable state (standby state) on the character selection screen 194 as a conversationable character.

The character selection screen 194 displays a plurality of character buttons 192a, 192b (viewing characters 114b1, 114b2). The character selection unit 176 accepts selection of the viewing character 114b.
In the following description, it is assumed that the viewer 112 selects the character button 192a and enjoys conversation with the viewing character 114b1.

FIG. 6 is a screen diagram of a viewer conversation screen 196 displayed on the viewing terminal 108. As shown in FIG.
The viewing character display unit 180 of the viewing terminal 108 displays a moving image of the viewing character 114b1 selected as the chat partner. The viewer 112 enjoys conversation with the viewing character 114b1. The viewer 112 cannot see the distributor 110 . The first message output unit 182 of the viewing terminal 108 outputs the voice (first message) from the distributor 110 in charge of the viewing character 114b1. The viewer 112 inputs a message (second message) to the viewing character 114b1 by text. Server 102 shares the first message and the second message between distribution terminal 104 and viewing terminal 108 .

The second message output unit 184 causes the message area 198 to display the second message as text. Since one distributor 110 converses with a plurality of viewers 112 simultaneously, the server 102 distributes the second message of each viewer 112 to the distribution terminal 104 and the plurality of viewers 112 all at once. Viewers 112 can view not only their own secondary messages but also the secondary messages of other viewers 112 through message area 198 . The second message output unit 184 arranges and displays the second messages in order of reception.

The viewer 112 can present a decoration item to the viewing character 114b1 by selecting the present button 200 (described later in connection with FIG. 13 and subsequent figures).

FIG. 7 is a screen diagram of a distributor conversation screen 206 displayed on the distribution terminal 104. As shown in FIG.
The distributor conversation screen 206 shown in FIG. 7 is displayed on the distribution terminal 104 of the distributor 110 in charge of the distribution character 114a1 (corresponding to the viewing character 114b1). The delivery character display unit 138 displays the delivery character 114a1 as a moving image. The viewing character 114b1 and the delivery character 114a are the same character and have the same movement. The distributor 110 converses with a plurality of viewers 112 (the viewers 112 who have selected the viewing character 114b1 as a conversation partner) while confirming the movement of the distribution character 114a1, which is their alter ego, on the distribution terminal 104. The distributor 110 inputs the first message by voice toward the distribution terminal 104 . The transmission unit 148 transmits the first message to the server 102 , and the live control unit 154 of the server 102 distributes the first message to the plurality of viewing terminals 108 .

Second message output unit 140 displays second messages from viewer 112 in message area 198 in chronological order. The distributor 110 speaks to an unspecified number of viewers 112 while watching the second messages displayed one after another in the message area 198 . Camera 208 (imaging unit 118 ) of distribution terminal 104 captures image of distributor 110 . The motion detection unit 130 detects the expression and motion of the distributor 110 from the captured image, and the motion information generation unit 144 generates motion information. The delivery character display unit 138 determines the motion of the delivery character 114a1 based on the motion information. The viewing character 114b1 is also displayed as a moving image on the viewing terminal 108 based on the same motion information, although the details will be described later.

The button display unit 142 displays a plurality of expression buttons 202a (big laugh), 202b (full smile), 202c (displeased), and 202d (confused) (hereinafter collectively referred to as "expression buttons 202"). When the button selection unit 134 detects that the facial expression button 202 has been selected, the specific facial expression determination unit 146 selects the "specific facial expression" corresponding to the facial expression button 202 (described later with reference to FIG. 11). Also, when the distributor 110 touches the bye-bye area 204, the character 114 makes a wave motion (described later with reference to FIG. 12).

FIG. 8 is a sequence diagram of processing for displaying character 114 on both distribution terminal 104 and viewing terminal 108 .
The motion detector 130 of the distribution terminal 104 detects the motion of the distributor 110 (S10). The motion information generator 144 generates motion information (S12). The action information is numerical data indicating the facial expression of the distributor 110, the angle of the body, the perspective of the distributor 110 from the imaging unit 118, whether or not the bye-bye area 204 has been touched, the selection result of the facial expression button 202, and the like (S12).

The transmission unit 148 of the distribution terminal 104 transmits the motion information to each viewing terminal 108 via the server 102 (S14). The distribution character display unit 138 generates a moving image of the distribution character 114a1 based on the action information (S16), and causes the distribution terminal 104 to display the distribution character 114a1 as a moving image (S18).

On the other hand, the viewing character display unit 180 of the viewing terminal 108 also generates a moving image of the viewing character 114b1 based on the received action information (S20), and causes the viewing terminal 108 to display the viewing character 114b1 as a moving image (S22).

Since the delivery character 114a1 and the viewing character 114b1 are the same character and their motions are determined based on the same action information, the delivery character 114a1 and the viewing character 114b perform the same actions in the same form in the delivery terminal 104 and the viewing terminal 108, respectively.

The distribution terminal 104 does not transmit the moving image (character image) itself of the character 114 to the viewing terminal 108 , but transmits only action information defining the action of the character 114 . Since the amount of data to be transmitted is smaller than that of transmitting the moving image itself, it is easy to quickly link the movements of the distribution character 114a1 and the viewing character 114b1. In other words, the viewer 112 can enjoy comfortable conversation with the character 114 (distributor 110) because the difference between the time until the movement information is reflected as the movement of the delivery character 114a1 and the time until the same movement information is reflected as the movement of the viewing character 114b1 is small.

The first message (voice message) and the action information may be integrated and transmitted from the distribution terminal 104 to the viewing terminal 108 . In this case, the viewing character display unit 180 makes it difficult for the motion of the viewing character 114b1 to deviate from the voice message. Also, transmitting the motion information for determining the moving image instead of the moving image itself is effective in suppressing the communication load on the Internet 106 .

FIG. 9 is a screen diagram showing the character 114 when the distributor 110 leans over.
When the distributor 110 tilts his body to the left or right, the motion detection unit 130 detects the body tilt. The motion information generator 144 includes the tilt angle and the tilt direction in the motion information. The viewing character display unit 180 tilts the viewing character 114b1 according to the action information. The distribution character display unit 138 also tilts the distribution character 114a1 according to the same action information. In this way, when the distributor 110 tilts his body to the left or right, the characters 114 on both the distributor side and the viewer side tilt their bodies to match the distributor 110 . When the distributor 110 tilts his body back and forth, the character 114 similarly tilts his body back and forth. The motion detection unit 130 may recognize forward tilting and backward tilting of the distributor 110 from the size of the distributor 110 appearing in the captured image.

Generally, it is said that human psychology appears in the inclination of the body. For example, it is said that when people are interested in conversation, they tend to lean forward (the posture of listening), and when they are upset, their bodies sway from side to side. By reflecting the body movement of the distributor 110 on the character 114, the viewer 112 can sense the psychology of the character 114 (distributor 110).

FIG. 10 is a screen diagram showing the character 114 when the distributor 110 changes the expression.
When the distributor 110 changes his facial expression, the action detection unit 130 detects the facial expression of the distributor 110, specifically, the size of the eyes, the shape of the mouth, the angle of the eyebrows, the depth and direction of wrinkles between the eyebrows, and the like. The motion information generator 144 generates feature vectors that define facial expressions based on these detection values. The feature vector becomes part of the motion information. When the distributor 110 closes one eye (wink), the viewing character display unit 180 winks the viewing character 114b1. The same applies to the distribution character 114a1.

In addition, the facial expression of the distributor 110 can also be reflected on the character 114 by numerically expressing various facial expressions such as sticking out the tongue and raising the corners of the mouth with feature vectors (motion information). When the broadcaster 110 smiles, the character 114 also smiles. When the broadcaster 110 purses his mouth, the character 114 also purses his mouth. By linking the facial expression of the distributor 110 and the facial expression of the character 114, the viewer 112 can feel the presence of the character 114 even closer.

Note that the followability to the movement of the distributor 110 differs depending on the character 114 . For example, viewing character 114b1 may be a character with a rich facial expression, and even a slight change in the facial expression of distributor 110 may be greatly reflected. On the other hand, the viewing character 114b2 is a calm and composed character, and even if the distributor's 110 facial expression changes greatly, the facial expression may not change much.

More specifically, when the corner of the mouth of the broadcaster 110 increases by 5 points, the angle of the mouth of the viewing character 114b1 increases by 10 points, and the angle of the mouth of the viewing character 114b2 increases by only 2 points. There may be an active character 114 that can easily move its body, or there may be a graceful character 114 that does not stick out its tongue. As described above, the extent to which action information is reflected as the movement of the character 114 differs depending on the character 114 . The personality of the character 114 is formed by the basic settings of the character 114 and the personality of the distributor 110 (facial expression and speaking ability).

FIG. 11 is a screen diagram showing the character 114 when making a specific facial expression.
The distributor 110 selects a specific facial expression (a cartoon-like facial expression) using the facial expression button 202 . FIG. 11 shows a specific facial expression when the viewing character 114b1 closes his eyes. Since the character 114 is expressed as a two-dimensional cartoon, it can show various specific facial expressions that are not possible for humans, but are commonly expressed in cartoons. For example, to indicate that the character 114 is angry, "smoke" may be emitted from the head of the character 114 as a cartoon representation of "anger", "blue streaks" may be raised on the temples, the face of the character 114 may be bright red, or the eyebrows may be raised. To indicate that the character 114 is confused, the face of the character 114 may be made completely blue, or "sweat" may be displayed on the face of the character 114 . In this way, a specific facial expression may be expressed as a so-called "comic symbol". A facial expression that is impossible for a human being, such as tears or a burst of laughter, or a large facial expression that is not easily shown by a human being may be prepared as an image pattern of a specific facial expression. When the facial expression button 202 is selected, the specific facial expression determination unit 146 causes the transmission unit 148 to transmit the facial expression ID indicating the specific facial expression corresponding to the facial expression button 202 to the viewing terminal 108 . The viewing character display unit 180 changes the facial expression of the viewing character 114b1 to a specific facial expression in correspondence with the facial expression ID. The facial expression of the distribution character 114a1 also changes to a specific facial expression according to the facial expression ID.

The character 114 can realize a specific facial expression familiar to comics, which is not possible for humans (distributor 110), and thus can realize non-verbal communication that is impossible for humans.

FIG. 12 is a screen diagram showing the character 114 waving his hand.
When the distributor 110 touches the bye-bye area 204 (see FIG. 7), the motion information generator 144 includes information indicating a "hand gesture" (hereinafter referred to as "bye-bye information") in the motion information. When receiving the bye-bye information, the viewing character display unit 180 causes the viewing character 114b1 to perform a hand waving motion.

FIG. 13 is a screen diagram when the viewer 112 selects a decoration item.
When the viewer 112 touches the present button 200, the output unit 172 of the viewing terminal 108 causes the provided item selection area 210 to be displayed. The provided item selection area 210 displays a list of decorative items 212 owned by the viewer 112 or available for purchase. The decoration item 212 is an image representing clothes and accessories that the character 114 can wear.

The viewer 112 selects a decoration item 212 to be presented to the character 114 from the provided item selection area 210 . The provided item selection unit 178 detects selection of the decoration item 212 . At this time, the output unit 172 may display an animation in which the selected decoration item 212 is thrown into the character 114 as a “twist”. In addition, the viewing character display unit 180 may display a surprised or delighted expression of the viewing character 114b1 when the decoration item 212 is presented.

FIG. 14 is a data structure diagram of the item storage unit 160. As shown in FIG.
The item storage unit 160 of the server 102 manages the decoration items 212 owned by the character 114 (distributor 110). According to FIG. 14, the character 114 with character ID (CID)=C01 (hereinafter referred to as “character 114 (C01)”) has many decoration items 212 such as decoration item 212 (M01) and decoration item 212 (M02). In other words, the item storage section 160 is a “clothing case” for each character 114 .

When the character 114 (distributor 110) receives the decoration item 212 from the viewer 112, the live control unit 154 associates the decoration item 212 with the character 114 and registers it. According to FIG. 14, character 114 (C01) received decoration item 212 (M02) on Nov. 8, 2018 from viewer 112 (P13). Also, the character 114 (C01) is presented with a decoration item 212 (M01) and a decoration item 212 (M03) from the viewer 112 (P24). Therefore, it can be inferred that the viewer 112 (P24) has a strong desire to support the character 114 (C01).

Depending on which character 114 (distributor 110) receives what kind of decoration item 212 and how much from which viewer 112, the popularity and support level of the character 114 can be confirmed.

FIG. 15 is a screen view when the character 114 wears the decoration item 212. As shown in FIG.
The distributor 110 can select a decoration item 212 that the distributor 110 possesses and allow the character 114 to wear the decoration item 212 . If the character 114 wears the decoration item 212 that the viewer 112 presents among a large number of decoration items 212 presented by a large number of viewers 112 (fans), the viewer 112 can feel satisfaction as if his sincerity was communicated to the character 114.例文帳に追加The broadcaster 110 can also have fun meeting the expectations of the viewers 112 (fans) by having the character 114 wear the decorative item 212 . Also, the viewer 112 can enjoy changing the fashion of the favorite character 114 by presenting the decoration item 212 . The distributor 110 may enjoy coordination by wearing a plurality of decorative items 212 on the character 114 at the same time.

FIG. 16 is a sequence diagram showing the process of giving the decoration item 212 to the character 114 and wearing the decoration item 212 on the character 114 .
The viewer 112 selects the decoration item 212 in the provided item selection area 210 . The wearing item selection unit 136 detects the selection of the decoration item 212 (S30). The transmission unit 148 transmits the item ID specifying the decoration item 212 to the server 102 (S32). The communication unit 152 of the server 102 notifies the distribution terminal 104 that the decorative item 212 has been presented (S34). Also, the live control unit 154 registers the decoration item 212 in the item storage unit 160 (S36). Through the above process, the viewer 112 presents the decoration item 212 to the character 114 .

The distributor 110 selects the decoration item 212 that the character 114 wants to wear when starting the live communication. First, the distributor 110 acquires a possession list of the decoration items 212 from the server 102 (S38). Wearing item selection unit 136 of distribution terminal 104 receives selection of decoration item 212 from a wearing item selection screen (not shown) (S40). The transmission unit 148 of the distribution terminal 104 transmits the item ID of the selected decoration item 212 to the server 102 (S42). The server 102 notifies the item ID to all viewing terminals 108 (S44).

The distribution character display unit 138 generates an image in which the character distribution character 114a wears the decoration item 212 (S46), and displays it on the distribution terminal 104 (S48). The viewing character display unit 180 of the viewing terminal 108 also generates an image in which the character viewing character 114b wears the decoration item 212 (S50), and causes the delivery terminal 104 to display the image (S52).

<Summary>
The live communication system 100 has been described above based on the embodiments.
The distribution terminal 104 transmits motion information to the viewing terminal 108 instead of the moving image of the character 114 itself. The viewing terminal 108 locally generates a moving image of the character 114 based on the action information. Since the data amount of action information is much smaller than the data amount of moving images, the viewer 112 can enjoy highly real-time conversation with the character 114 (distributor 110).

Due to the high communication throughput, in the live communication system 100, one broadcaster 110 and multiple viewers 112 can enjoy conversations in parallel. Therefore, it is possible to create a place such as a talk session in which a plurality of viewers 112 enjoy one character 114 . Since the distributor 110 is also provided with topics from a large number of viewers 112, it is thought that it will be easier to grasp the stem of the story.

The character 114 can communicate not only with the voice of the distributor 110 but also with non-verbal communication such as body movements and facial expressions. The character 114 is essentially a virtual existence, but by reflecting the words and attitude of the distributor 110, it can exhibit a unique presence that makes it feel real even though it is virtual.

A viewer 112 freely selects a partner with whom he/she wants to chat from among a plurality of characters 114 (distributors 110). The broadcaster 110 can sense the popularity of the character 114 from the number of viewers 112 who nominated the character 114 that he or she is in charge of, or from the decoration items 212 provided by the viewers 112 . Such a popular mechanism is considered to motivate the distributor 110 to improve their service spirit and conversation ability.

A viewer 112 can show affection for the character 114 through a decorative item 212 . The broadcaster 110 can show appreciation to the viewers 112 by having the character 114 , which is an incarnation of himself, wear the decoration item 212 . Also, if the character 114 puts on the decoration item 212 presented by the viewer 112, the viewer 112 can enjoy a feeling of superiority as if he/she was selected from a large number of viewers 112 and a feeling of intimacy as if the character 114 was connected.

It should be noted that the present invention is not limited to the above-described embodiments and modifications, and can be embodied by modifying constituent elements without departing from the scope of the invention. Various inventions may be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments and modifications. Also, some components may be deleted from all the components shown in the above embodiments and modifications.

[Modification]
<Message>
In the present embodiment, it has been described that the first message is voice and the second message is text. Alternatively, the first message may be text and the second message may be voice. Also, a message may be conveyed by an icon image such as a stamp image or a pictogram.

<Hand movements>
In this embodiment, the action of the character 114 waving his hand has been described. As motions using hands, the character 114 may be made to take various poses such as a pose of putting both hands together to say "please", a pose of raising hands to indicate "stop", and a pose of "banzai" raising both hands.

In the present embodiment, it has been described that the character 114 waves when the distributor 110 touches the bye-bye area 204 . As a modification, the motion detection unit 130 may detect hand motions of the distributor 110 . The motion information generation unit 144 may include hand motions of the distributor 110 in the motion information. Then, the delivery character display unit 138 and the viewing character display unit 180 may move the hand of the character 114 in accordance with the action information.

<Body movement>
In the present embodiment, it has been described that the movement of the body of the distributor 110 is detected from the captured image. As a modification, the distributor 110 may instruct the action of the character 114 using an input interface such as a button. For example, various actions such as looking down, tilting the head, trembling, jumping, etc. may be instructed from the input interface.

<Specific expression>
In the present embodiment, it has been described that the distributor 110 causes the facial expression button 202 to display a specific facial expression. As a modification, the distributor 110 may indicate a specific facial expression using his or her own facial expression. For example, when the distributor 110 strongly closes both eyes, the specific facial expression determination unit 146 may determine that the facial expression condition for executing the "specific facial expression with closed eyes" as shown in FIG. 11 is satisfied. At this time, the facial expression ID indicating the "closed eyes specific facial expression" is transmitted from the distribution terminal 104 to the viewing terminal 108, and the viewing character display unit 180 changes the facial expression of the character 114 to the "closed eyes specific facial expression". A variety of facial expression conditions may be set, such as keeping the eyes closed for one second or longer, or opening the mouth to a size greater than a threshold, and a specific facial expression may be associated with each of a plurality of facial expression conditions. By detecting not only the facial expression but also the pose of the distributor 110, the character 114 may be made to take a pose that he is good at (fine pose).

<Calibration>
The data processing unit 120 of the distribution terminal 104 may further include a “calibration execution unit” that performs facial expression correction (calibration) of the distributor 110 . Some of the distributors 110 have rich facial expressions, while others have poor facial expressions. When the distributor 110 is newly registered, the calibration execution unit may issue various operation instructions such as ``speak out loud'' and ``shake your body'', and the action detection unit 130 may detect the movement of the distributor 110 in response to the action instructions. For example, if the distributor 110a opens his mouth wide when uttering "ah", the calibration execution unit may set the character 114 to open its mouth wide only when the distributor 110a opens his mouth wide to some extent. Also, if the distributor 110b does not open his mouth wide, the character 114 may be set to open his mouth wide even when the distributor 110b opens his mouth slightly. According to such a control method, it is possible to absorb the difference in movement for each distributor 110 and appropriately change the character 114 according to the movement of the distributor 110 .

<Others>
When the decoration item 212 is provided to the character 114 by the viewer 112, the character 114 may wear the decoration item 212 immediately. In the present embodiment, the server 102 relays communication between the distribution terminal 104 and the viewing terminal 108, but the distribution terminal 104 and the viewing terminal 108 may communicate directly via peer-to-peer (P2P). Also, the second message acquisition unit 174 of the distribution terminal 104 may include a voice changer for converting the voice quality of the distributor 110 and transmitting it to the viewing terminal 108 .

100 live communication system, 102 server, 104 distribution terminal, 106 Internet, 108 viewing terminal, 110 distributor, 112 viewer, 114 character, 114a distribution character, 114b viewing character, 116 user interface processing unit, 118 imaging unit, 120 data processing unit, 122 communication unit, 124 data storage unit, 126 input unit, 128 output unit, 130 motion detection Section 132 First Message Acquisition Section 134 Button Selection Section 136 Wearing Item Selection Section 138 Distributed Character Display Section 140 Second Message Output Section 142 Button Display Section 144 Action Information Generation Section 146 Specific Facial Expression Judgment Section 148 Transmission Section 150 Reception Section 152 Communication Section 154 Live Control Section 156 Data Storage Section 158 Character Storage Section 160 Item Storage Section 16 2 user interface processing unit 164 communication unit 166 data processing unit 168 data storage unit 170 input unit 172 output unit 174 second message acquisition unit 176 character selection unit 178 provided item selection unit 180 viewing character display unit 182 first message output unit 184 second message output unit 186 transmission unit 190 reception unit 192 character button 194 character selection screen 19 6 viewer conversation screen, 198 message area, 200 present button, 202 expression button, 204 bye-bye area, 206 broadcaster conversation screen, 208 camera, 210 provided item selection area, 212 decoration item

Claims (12)

Including the distribution terminal operated by the distributor and the viewing terminal operated by the viewer,
The delivery terminal is
an imaging unit that captures an image of a distributor;
a motion information generation unit that generates motion information indicating features of the motion of the distributor from the captured image of the distributor;
a first message obtaining unit that receives a first message from a distributor;
a transmission unit that transmits the operation information and the first message;
The viewing terminal is
a receiver that receives the motion information and the first message;
a viewing character display unit that displays a moving image of a character on the screen of the viewing terminal and moves the character according to the motion information;
and a first message output unit that outputs the first message in accordance with the action of the character. The delivery terminal is
a distribution character display unit that displays a moving image of the character on the screen of the distribution terminal and moves the character according to the movement information;
wherein, when transmitting the motion information, the transmitting unit does not transmit moving image data of the character to the viewing terminal;
2. The live communication system according to claim 1, wherein said distribution terminal and said viewing terminal independently display a moving image of a character based on said motion information. the distribution terminal is simultaneously connected to a plurality of the viewing terminals;
the transmission unit of the distribution terminal simultaneously transmits the operation information and the first message to the plurality of viewing terminals;
The viewing terminal further comprises a second message acquisition unit that receives a second message from the viewer,
The delivery terminal is
a receiver that receives the second message;
a second message output unit that outputs the second message to the distributor,
3. The live communication system according to claim 1, wherein the second message output unit sequentially displays a plurality of second messages received from a plurality of viewing terminals to the distributor. The motion information generating unit of the delivery terminal sets, as the motion information, a feature vector indicating the facial expression of the deliverer from the face image of the deliverer,
4. The live communication system according to any one of claims 1 to 3, wherein the viewing character display unit of the viewing terminal changes the facial expression of the character based on the feature vector. 5. The live communication system according to claim 4, wherein the viewing character display unit of the viewing terminal changes the facial expression of the character to a predetermined specific facial expression when the feature vector indicating the facial expression of the broadcaster satisfies a predetermined facial expression condition. The motion information generating unit of the distribution terminal sets the relative positions of the distributor and the distribution terminal as the motion information by detecting forward, backward, and lateral movements of the distributor, and
6. The live communication system according to claim 1, wherein the viewing character display unit of the viewing terminal changes both or one of the position and size of the character on the screen of the viewing terminal based on the relative position. The distribution terminal further comprises a button display unit for displaying a plurality of facial expression buttons on the distribution terminal,
The transmission unit of the distribution terminal further transmits to the viewing terminal a facial expression ID that identifies the facial expression button selected from the plurality of facial expression buttons,
The receiving unit of the viewing terminal receives the facial expression ID,
7. The live communication system according to claim 1, wherein the viewing character display unit of the viewing terminal changes the facial expression of the character to a specific facial expression corresponding to the facial expression ID. The viewing terminal is
further comprising a character selection unit that accepts selection of one of the plurality of characters from the viewer;
8. The live communication system according to any one of claims 1 to 7, wherein the viewing character display unit of the viewing terminal changes the movement amount of the character according to the movement information in accordance with the selected character. The motion information generating unit of the distribution terminal further sets hand motion as the motion information,
9. The live communication system according to any one of claims 1 to 8, wherein the viewing character display unit of the viewing terminal causes the character to make a hand-waving motion when motion information indicating a hand motion is received. The viewing terminal is
a provision item selection unit that receives a selection of a decoration item from a viewer;
a transmission unit that transmits the selected decoration item to the distribution terminal;
The delivery terminal is
a receiving unit that receives an item ID that identifies a decoration item from the viewing terminal;
a wearing item selection unit that receives selection of a decoration item by the distributor from among the plurality of decoration items received from the viewer;
the transmission unit of the distribution terminal transmits an item ID of the selected decoration item to the viewing terminal;
The receiving unit of the viewing terminal receives an item ID from the viewing terminal,
3. The live communication system according to claim 1, wherein the viewing character display unit of the viewing terminal displays the character after adding the decoration item corresponding to the received item ID as a costume of the character. connected to multiple viewing terminals operated by multiple viewers,
an imaging unit that captures an image of a distributor;
a motion information generation unit that generates motion information indicating features of the motion of the distributor from the captured image of the distributor;
a first message obtaining unit that receives a first message from a distributor;
a transmitter configured to transmit the motion information and the first message;
a receiving unit that receives a second message from the plurality of viewing terminals;
a second message output unit that outputs the second message to the distributor,
The distribution terminal, wherein the second message output unit sequentially displays the plurality of second messages received from the plurality of viewing terminals to the distributor. A function to image the distributor,
A function of generating motion information indicating characteristics of the motion of the distributor from the captured image of the distributor;
a function of accepting a first message from a distributor;
a first program that causes a distribution terminal to perform a function of transmitting the operation information and the first message;
a function of receiving said operational information and said first message;
a function of displaying a moving image of a character on the screen of the viewing terminal and moving the character according to the movement information;
A live communication program, comprising: a function of outputting the first message in accordance with the action of the character;

Images