JP2020060898A - Information processor and program - Google Patents

Abstract

To provide an information processor capable of reproducing character's movement and voice while synchronizing each other with high accuracy.SOLUTION: An information processor 2 comprises: a storage part 34 storing an avatar model; a receiving part 31 that receives multiple first packets including a feature quantity of a subject and a first time stamp and multiple second packets including voice data and a second time stamp; an avatar output processing part 33 that extracts a feature quantity from each of the multiple first packets and makes a display unit display an avatar obtained by inputting the feature quantity into the avatar model; and a voice output processing part 35 that extracts voice data from each of the multiple second packets and makes a voice output part 44 output a voice based on the voice data. The avatar output processing part 33 acquires the second time stamp included in each of the multiple second packets and makes the display unit 43 display the avatar so that the voice and the avatar are synchronized based on the first time stamp and the second time stamp.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an information processing device and a program.

  Conventionally, a feature amount extracted from image data obtained by capturing an image of an individual's face is applied to an avatar model to display an avatar that reflects the facial expression of the individual's face. For example, in Patent Document 1, a facial recognition component that extracts data indicating a facial expression of an individual from a motion data stream and a facial recognition component that reflects the facial expression of an individual are reflected based on the data indicating the facial expression of the individual. And a render component for animating a character. With this system, it is possible to display a character on which a personal expression is reflected on the display.

Special table 2013-535051 gazette

  In the system described in Patent Document 1, in order to improve the realism, it is possible to further add voice data generated by an individual to a data stream and output a voice based on the voice data while displaying a character. However, it has been difficult to accurately reproduce the movement of the character and the voice in synchronization.

  Therefore, it is an object of the present invention to provide an information processing device capable of accurately synchronizing the motion of a character with the voice and reproducing the same.

  An information processing apparatus according to an aspect of the present invention includes a storage unit that stores an avatar model, a plurality of first packets that include a feature amount of a subject and a first time stamp, and a plurality of audio packets that include a second time stamp. A receiving unit that receives the second packet; an avatar output processing unit that displays the avatar obtained by extracting the feature amount from each of the plurality of first packets and inputting the feature amount into the avatar model; A voice output processing unit that extracts voice data from each of the second packets and outputs a voice based on the voice data to a voice output unit, and the avatar output processing unit is included in each of the plurality of second packets. The second avatar is acquired and the avatar is displayed on the display unit so that the voice and the avatar are synchronized based on the first time stamp and the second time stamp. Make, the information processing apparatus.

  According to this aspect, it is possible to accurately reproduce the movement of the character and the voice in synchronization.

  According to the present invention, it is possible to provide an information processing apparatus capable of accurately reproducing a motion of a character and a voice in synchronization with each other.

It is a network block diagram which shows an example of the avatar operation system 1 which concerns on 1st Embodiment. It is a block diagram which shows the functional structure of the user terminal 2A which concerns on 1st Embodiment. It is a schematic block diagram which shows the functional module of a transmission process part. It is a schematic block diagram which shows the functional module of a reception process part. It is a schematic block diagram which shows the data structure of a facial expression packet. It is a schematic block diagram which shows the data structure of a voice packet. 3 is a block diagram showing a functional configuration of the server device 3. FIG. It is an operation sequence diagram for explaining the initial setting process. It is an operation sequence diagram for explaining an avatar operation process. It is an operation | movement flowchart for demonstrating a facial expression packet generation process. 6 is an operational flow for explaining a voice packet generation process. It is an operational flow for explaining a voice output process. 9 is an operation flow for explaining avatar display processing.

  A preferred embodiment of the present invention will be described with reference to the accompanying drawings. (Note that, in each of the drawings, those denoted by the same reference numerals have the same or similar configurations.)

[First Embodiment]
(1) Configuration (1-1) Avatar operation system 1
FIG. 1 is a network configuration diagram showing an example of an avatar operation system 1 according to the first embodiment.

  As shown in FIG. 1, the avatar operation system 1 includes a user terminal 2A, a user terminal 2B, and a server device 3. The server device 3 is communicatively connected to each of the user terminals 2A and 2B via a communication network such as the Internet. Hereinafter, the user terminals 2A, 2B, etc. may be collectively referred to as “user terminal 2”.

  In the avatar operation system 1, one user terminal 2 transmits the feature amount of the subject to the other user terminal 2 and the feature amount received by the other user terminal 2 is applied to a predetermined avatar model, so that another An avatar (character) is output at the user terminal 2. Hereinafter, a device or a function that transmits the feature amount of the subject may be referred to as a “tracker”, and a device or a function that receives the feature amount of the subject and outputs an avatar may be referred to as a “viewer”.

(1-2) User Terminal FIG. 2 is a block diagram showing a functional configuration of the user terminal 2A according to the first embodiment. In the following, the user terminal 2A will be described as having both tracker and viewer functions, but the user terminal 2A may have only one of the tracker and viewer functions. Since the user terminal 2B also has the same configuration as the user terminal 2A, the description thereof will be omitted.

  The user terminal 2A is an example of an information processing device, and is configured by, for example, a computer including a memory such as a ROM and a RAM and a processor such as a CPU. Computer programs, data and the like are stored in the memory of the user terminal 2A. The computer program may be installed in the memory of the user terminal 2A from a computer-readable portable recording medium such as a CD-ROM, or may be downloaded from another information processing device via a communication network to the user terminal. It may be installed in the memory of 2A. The processor of the user terminal 2A centrally controls the operation of each unit of the user terminal 2A based on a computer program stored in the memory of the user terminal 2A.

  The user terminal 2A includes, for example, a user terminal setting processing unit 10, a transmission processing unit 20, a reception processing unit 30, an imaging unit 41, a voice input unit 42, a display unit 43, and a voice output unit 44. . The user terminal setting processing unit 10, the transmission processing unit 20, and the reception processing unit 30 are functional modules implemented by a processor included in the user terminal 2A based on a program stored in the memory of the user terminal 2A. . The imaging unit 41, the voice input unit 42, the display unit 43, and the voice output unit 44 may be provided outside the user terminal 2A, respectively.

  As will be described later, the user terminal setting processing unit 10 communicates with the server device 3 and performs various setting processing relating to the room ID, the client ID, and the like.

  The transmission processing unit 20 is an example of a functional module that realizes a tracker function, generates a facial expression packet (first packet) based on the moving image data generated by the imaging unit 41, and uses the facial expression packet for other information processing. Send to device. The transmission processing unit 20 also generates a voice packet (second packet) based on the voice data generated by the voice input unit 42, and transmits the voice packet to another information processing device.

  The reception processing unit 30 is an example of a functional module that realizes a viewer function, receives a facial expression packet from another information processing device, and displays an avatar on the display unit 43 based on the facial expression packet. Further, the reception processing unit 30 receives a voice packet from another information processing device and outputs a voice to the voice output unit 44 based on the voice packet.

  The imaging unit 41 is configured by, for example, a lens and an imaging element, and generates moving image data by imaging the subject and / or the surrounding environment of the subject. The voice input unit 42 is configured by, for example, a microphone, and generates voice data based on the voice emitted by the subject and / or the surrounding environment of the subject. The display unit 43 is composed of, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like, and displays an image or a moving image based on the image display data or the moving image display data. The audio output unit 44 is composed of, for example, a speaker and outputs audio based on audio data.

(1-2-1) Transmission Processing Unit FIG. 3 is a schematic configuration diagram showing functional modules of the transmission processing unit 20.

  As described above, the transmission processing unit 20 is an example of a functional module that realizes the tracker function. As shown in FIG. 3, the transmission processing unit 20 includes, for example, a facial expression packet processing unit 21, an audio packet processing unit 22, a moving image data buffer 23, and an audio data buffer 24.

  The facial expression packet processing unit 21 includes, for example, a partial moving image data extraction unit 211, a feature amount extraction unit 212, a first time stamp unit 213, and an facial expression packet generation unit 214.

  The partial moving image data extraction unit 211 extracts the partial moving image data from the moving image data generated by the image pickup unit 41 every predetermined first cycle. Here, the length of the first cycle is not particularly limited, but may be set arbitrarily according to the capacity of the facial expression packet, and may be, for example, 1 ms, 2 ms, 5 ms or the like.

  The feature amount extraction unit 212 extracts the feature amount of the subject from the partial moving image data. Here, the feature amount is information indicating the feature of the subject. The feature amount may be a feature amount of at least one part included in the subject, or may be a feature amount of a part included in the face of the subject. The feature amount may be, for example, a feature amount of the subject's forehead, eyebrows (right eyebrow, left eyebrow), eyes (left eye, right eye), nose, mouth, ear, hair, and the like.

  The first time stamp unit 213 generates a first time stamp. Here, the first time stamp is, for example, a time stamp corresponding to the time when the partial moving image data was extracted by the partial moving image data extraction unit 211. The first time stamp may be, for example, a time stamp corresponding to an arbitrary time (including the start time and the end time) between the start time and the end time of the partial moving image data.

  The facial expression packet generator 214 generates an facial expression packet (first packet) including the feature amount extracted by the feature amount extractor 212 and the first time stamp generated by the first time stamp unit 213. The facial expression packet generator 214 also transmits the generated facial expression packet to another information processing device via the communication network.

  The voice packet processing unit 22 includes, for example, a partial voice data extraction unit 221, a second time stamp unit 222, and a voice packet generation unit 223.

  The partial voice data extraction unit 221 extracts the partial voice data from the voice data generated by the voice input unit 42 every predetermined second cycle. Here, the length of the second cycle is not particularly limited, but may be set arbitrarily according to the capacity of the voice packet, and may be, for example, 30 ms, 60 ms, 100 ms or the like. The length of the first cycle and the length of the second cycle may be different.

  The second time stamp unit 222 generates a second time stamp. Here, the second time stamp is, for example, a time stamp corresponding to the time when the partial audio data was extracted by the partial audio data extracting unit 221.

  The voice packet generation unit 223 generates a voice packet (second packet) including the partial voice data extracted by the partial voice data extraction unit 221 and the second time stamp generated by the second time stamp unit 222.

(1-2-2) Reception Processing Unit FIG. 4 is a schematic configuration diagram showing functional modules of the reception processing unit 30.

  As described above, the reception processing unit 30 is an example of a functional module that realizes the viewer function. As shown in FIG. 4, the reception processing unit 30 includes, for example, a reception unit 31, a voice buffer processing unit 32A, a facial expression buffer processing unit 32B, an avatar output processing unit 33, an avatar model storage unit 34, and a voice output processing. And a part 35.

  The receiving unit 31 receives packets from another information processing device such as the server device 3 and rearranges the received packets based on, for example, the sequence ID.

  The voice buffer processing unit 32A and the facial expression buffer processing unit 32B are processing units configured by a part of the memory and the processor of the user terminal 2A. The voice buffer processing unit 32A buffers the voice packets rearranged by the receiving unit 31 by sequentially storing them, and discards the delayed voice packets. The facial expression buffer processing unit 32B buffers the facial expression packets rearranged by the receiving unit 31 by sequentially storing them, and discards the delayed facial expression packets.

  The avatar output processing unit 33 sets a playback motion by extracting a feature amount from a predetermined facial expression packet based on the facial expression packet stored in the facial expression buffer processing unit 32B, and sets the playback motion in the avatar model storage unit 34. By applying to the stored avatar model, avatar display data is generated. At this time, the avatar output processing unit 33 sets the reproduction motion based on the second time stamp received from the audio output processing unit 35 so that the avatar can be moved in synchronization with the audio. The method of setting the playback motion will be described later. Then, the avatar output processing unit 33 outputs the display data to the display unit 43 and causes the display unit 43 to display the avatar.

  The avatar model storage unit 34 is a part of the memory of the user terminal 2A and stores the avatar model. For example, when the feature amount of the subject extracted by the feature amount extraction unit 212 is input, the avatar model outputs display data for displaying an avatar in a mode corresponding to the feature amount. The avatar may be a character imitating a human being, an animal, a fictitious creature, an inanimate object, or any other form.

  The audio output processing unit 35 extracts audio data from the audio packet stored in the audio buffer processing unit 32A, and outputs audio from the audio output unit 44 based on the audio data. The voice output processing unit 35 also extracts a time stamp from the voice packet that outputs the voice and feeds the time stamp back to the facial expression buffer processing unit 32B.

(1-2-3) Data structure of packet <expression packet>
FIG. 5A is a schematic configuration diagram showing a data structure of a facial expression packet.

  The facial expression packet preferably has a data structure conforming to the UDP (User Datagram Protocol) in the avatar operation system 1 which requires a particularly timely operation of the avatar because the facial expression packet has a small load on communication processing. However, the facial expression packet may have a data structure conforming to any other protocol such as TCP (Transport Control Protocol).

  As shown in FIG. 5A, the facial expression packet includes, for example, a sequence ID, a packet type, a first time stamp, a client ID, a room ID, a data length, and facial expression data.

  The sequence ID is identification information for identifying the packet. The packet type is information indicating the type of packet, and includes “expression packet” and “voice packet”. The first time stamp is the first time stamp generated by the first time stamp unit 213. The client ID is data provided from the server device 3 in the initial setting process, and is identification information unique to the user terminal 2A as a client. The room ID is identification information for identifying the room set by the server device 3. The data length is information indicating the data length of the packet. The facial expression data is data indicating feature amounts of the subject such as the forehead, eyebrows (right eyebrow, left eyebrow), eyes (left eye, right eye), nose, mouth, ears, hair, etc., extracted by the feature amount extraction unit 212.

<Voice packet>
FIG. 5B is a schematic configuration diagram showing a data structure of a voice packet.

  As described above, the voice packet preferably has a data structure compliant with UDP, but not limited to UDP, it may have a data structure compliant with any other protocol such as TCP.

  As shown in FIG. 5B, the voice packet includes, for example, a sequence ID, a packet type, a second time stamp, a client ID, a room ID, a data length, and voice data. The sequence ID, the packet type, the client ID, the room ID, and the data length are the same as those included in the facial expression packet, and thus the description thereof will be omitted.

  The second time stamp is the second time stamp generated by the second time stamp unit 222. The audio data is the partial audio data extracted by the partial audio data extraction unit 221.

(1-3) Server Device FIG. 6 is a block diagram showing a functional configuration of the server device 3.

  The server device 3 is composed of, for example, a computer including a memory such as a ROM and a RAM and a processor such as a CPU. Computer programs, data and the like are stored in the memory of the server device 3. The computer program may be installed in the memory of the server device 3 from a computer-readable portable recording medium such as a CD-ROM, or may be downloaded from another information processing device via a communication network to the server device. 3 may be installed in the memory. The processor of the server device 3 centrally controls the operation of each part of the server device 3 based on a computer program stored in the memory of the server device 3.

  As shown in FIG. 6, the server device 3 includes, for example, a server device setting processing unit 50 and a packet processing unit 60. The server device setting processing unit 50 and the packet processing unit 60 are functional modules implemented by a processor included in the server device 3 based on a program stored in the memory of the server device 3.

  The server device setting processing unit 50 performs setting processing related to the room ID and the client ID with the user terminal 2.

  The packet processing unit 60 analyzes the packet (including the facial expression packet and the voice packet) received from the user terminal 2 and transmits the packet to the desired user terminal 2.

(2) Process of avatar operation system 1 (2-1) Initial setting process FIG. 7 is an operation sequence diagram for explaining the initial setting process.

(S101)
First, the user terminal setting processing unit 10 of the user terminal 2 transmits a room ID list request to the server device 3.

(S102)
Next, the server device setting processing unit 50 of the server device 3 acquires the room ID list from the internal or external memory of the server device 3 and transmits it to the user terminal 2.

(S103)
Next, the user terminal setting processing unit 10 transmits a client ID generation request to the server device 3. At this time, the user terminal setting processing unit 10 includes, for example, the room ID selected from the room ID list in the client ID generation request.

(S104)
Next, the server device setting processing unit 50 generates a client ID based on the client ID generation request. At this time, the server device setting processing unit 50 stores the generated client ID in the internal or external memory of the server device 3 in association with the selected room ID included in the client ID generation request.

(S105)
Next, the server device setting processing unit 50 transmits the generated client ID to the user terminal 2.

(S106)
Next, the user terminal setting processing unit 10 stores the client ID received from the server device 3 in the memory of the user terminal 2. This completes the initialization process.

(2-2) Avatar Operation Processing FIG. 8 is an operation sequence diagram for explaining the avatar operation processing.

  Here, in the avatar operation processing, one information processing device (tracker) transmits data including the feature amount of the subject to another information processing device (viewer), and causes the viewer to display the avatar corresponding to the feature amount. Operates the avatar. Hereinafter, a case where the user terminal 2A functions as a tracker and the user terminal 2B functions as a viewer will be described as an example.

(S201)
First, the transmission processing unit 20 of the user terminal 2A, which is a tracker, performs a packet generation process to generate a packet (expression packet or voice packet). Details of the packet generation process will be described later.

(S202)
Next, the transmission processing unit 20 of the user terminal 2A transmits the generated packet to the server device 3 via the communication network.

(S203)
Next, the packet processing unit 60 of the server device 3 analyzes the packet received from the user terminal 2A and identifies the transmission destination of the packet. Specifically, the packet processing unit 60 acquires the room ID and the client ID included in the packet, refers to the room ID list stored in the memory inside or outside the server device 3, and determines the destination of the packet. Identify.

(S204)
Next, the packet processing unit 60 of the server device 3 transmits the packet to the specified destination via the communication network according to the result of the packet analysis process. Here, it is assumed that the user terminal 2B that is a viewer is specified as the transmission destination.

(S205)
Next, the reception processing unit 30 of the user terminal 2B, which is a viewer, performs output processing based on the packet received from the server device 3. Details of the output processing will be described later. This is the end of the avatar operation process.

(2-2-1) Packet Generation Process Hereinafter, the packet generation process by the user terminal 2A will be described. The packet generation process includes a facial expression packet generation process for generating a facial expression packet and a voice packet generation process for generating a voice packet.

<Expression packet generation processing>
FIG. 9 is an operation flow diagram for explaining the facial expression packet generation process.

(S301)
First, the imaging unit 41 captures a subject to generate moving image data (a plurality of image data that is continuous in time series), and stores (buffers) the moving image data in the moving image data buffer 23.

(S302)
Next, the partial moving image data extraction unit 211 of the facial expression packet processing unit 21 determines whether the first cycle has elapsed. When it is determined that the first cycle has not elapsed (S302; No), the process returns to S301.

(S303)
When it is determined that the first period has elapsed (S302; Yes), the partial moving image data extraction unit 211 extracts the partial moving image data included in one first period from the moving image data stored in the moving image data buffer 23. To do.

(S304)
Next, the feature amount extraction unit 212 of the facial expression packet processing unit 21 extracts the feature amount of the subject included in the partial moving image data based on the extracted partial moving image data.

(S305)
Next, the first time stamp unit 213 of the facial expression packet processing unit 21 generates, for the extracted partial moving image data, a first time stamp according to the time when the partial moving image data was extracted by the partial moving image data extraction unit 211. To do. Here, the first time stamp may be, for example, an arbitrary time (including the start time and the end time) from the start time to the end time of the partial moving image data.

(S306)
Next, the facial expression packet generation unit 214 of the facial expression packet processing unit 21 generates a facial expression packet including the feature amount extracted by the feature amount extraction unit 212 and the first time stamp. This is the end of the facial expression packet generation process.

<Voice packet generation processing>
FIG. 10 is an operation flow for explaining the voice packet generation process.

(S401)
First, the voice input unit 42 generates voice data based on the voice emitted from the subject or the environment surrounding the subject, and stores (buffers) the voice data in the voice data buffer 24.

(S402)
Next, the partial voice data extraction unit 221 of the voice packet processing unit 22 determines whether the second cycle has elapsed. When it is determined that the second cycle has not elapsed (S402; No), the process returns to S401.

(S403)
When it is determined that the second cycle has elapsed (S402; Yes), the partial audio data extraction unit 221 extracts the partial audio data included in one second cycle from the audio data stored in the audio data buffer 24. To do.

(S404)
Next, the second time stamp unit 222 of the voice packet processing unit 22 generates a second time stamp for the extracted partial voice data according to the time when the partial voice data was extracted by the partial voice data extraction unit 221. To do. Here, the second time stamp may be, for example, an arbitrary time (including the start time and the end time) from the start time to the end time of the partial audio data.

(S405)
Next, the voice packet generator 223 of the voice packet processor 22 generates a voice packet including the partial voice data extracted by the partial voice data extractor 221 and the second time stamp. This completes the voice packet generation process.

(2-2-2) Output Process Hereinafter, the output process by the user terminal 2B will be described. The output process includes a voice output process and an avatar display process.

<Voice output processing>
FIG. 11 is an operation flow for explaining the audio output process.

(S501)
First, the reception unit 31 of the user terminal 2B receives the voice packet transmitted from the server device 3.

(S502)
Next, the reception unit 31 rearranges the received voice packets based on the sequence ID.

(S503)
Next, the voice buffer processing unit 32A buffers the voice packets rearranged by the receiving unit 31.

(S504)
Next, the audio buffer processing unit 32A determines whether or not a predetermined waiting time has elapsed. Here, the length of the predetermined waiting time is not particularly limited, but is, for example, 50 ms. When it is determined that the predetermined waiting time has not elapsed (S604; No), the process returns to S603 described above.

(S505)
When the audio buffer processing unit 32A determines that the predetermined waiting time has elapsed (S604; Yes), the audio data is extracted from the buffered audio packet, and the audio data is output to the audio output unit 44. , Output audio.

(S506)
Further, the audio output processing unit 35 extracts the second time stamp included in the reproduced audio packet and transmits the second time stamp to the facial expression buffer processing unit 32B. With that, the voice output process is completed.

<Avatar display processing>
FIG. 12 is an operation flow for explaining the avatar display process. In the following operation flow, the facial expression buffer processing unit 32B is assumed to intermittently receive the second time stamp from the voice output processing unit 35.

(S601)
First, the receiving unit 31 of the user terminal 2B receives the facial expression packet transmitted from the server device 3.

(S602)
Next, the reception unit 31 rearranges the received facial expression packets based on the sequence ID.

(S603)
Next, the facial expression buffer processing unit 32B buffers the facial expression packets rearranged by the receiving unit 31. At this time, if the facial expression packet including the first time stamp older than the received second time stamp is buffered, it is discarded as a delayed packet.

(S604)
Next, the avatar output processing unit 33 sets a reproduction motion. The process is executed based on the first time stamp and the second time stamp so that the avatar and the voice are synchronized. Specifically, starting from the facial expression data corresponding to the first time stamp of the past time closest to the time when the voice is output (the time indicated by the second time stamp), the future time closest to the elapsed time of the reproduction time. The reproduction motion is set so as to end with the facial expression data corresponding to the first time stamp of.

(S605)
Next, the avatar output processing unit 33 generates the display data related to the avatar by inputting the facial expression data (feature amount) included in the reproduced motion into the avatar model stored in the avatar model storage unit 34, and the display data concerned. Is output to the display unit 43 to display the avatar on the display unit 43. As described above, since the avatar output processing unit 33 sets the reproduction motion based on the first time stamp and the second time stamp, the avatar's voice and facial expression are reproduced in synchronization. This is the end of the avatar display process.

[Modification]
When each voice packet generated by a plurality of user terminals relates to the same room ID, the voice included in each voice packet may be combined to perform a mixing process of generating a synthesized voice. The mixing process may be executed by the server device 3 or the user terminal 2 that receives the packet, for example. When the server device 3 executes the mixing process, for example, the voice included in each voice packet is extracted, and the voices are synchronously combined based on the time stamp included in each voice packet. Then, a new voice packet including the synthesized voice may be generated, and the new voice packet may be transmitted to the desired user terminal 2 instead of each voice packet. When the user terminal 2 that receives the packet executes the mixing process, for example, the voice included in each received voice packet is extracted, and the voice is synchronously output based on the time stamp included in each voice packet.

  In the above-described embodiment, the feature amount based on the moving image data and the voice data are stored in different packets (a facial expression packet and a voice packet), respectively. However, the user terminal 2 may store the feature amount based on the moving image data and the audio data in a single packet, generate a single time stamp, and include this in the single packet. Good.

  In the above-described embodiment, the facial expression packet related to the first time stamp older than the second time stamp is discarded. In this regard, the avatar motion may be complemented before and after the discarded facial expression packet.

  The embodiments described above are for facilitating the understanding of the present invention and are not for limiting the interpretation of the present invention. Each element included in the embodiment and its arrangement, material, condition, shape, size and the like are not limited to the exemplified ones and can be appropriately changed. Further, the configurations shown in different embodiments can be partially replaced or combined.

DESCRIPTION OF SYMBOLS 1 ... Avatar operation system, 2 2A, 2B ... User terminal, 10 ... User terminal setting processing part, 20 ... Transmission processing part, 21 ... Facial expression packet processing part, 211 ... Partial moving image data extraction part, 212 ... Feature amount extraction part 213 ... First time stamp section, 214 ... Facial expression packet generation section, 22 ... Voice packet processing section, 221 ... Partial voice data extraction section, 222 ... Second time stamp section, 223 ... Voice packet generation section, 23 ... Video data Buffer, 24 ... Voice data buffer, 30 ... Reception processing unit, 31 ... Reception unit, 32A ... Voice buffer processing unit, 32B ... Facial expression buffer processing unit, 33 ... Avata output processing unit, 34 ... Avata model storage unit, 35 ... Voice Output processing unit, 41 ... Imaging unit, 42 ... Voice input unit, 43 ... Display unit, 44 ... Voice output unit, 3 ... Server device, 50 ... Server device setting processing unit 60 ... packet processing unit

Claims (7)

A partial moving image data extraction unit that extracts partial moving image data for each predetermined first period from the moving image data generated by the imaging unit,
A feature amount extraction unit that extracts the feature amount of the subject included in the partial moving image data,
A first time stamp unit that generates a first time stamp according to the time when the partial moving image data is extracted by the partial moving image data extraction unit;
A first packet generation unit that generates a first packet including the characteristic amount and the first time stamp and transmits the first packet;
A partial voice data extraction unit for extracting partial voice data from the voice data generated by the voice input unit at every predetermined second cycle;
A second time stamp unit for generating a second time stamp according to the time when the partial audio data is extracted by the partial audio data extracting unit;
A second packet generator that generates a second packet including the partial audio data and the second time stamp, and transmits the second packet;
An information processing apparatus including.   The information processing apparatus according to claim 1, wherein the feature amount is a feature amount of at least one part included in the subject.   The information processing apparatus according to claim 2, wherein the at least one part is a part included in the face of the subject.   The information processing apparatus according to claim 1, wherein the audio data includes audio data emitted from the subject. In a computer with a storage unit,
A partial moving image data extraction unit that extracts partial moving image data for each predetermined first period from the moving image data generated by the imaging unit,
A feature amount extraction unit that extracts the feature amount of the subject included in the partial moving image data,
A first time stamp unit that generates a first time stamp according to the time when the partial moving image data is extracted by the partial moving image data extraction unit;
A first packet generation unit that generates a first packet including the characteristic amount and the first time stamp and transmits the first packet;
A partial voice data extraction unit for extracting partial voice data from the voice data generated by the voice input unit at every predetermined second cycle;
A second time stamp unit for generating a second time stamp according to the time when the partial audio data is extracted by the partial audio data extracting unit;
A second packet generator that generates a second packet including the partial audio data and the second time stamp, and transmits the second packet;
A program for realizing. A storage unit that stores the avatar model,
A receiving unit for receiving a plurality of first packets including a feature amount of a subject and a first time stamp, and a plurality of second packets including audio data and a second time stamp;
An avatar output processing unit that displays the avatar obtained by extracting the feature amount from each of the plurality of first packets and inputting the feature amount into the avatar model;
A voice output processing unit that extracts the voice data from each of the plurality of second packets and outputs a voice based on the voice data to a voice output unit;
The avatar output processing unit acquires the second time stamps included in each of the plurality of second packets, and synchronizes the voice and the avatar based on the first time stamp and the second time stamp. An information processing device for displaying an avatar on the display unit. In a computer with a storage unit,
A receiving unit for receiving a plurality of first packets including a feature amount of a subject and a first time stamp, and a plurality of second packets including audio data and a second time stamp;
An avatar output processing unit that displays the avatar obtained by extracting the feature amount from each of the plurality of first packets and inputting the feature amount into an avatar model;
A program for realizing the voice output processing unit that extracts the voice data from each of the plurality of second packets and outputs the voice based on the voice data to a voice output unit,
The avatar output processing unit acquires the second time stamps included in each of the plurality of second packets, and synchronizes the voice and the avatar based on the first time stamp and the second time stamp. A program for displaying an avatar on the display unit.

Images