JPH11102450A - Device and method for transmitting and receiving data stream showing three-dimensional virtual space - Google Patents

Abstract

PROBLEM TO BE SOLVED: To decrease the quantity of data to be transferred by transferring sound data which accompany the movements of a skeleton structure via a sound stream together with a synchronizing method, a compressing method or the respective inter-stream correspondence information. SOLUTION: A three-dimensional virtual space generation part 4 is connected to a 3DCG data stream receiving part 3, via a data bus and receives the input of a 3DCG data stream DS from the part 3. Then the part 4 generates 3DCG showing a three-dimensional space, based on the stream DS and also generates a sound data stream Sa. A sound generation part 5 is connected to the part 4 via a signal line and obtains the input of the stream Sa to reproduce the voices. In other words, a device 100 simultaneously transmits plural 3DCG data streams which construct an encoding data stream DS1 via a 3DCG data stream transmitting part 1. A client part PCC generates a 3D animation, including the voices based on the stream DS which are received via a sending part TR.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a system for transmitting and receiving a data stream that forms a three-dimensional virtual space represented by three-dimensional computer graphics (hereinafter referred to as "3DCG"). The present invention relates to an apparatus and a method for transmitting and receiving a data stream representing a three-dimensional skeleton structure for efficiently transmitting and receiving a natural movement of a character having a complicated skeleton structure.

[0002]

2. Description of the Related Art In recent years, as a field of application of 3DCG,
Virtual malls (Virtual Mall) and electronic commerce (Electronic Commerc) on the Internet such as W (World-Wide Web)
e) and related homepages are attracting attention. In particular, due to the rapid development of the Internet, an environment for easily handling relatively high-quality 3DCG such as games and movies in a home is being prepared.

As shown in FIG. 8, in the WWW or the like, a plurality of client PCCs such as personal computers are connected to a machine PCS called a server such as a personal computer or a workstation via the Internet.
1 to PCCn are connected. Then, the client can obtain information transmitted from the server by reconstructing the image, sound, text, arrangement information, and the like downloaded from the server as needed.
This communication between the server and the client usually uses TCP
/ IP (Transmission Control Protocol / Internet Pro
tocol).

Conventionally, data provided from the server side is:
It mainly consisted of only text data and image data, but recently, VRML (Virtual Reality Modeling Lan
guage) and VRML browsers have been standardized, and there is a movement to transfer 3DCG data itself such as shapes and scenes.

[0005] The VRML will be briefly described. H
Like TML (Hypertext Markup Language)
In the conventional data format mainly composed of images and texts, enormous transfer time and transfer cost are required to transfer image data, especially moving image data. For this reason, current devices have restrictions on network traffic.
On the other hand, in 3DCG, all of the viewpoint information and light source information, including the shape, are processed using three-dimensional data. However, due to recent advances in computer graphics (hereinafter referred to as “CG”) technology, The image quality of the image created by is rapidly improving. Therefore, transferring the CG data as it is is much more efficient in terms of data amount.

Normally, CG data can express an image of the same quality as the conventional image data with a data amount of 1/100 or less of the conventional image data.
It has a compression ratio of 00 or more. Therefore, CG data, especially 3
There is a move to use DCG data as a standard. As one of the efforts for standardization of such 3DCG data transfer, the aforementioned VRML has been proposed (VR
ML Ver2.0). VRML Ver 2.0 defines shape data called primitives, various light source data, viewpoint data, texture data, and other data formats, and a method of specifying rigid body movement. Such a data format is referred to as a VR format (V
RF).

On the other hand, an animation technique for generating an image in real time has recently attracted attention in the conventional CG field. By using this real-time animation technique, a device has been devised to reproduce the real movement of the CG character mainly in commercials and movies. As one of them, a complex shape such as a human is represented by a skeletal structure, and a complex movement can be reproduced naturally by defining a movement amount of a joint of the skeleton which changes every moment.

[0008]

In the 3DCG modeling language used in the conventional Internet environment centered on such VRML, it is possible to set an operation in real time in a shape having a complicated structure such as a human. Did not. Further, a plurality of users share a virtual three-dimensional space in a virtual space via a network, and cannot interactively operate each other's incarnation (Avatar) in the space. Further, in reality, voice or music cannot be transmitted or received in real time in a manner synchronized with the operation of an incarnation (hereinafter, referred to as an “avatar”) in a virtual three-dimensional space. Therefore, the present invention makes it possible to set an operation in real time in a shape having a complicated structure such as a human through a communication path, and to allow a plurality of users to operate in a virtual space via a network. Sharing a dimensional space, it is possible to interact with each other's avatars in that space. In addition, virtual 3
Provided are an apparatus and method for transmitting or receiving a data stream representing a three-dimensional virtual space so as to allow voice or music to be transmitted or received in real time in a manner synchronized with the operation of an avatar in the three-dimensional space. The purpose is to:

[0009]

In order to achieve this object, the present invention provides a transmitting unit, a receiving unit, and a transmitting unit for connecting the transmitting unit and the receiving unit. A shape stream, which is a data stream indicating the shape of the skeletal structure in the three-dimensional computer graphics, an operation stream, which is a data stream indicating motion, and a sound stream, which is a data stream indicating sound synchronized with the motion, are transmitted to the receiving unit. The receiving unit is a device for transmitting and receiving a data stream representing a three-dimensional virtual space, such that the transmitted operation stream and the sound stream are synchronized to generate three-dimensional computer graphics,
The transmitting unit has a transmitting unit that simultaneously transmits the shape stream, the operation stream, and a sound stream synchronized with the operation stream as a plurality of streams in accordance with a format of a predetermined data stream. The plurality of streams are transmitted from the transmission unit to the reception unit, and the reception unit receives the plurality of streams transmitted from the transmission unit via the transmission unit, and transmits the received streams. A data stream receiving unit that determines a type and a format, receives a plurality of streams, and performs necessary restoration processing on the streams according to each of the determined streams; and a shape stream or a shape stream among the streams received by the receiving unit. The shape of the skeleton structure is stored in the three-dimensional virtual space using the three-dimensional shape data read in advance. Or a three-dimensional virtual space generating means for generating a three-dimensional virtual space and further moving the shape of the skeletal structure using a motion stream, and in synchronism with the movement of the skeletal structure when the stream contains sound data. An apparatus for transmitting and receiving a data stream representing a three-dimensional virtual space, comprising: a sound reproducing unit that reproduces sound data.

Further, the present invention comprises a transmitting section, a receiving section, and transmission means for connecting the transmitting section and the receiving section bidirectionally, and the transmitting section has a skeleton structure in three-dimensional computer graphics. A shape stream, which is a data stream indicating a shape, an operation stream, which is a data stream indicating a motion, and a sound stream, which is a data stream indicating a sound synchronized with the motion, are transmitted to a receiving unit. An apparatus for transmitting and receiving a data stream representing a three-dimensional virtual space so as to generate three-dimensional computer graphics by synchronizing the stream and the sound stream, wherein the transmitting unit responds to a request from the receiving unit The shape stream, the motion stream, and the sound stream synchronized with the motion stream are determined in advance in real time. Transmitting means for simultaneously transmitting the plurality of streams as a plurality of streams according to the format of the data stream, wherein the transmitting means interactively transmits the plurality of streams from the transmitting section to the receiving section, and the receiving section comprises Receiving the plurality of streams transmitted from the transmitting unit via the transmission means, determining the type and format of the received stream, receiving the plurality of streams in real time as necessary, and determining each of the streams; A data stream receiving means for applying necessary restoration processing to the stream in accordance with the stream, and a three-dimensional virtual space using a shape stream or pre-read three-dimensional shape data among the streams received by the receiving unit. Generate the shape of the skeletal structure or a three-dimensional virtual space, and further use the motion stream to A three-dimensional virtual space generating means for moving the shape, and a sound reproducing means for reproducing the sound data in synchronism with the movement of the skeletal structure when the stream includes sound data. Device for transmitting and receiving a data stream representing

The present invention also provides a shape stream, which is a data stream indicating the shape of a skeletal structure in three-dimensional computer graphics, a motion stream, which is a data stream indicating motion, and a sound indicating sound synchronized with the motion. A stream is transmitted to a receiving unit, and the receiving unit transmits and receives a data stream representing a three-dimensional virtual space so that the transmitted operation stream and the sound stream are synchronized to generate three-dimensional computer graphics. A transmitting step of simultaneously transmitting the shape stream, the operation stream, and the sound stream synchronized with the operation stream as a plurality of streams according to a predetermined data stream format; and Unit to connect the plurality of streams to the transmitting unit. A transmitting step of transmitting to the receiving unit, receiving the transmitted plurality of streams, determining the type and format of the received stream, receiving the plurality of streams, and responding to each determined stream. A data stream receiving step of performing a stream restoration process, and a skeleton structure shape or a three-dimensional virtual shape in a three-dimensional virtual space using a shape stream or pre-read three-dimensional shape data of the stream received by the receiving unit. A three-dimensional virtual space generating step of generating a space and further moving the shape of the skeletal structure using the motion stream; and, if the stream contains sound data, reproducing the sound data in synchronization with the movement of the skeletal structure. Transmitting and receiving a data stream representing a three-dimensional virtual space, comprising: The law provides.

The present invention further provides a shape stream, which is a data stream indicating the shape of a skeleton structure in three-dimensional computer graphics, a motion stream, which is a data stream indicating motion, and a sound indicating sound synchronized with the motion. A stream is transmitted to a receiving unit, and the receiving unit transmits and receives a data stream representing a three-dimensional virtual space so that the transmitted operation stream and the sound stream are synchronized to generate three-dimensional computer graphics. Then, in response to a request from the receiving unit, the shape stream, the operation stream, and the sound stream synchronized with the operation stream are simultaneously converted into a plurality of streams according to a predetermined data stream format in real time. Transmitting step of transmitting, the transmitting unit and the receiving unit Bi-directionally connecting and transmitting the plurality of streams interactively from the transmitting unit to the receiving unit; and receiving the plurality of transmitted streams and the type and format of the received streams. A data stream receiving step of receiving a plurality of streams in real time and performing a stream restoration process according to each of the determined streams;
A shape of a skeletal structure or a three-dimensional virtual space is generated in a three-dimensional virtual space using a shape stream or three-dimensional shape data read in advance among the streams received by the receiving unit, and a skeleton is generated using an operation stream. A step of generating a three-dimensional virtual space that moves the shape of the structure, and a step of reproducing the sound data in synchronization with the movement of the skeletal structure when the stream includes sound data. A method for transmitting and receiving a data stream representing a dimensional virtual space is provided.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below in detail with reference to the accompanying drawings. First, FIG. 1 shows a minimum configuration example of an apparatus 100 for transmitting and receiving a data stream representing a three-dimensional virtual space according to an embodiment of the present invention. The device 100 includes a server unit PCS on the transmission side, a client unit PCC on the reception side, and a transmission unit TR connecting the two. The server unit PCS generates a 3DCG data stream representing a three-dimensional virtual space according to a user's instruction normally input by a stream authoring unit provided outside. The operation of the stream authoring means will be described later.

The server PCS encodes (format converts) the 3DCG data stream DS forming a three-dimensional virtual space including the avatar (incarnation) of the user,
Generate an encoded data stream DS1. In addition,
4 and 5 and 11, the 3DCG data stream DS and the encoded data stream DS
One format VRF will be described.

The transmission unit TR bidirectionally connects the server unit PCS on the transmission side and the client unit PCC on the reception side, transmits the encoded data stream DS1 from the server unit PCS to the client unit PCC, and inputs from the client unit PCC. The changed data relating to the 3DCG data to be transmitted is transmitted to the server PCS. That is, the server unit PCS reformat-converts the input encoded data stream DS1, decodes (reverse-format converts) the 3DCG data stream DS, and reproduces a 3DCG image representing a three-dimensional virtual space. In addition,
Similarly, when the user operates his / her avatar on the server PCS to cause a change in the three-dimensional virtual space, the server PCS generates and transmits image change data DM representing the change. The data is transmitted to the client PCC via the unit TR.

Normally, the server section PCS and the client section PCC are each realized mainly by software in a computer device represented by a workstation or a personal computer. Server part PCS is 3
The transmission unit TR includes a 3D CG data stream transmission unit 2. And
The client unit PCC includes a 3DCG data stream receiving unit 3, a three-dimensional virtual space generating unit 4, and a sound generating unit 5. The three-dimensional virtual space generating unit 4 is connected to the 3DCG data stream receiving unit 3 via a data bus, and receives a 3DCG data stream D from the 3DCG data stream receiving unit 3.
S input is received.

The three-dimensional virtual space generator 4 generates a three-dimensional virtual space based on the 3DCG data stream DS.
A DCG is generated, and a sound data stream Sa is generated. The sound generation unit 5 is composed of a signal line and a three-dimensional virtual space generation unit 4.
To receive the input of the sound data stream Sa and reproduce the sound.

That is, the apparatus 100 transmits the encoded data stream DS from the 3DCG data stream transmitting unit 1.
1 are transmitted at the same time. The client PCC generates a 3D animation including audio based on the plurality of 3DCG data streams received via the transmission unit TR.

In this example, as described above, the minimum configuration of the apparatus 100, that is, the transmission unit T
Although the example of the single-user network of FIG. 1 in which one client unit PCC is connected via R has been described, as shown in the multi-user network of FIG. 8, a plurality of client units PCCn are connected via the transmission unit TR.
(N is a natural number).

FIG. 2 shows an example of the configuration of the 3DCG data stream transmitting section 1 configured in the server section PCS. 3
The DCG data stream transmission unit 1 uses 3
It is connected to the DCG data stream transmission unit 2 and exchanges request signals Sr generated by both the client PCC and the server PCS. The 3DCG data stream transmission unit 1 generates an avatar skeleton data stream Ss
, A skeleton stream transmitting unit 11 for generating an avatar motion data stream Sm, a sound stream transmitting unit 13 for generating an avatar sound data stream Sa, and an avatar data stream S
A stream conversion unit 14 performs an encoding process for converting s, Sm, and Sa into a predetermined format, and a state management unit 15 performs a state management process in the apparatus 100 such as a client ID and an avatar state.

The stream converter 14 is bidirectionally connected to both the skeletal structure stream transmitter 11 and the 3D CG data stream transmitter 2, and encodes the avatar skeletal data stream Ss into the skeletal data stream Ss1 to transmit the 3D CG data stream. Supply to section 2. Similarly, the stream conversion unit 14 includes the motion stream transmission unit 1
The avatar motion data stream Sm is also bidirectionally connected to the 2D and 3D CG data stream transmitting unit 2 to encode the avatar's motion data stream Sm1 and supply it to the 3D CG data stream transmitting unit 2. Further, the stream converter 14 is connected to both the sound stream transmitter 13 and the 3DCG data stream transmitter 2, and encodes the sound data stream Sa into a sound data stream Sa1,
The data is supplied to the 3DCG data stream transmission unit 2.

The 3DCG data stream transmitting section 1 receives the skeleton data stream Ss from the skeleton structure stream transmitting section 11 and the motion data stream Sm from the motion stream transmitting section 12 in response to the request signal Sr transmitted from the client PCC. And a sound data stream Sa is generated from the sound stream transmission unit 13, and each is output to the stream conversion unit 14. The skeleton data stream Ss includes hierarchical data that defines the hierarchical relationship of the skeleton structure, and information indicating the initial state of the hierarchical data, the state of the joints of the skeleton, the shape associated with the skeleton, and the correspondence relationship with the skeleton structure. Motion data stream Sm
Contains motion information that is the basis of the skeletal structure. The sound data stream Sa includes information representing a sound synchronized with the motion information represented by the motion data stream Sm.

The stream converter 14 adds unique identification information to each of the data streams Ss, Sm, and Sa to convert the skeletal data stream Ss1, motion data stream Sm1, and sound data stream Sa1 of different formats. Generate and output to the 3DCG data stream transmission unit 2. As a format of the data streams Ss1, Sm1, and Sa1, as described above, a VRML format or a format obtained by extending the VRML format is preferably used.
The VRML format is VRML Ver.
It is standardized as 2.0. CG shape data that does not need to be transmitted in a stream is stored in a normal HTTP (Hypert
ext Markup Language) and may be transferred in advance via a network or read from a file on a local machine.

FIG. 3 shows a detailed structure of the 3DCG data stream receiving section 3. The 3DCG data stream receiving unit 3, like the 3DCG data stream transmitting unit 1,
It is bidirectionally connected to the 3DCG data stream transmission unit 2 and exchanges request signals Sr. The 3DCG data stream receiving unit 3 includes a stream converting unit 31, a skeleton structure stream receiving unit 32, a motion stream receiving unit 33, and a sound stream receiving unit 34.

The stream converter 31 is bidirectionally connected to the 3DCG data stream transmitter 2 and the skeletal structure stream receiver 32, and the skeletal data stream input from the 3DCG data stream transmitter 1 via the 3DCG data stream transmitter 2 Ss1 is reformatted, decoded into the original skeleton data stream Ss, and supplied to the skeleton structure stream receiving unit 32. Similarly, the stream conversion unit 31 includes the 3DCG data stream transmission unit 2
The motion data stream Sm1 is bidirectionally connected to the motion stream receiving unit 33, decodes the motion data stream Sm1 into a motion data stream Sm, and supplies the motion data stream Sm to the motion stream receiving unit 33.
Further, the stream converter 31 is bidirectionally connected to the 3DCG data stream transmitter 2 and the sound stream receiver 34, and converts the sound data stream Sa1 to the sound data stream S.
a, and supplies it to the sound stream receiving unit 34.

That is, the skeleton data stream Ss1, the motion data stream Sm1, and the sound data stream Sa1 transmitted from the 3DCG data stream transmission unit 1.
Is input to the stream converter 31 via the 3DCG data stream transmitter 2. Stream converter 3
1 is the input streams Ss1, Sm1, and S
a1 is converted back to the original format if necessary,
Skeletal data stream Ss, motion data stream S
m, and a sound data stream Sa, and the skeleton structure stream receiving unit 32 and the motion stream receiving unit 3
3 and the sound stream receiving unit 34. As described above, the encoded 3DCG data stream DS1 transmitted from the 3DCG data stream transmission unit 2 is decoded into the 3DCG data stream DS, and each of the encoded data is transmitted as a separate skeletal structure, motion, and sound data stream Ss, Sm, and Sa. Receive.

Hereinafter, the operation of the apparatus 100 will be briefly described. The 3DCG data stream transmission unit 1 of the server unit PCS, in response to request signals Sr from the plurality of client units PCC, performs hierarchical data defining the hierarchical relationship of the skeleton structure of the avatar, the initial state of this hierarchical data, and the joint state of the skeleton. The constraint condition, the shape data associated with the skeleton, and the data on the correspondence relationship with the skeleton structure are transmitted as a stream. The stream converter 14 performs format conversion and compression processing of the 3DCG data stream DS to generate an encoded data stream DS1. When the 3DCG data stream DS is a composite type, the motion or sound is decomposed into separate streams, and the skeleton data stream Ss is compressed as necessary.

The avatar skeleton data is disclosed in Japanese Patent Application No. 9-100453 by the present applicant (April 1997).
(Submitted on March 17), the avatar skeleton data structure will be briefly described later with reference to FIGS. 9 and 10.

Referring to FIG. 4 and FIG.
2 shows two examples of a standard format of the DCG data stream DS. First, in the first example, as shown in FIG. 4, each stream, for example, a skeleton data stream Ss is composed of a main header MH, a subheader SH, and a data body DB. Order, compression / non-compression, stream start time (or frame number), end time (or frame number), presence / absence of loop, time stamp (used for synchronization)
Presence / absence, pitch for playback, etc. are written. The subheader SH and the data body DB form a pair, and the size of the data and the like are specified in the subheader SH. In the following description, unless otherwise specified, the skeleton data stream Ss, the motion data stream Sm, and the sound data stream Sa are all represented in this format.

Further, in the second example shown in FIG. 5, the operation and the sound, and the operation and the operation are sequentially arranged in one stream in the composite type data stream. In this case, the data arranged sequentially is usually data that needs to be synchronized at the time of reproduction.
The operation and sound data are appropriately divided into units called packets, which are appropriately finely divided into a length necessary for synchronization and arranged. This data arrangement is specified in the main header.

Next, FIGS. 6 and 7 show examples of stream packets that are the above-described finely divided packets.
Fig. 6 shows an animation stream (Fig. 6) and an operation stream (Fig. 7).

The data structures of the avatar skeleton data stream Ss and the motion data stream Sm will be described below with reference to FIGS. 9, 10 and 11. First, FIG. 9 (FIG. 9) shows a basic skeleton structure. For the sake of simplicity, a skeletal structure representing the avatar's elbow will be described. That is, the two links LK1 and LK2 are connected by the joint J in a hierarchical relationship. For example, if the link LK1 is the upper arm, the link LK1 has the upper and lower hierarchical structure in which the link LK2 representing the parent and the forearm is the child.

Next, FIG. 10 shows a general hierarchical structure of the entire avatar. In the figure, each of the blocks indicated by a straight line corresponds to the joint J, and each of the blocks indicated by a broken line indicates the leading end of each hierarchical system. That is, in this example, the avatar is connected to three primary systems R1, R2, and R3 from the root R of the main body. The primary line R1 has a right foot base JR1-1 and a right knee JR1-
2. Right ankle JR1-3 and right ankle JR1-4, the depth of which is deeper, and the right foot system R connected to the avatar route R
1. Similarly, JR2-1, JR2-2,
The left foot system R2 is configured by JR2-3 and JR2-4. However, since the primary line R3 represents the upper body, the three secondary lines R31, R3 from the spine root JR3-1.
2, and R33. Secondary system R31
Is right collar JR31-1, right shoulder JR31-2, right elbow JR3
The right arm system includes a hierarchy of 1-3, right wrist JR31-4, and right hand JR31-5. Similarly, the secondary system R32 includes a left collar JR32-1, a left shoulder JR32-2, a left elbow JR32-3, a left wrist JR32-4, and a left hand JR32-.
A left arm system composed of five layers is configured. Secondary system R
Reference numeral 33 denotes a head system composed of a hierarchy of a neck JR 33-1 and a crown JR 33-2.

[0034] If necessary, the hierarchy of each system can be further deepened and subdivided to represent an avatar having a degree of freedom of action closer to that of a real human. For example, a person who gives a finger to the left and right hand ends JR31-5 and JR32-5 only needs to normally give 15 joints each corresponding to a joint of the hand.

FIG. 11 shows a format in which motion data for the above skeleton structure (initial value) is shown in the motion data stream Sm. The basic motion data is
Is rotation angle time-series data indicating the angle of rotation about the X, Y, and Z axes in the local coordinate system over time. Further, it is time-series data of three elements of X, Y, and Z of the position coordinates of the route R in the world coordinate system and three elements of the direction vector of the position of the route R. The data shown in this way is repeated by the number of frames required in time series,
It is recorded in the motion data stream Sm.

Referring again to FIG. 3, client unit PC
The processing in C will be described. Stream receiving unit 3
In 2, 33, and 34, the respective streams Ss, Sm, and Sa transmitted from the 3DCG data stream transmission unit 2 and subjected to stream conversion are received. Normally, the required number of processes (or threads) serving as reception ports for receiving each stream are simultaneously generated in a required number in the computer PCC on the receiving side. This multiple process
It communicates with the status management unit 15 on the transmission side of the 3DCG data stream transmission unit 1 shown in FIG.

Further, the stream receiving units 32, 33, and 34 buffer the stream in the shared memory (or the shared buffer) as needed through a plurality of processes serving as reception ports. The buffered streams are sent to the three-dimensional virtual space generation unit 4 while synchronizing the time between the streams with the data streams Ss and Ss.
Pass m and Sa. Also, as described above, the normal HT
The skeleton structure stream is reproduced based on the CG shape data previously transferred via the network according to the TP or read from a file on the local machine. Further, in synchronization with the transmitted skeletal structure stream, the motion stream Sm or the sound stream Sa is transferred to the three-dimensional virtual space generation unit 4 in real time. In the three-dimensional virtual space generation unit 4, based on the stream input from the 3DCG data stream reception unit 3, the 3DCG data stream transmission unit 1 of the server PCS generates the 3DCG data stream transmission unit 1.
Play G. That is, the three-dimensional virtual space generation unit 4
The three-dimensional virtual space is reproduced using a VRML-only browser that generates a three-dimensional virtual space. In the above-described mechanism, the data stream temporarily stored in the shared buffer is transmitted from each buffer to the above-described stream receiving units 32, 33, and 34 according to a trigger from the three-dimensional virtual space unit 4.
, And then to the specified node in the three-dimensional virtual space defined by VRML. In the VRML node, usually, a three-dimensional shape, an object to emit sound, and a joint shape of a skeleton are designated in advance.

FIGS. 12 and 13 show three-dimensional scenes described in VRML Ver. 2.0. FIG. 12 shows a VRML description and FIG. 13 shows a scene structure corresponding to the description.
Is shown in In FIG. 13, a portion represented by a circle is called a node, and a node described by a shape corresponds to each object existing in a three-dimensional space. Also, the node represented by Transform corresponds to movement, rotation, etc., and 3
Hierarchical information of a dimensional shape is expressed in VRML by a nest relationship separated by parentheses. In FIG. 12, the Translatio
An example is shown in which when the parameters after n and Rotation are changed, two Shapes move and rotate according to the values. According to the basic mechanism described above, the position of each joint of the skeletal structure is changed by changing the Rotation value of each joint of the specific three-dimensional skeletal structure received by the motion stream receiving unit 33 shown in FIG. 3 in time series. Is done. That is, a human having a skeletal structure in the three-dimensional virtual space moves in real time according to the data of the motion stream. As for the sound stream, a node of an object corresponding to a sound source is set as a shape in a three-dimensional scene represented by a scene tree structure as shown in FIG. When the audio stream is triggered by a user (usually using a VRML sensor node), the audio stream receiving unit 34 shown in FIG.
3 is transmitted to the Shape node corresponding to the sound source shown in FIG.

The reproduction of the received sound data is passed directly to the sound generation unit 5 and reproduced in a normal personal computer or the like. The sound reproducing unit 5 includes a digital signal processor (DSP) for sound reproduction and a sound source.
In addition, generally, a format such as Wav, Aiff, or MIDI is used for sound data, but PCM data may be used for sound or the like. Regarding the synchronization of the motion stream and the sound stream, the internal time is usually managed in the three-dimensional virtual space generation unit 4, and the synchronization can usually be achieved by specifying the start time and the end time. .

FIG. 14 shows the CG in the three-dimensional virtual space generation unit 4.
The reproduction of the stream is shown in time series. As described above, the sound stream is generated by the sound generation unit 5, and the start time t0 and the end time
Set t1 for playback. The motion stream varies depending on the performance of the receiving personal computer and the complexity of the three-dimensional scene. For this reason, the operation can be sent or hastened for the sound. To eliminate this and synchronize the sound with the operation, a time stamp TS (indicated by △) is inserted in advance in the data of the separate stream. The use of the time stamp makes it easy to match the operation reproduction to the time stamp position of the sound stream with respect to the sound reproduction.
In other words, at any corresponding time stamp position, if the playback of the motion is delayed by judging from the time stamp position,
The three-dimensional virtual space generation unit 4 performs a skipping process for reproducing the motion up to the time stamp position thereafter. Much more
If the playback of the motion is finished at the same time stamp position,
The three-dimensional virtual space generation unit 4 performs a process of repeating the motion reproduction until the sound reproduction ends up to the time stamp position.
Alternatively, as shown in the lower part of FIG. 14, if a composite type is used as the format of the CG data stream, and further divided into the minimum packets to be synchronized, the sound and the operation can be synchronized between the packets and reproduced. .

The basic operation of the stream authoring means described above with reference to FIG. 1 will be briefly described.
The stream authoring means includes a skeleton data stream Ss having an arbitrary data length and a motion data stream S
m and the audio data stream Sa are cut into arbitrary lengths necessary for synchronization and converted into the packet format shown in FIGS. The data length of the skeleton data stream Ss depends on conditions such as the number of joints and degrees of freedom, the data length of the motion data stream Sm depends on the duration of the operation, and the data length of the sound data stream Sa depends on the sound generation time. Decide. Then, as shown in FIG. 14, time stamp information necessary for synchronization is added to each stream.
Further, if necessary, each stream is connected to generate a new stream. In this way, the edited stream is registered in the server PCS.

The apparatus 100 according to the present invention will be described below with reference to the flowcharts shown in FIGS.
Will be described. FIG. 15 shows sharing of a three-dimensional space in a so-called multi-user network in which a server PCS and a plurality of client PCs in the apparatus 100 are connected. Here, as shown in FIG. 8, when a plurality of client units PCC2 to PCCn are connected to a single server unit PCS and a three-dimensional space is shared with each other, a new client unit PC is newly added.
The case where CC1 is connected to the server PCS will be described.

In step S1, the client PCC1
Requests the ID from the server PCS. Step S3
Then, the server PCS issues an ID to the client PCC1. At this point, the client PCC
1 and the avatar of the client PCC1 appear in the three-dimensional virtual space of the server PCS. In step S5,
It is determined whether or not the user has operated the avatar in the client PCC1. With the detection of avatar operation,
Proceed to the next step S7. Until then, this step S5
Is repeatedly determined to be NO, and the avatar operation presence / absence determination is continued.

In step S7, the client PCC1
3D converts the image change data DM of the motion data stream Sm including the avatar movement information based on the user's operation into 3D.
It is generated by the CG data stream receiving unit 3 and transmitted to the 3DCG data stream transmitting unit 1 of the server PCS via the transmission unit TR. The server PCS
Identifies the user, that is, the client PCC1, by the ID transmitted together with the image change data DM.
In step S9, it is determined whether or not the image change data DM and ID transmitted from the client unit PCC1 need to be transmitted to any of the other client units PCC2 to PCCn. This determination is usually made based on the request signal Sr from the client PCC1, but the client PCC1 may instruct the server PCS in advance at step S1.

In step S11, the client PCC
1, other currently connected client unit PCC2
To PCCn to get a response. Needless to say, such information may be determined in advance in both the client PCC and the server PCS. In step S13, after the motion information of the avatar of the client PCC1 is transmitted to the client answered by the client PCC1 in step S11 using the image change data DM, the process returns to step S5, and the process is repeated until the client PCC1 leaves the network. The processing from S5 to S13 is repeated.

Next, a method of transmitting and receiving three-dimensional virtual space data in a so-called single user network in which only one client PCC1 is connected to the server PCS in the apparatus 100 will be described with reference to FIG.

First, in step S21, the reading destination of the 3DCG data stream DS representing the three-dimensional virtual space is determined. For example, if the 3DCG data stream DS is present in the client PCC1, it is determined as NO and the process proceeds to step S23. In step S23, 3DC
The G data stream DS is read from a storage device (not shown) of the client PCC1 and supplied to the three-dimensional virtual space generation unit 4 of the client PCC1 to generate a three-dimensional virtual space image. On the other hand, in step S21, Y
If it is determined to be ES, the process proceeds to step S25. In step S25, the 3DCG data stream DS1 is
The three-dimensional virtual space generation unit 4 is read from a storage device (not shown) of the server unit PCS and is read from the storage unit (not shown) of the client unit PCC1.
And a three-dimensional virtual space image is generated.

In step S27, the client PCC
1 requests an ID from the server PCS. Step S2
At 9, the server PCS issues an ID to the client PCC1.

In step S31, the client PCC
The destination from which the 1 reads the 3D CG data stream DS of the avatar is determined. For example, if the avatar data is in the client PCC1, the client PCC1
Since the request signal Sr does not make an avatar data read request from the server unit PCS, the determination is NO and the process proceeds to step S33. In step S33, the 3D CG data stream DS of the avatar is read from the storage device of the client PCC1 and the client PCC1 is read.
Is supplied to the three-dimensional virtual space generation unit 4. And 3
An avatar image is generated in the three-dimensional virtual space based on the DCG data stream DS. On the other hand, step S31
If YES is determined, the process proceeds to step S35.
In step S35, based on the request signal Sr from the client PCC1, the server PCS encodes the avatar skeleton data stream Ss and the motion data stream Sm as needed (format conversion).
And transfers it to the client PCC1.

In step S37, the client PCC
1 is a skeletal data stream Ss (Ss1) and a motion data stream Sm transferred from the server PCS.
(Sm1) is received. In step S39, it is determined whether the received data streams Ss (Ss1) and Sm (Sm1) need to be decoded (inverted format conversion). If the inverse conversion is not required, step S41
Proceed to.

In step S41, the data stream S
The s and Sm are supplied to the three-dimensional virtual space generation unit 4 via the skeleton structure stream reception unit 32 and the motion stream reception unit 33 without being subjected to the inverse format conversion by the stream conversion unit 31. On the other hand, if you need an inverse transformation,
YES is determined and the process proceeds to step S43. Step S
At 43, the data streams Ss1 and Sm1 are converted by the stream converter 31 into data streams Ss and Sm1.
m (reverse format conversion), decompose into predetermined units by the skeleton structure stream receiving unit 32 and the motion stream receiving unit 33, and then supply them to the three-dimensional virtual space generating unit 4. Then, the three-dimensional virtual space generation unit 4 generates a three-dimensional virtual space image reflecting the change in the motion of the avatar.

Next, referring to FIG. 17, a description will be given of the synchronization processing of the 3DCG data stream DS at the time of 3DCG reproduction by the 3D virtual space generation unit 4 of the client PCC.

In step S51, the internal clock (TimeSensor) is reset in response to the issuance of the 3D CG scene drawing start command from the three-dimensional virtual space generation unit 4. That is, in response to a request from the server PCS, each client PCC simultaneously resets its own internal clock (TimeSensor).

In step S53, the client PCC
At 1, it is determined whether the user has operated the avatar. When the avatar operation is detected, the process proceeds to the next step S55. Until then, the determination in step S53 is NO, and the avatar operation presence / absence determination is continued.

In step S55, the skeleton data stream Ss, the motion data stream Sm, and the sound data stream Sa described in the VRML format are read from the primary memory (not shown) of the three-dimensional virtual space generation unit 4.

In step S57, it is determined whether or not synchronous reproduction is necessary based on the description in the main header MH in each data stream. NO if synchronous playback is not required
And the process proceeds to step S59. Step S59
Then, based on the internal clock (TimeSensor), the motion data stream Sm is reproduced at the time designated in the VRML format, and the sound generator 5 reproduces the sound data stream Sa. That is, as soon as the sound data format is sent from the three-dimensional virtual space generation unit 4 to the sound generation unit 5 together with the start signal, the sound reproduction is started.

On the other hand, when synchronous reproduction is necessary, YES is determined in the step S57, and the process proceeds to a step S61. In step S61, the 3DCG data stream D to be reproduced
It is determined whether or not the content of S is the sound data stream Sa. If YES in step S61, step S61
Go to 63. In step S63, the sound generation unit 5 feeds back the time stamp information shown in FIG. 14 to the three-dimensional virtual space generation unit 4.

On the other hand, when the contents of the 3DCG data stream DS are CG data, NO is determined in the step S61, and the process proceeds to a step S65. Step S65
Then, the time stamp information fed back from the sound generator 5 is input to the three-dimensional virtual space generator 4 as a synchronization signal. In the case of a voice packet, the end signal is input to the three-dimensional virtual space generation unit 4 as a synchronization signal.

In step S67, it is determined whether the reading of the CG data has been completed based on the time stamp. If completed, the determination is YES, and the process proceeds to step S69. In step S69, CG
The reproduction is immediately stopped, and the process proceeds to the next step S71. The timing of CG reproduction is based on the internal clock of VRML.
(TimeSensor) based on management.

In step S71, the server unit P of the 3DCG data stream DS for further CG drawing reproduction
The presence or absence of an input from the CS to the three-dimensional virtual space generation unit 4 is determined. If there is an input, it is determined as YES, and the process returns to step S51. On the other hand, when there is no input, the determination in step S71 is continued.

On the other hand, if NO is determined in the step S67, the process proceeds to a step S73. In step S73, after the CG reproduction is extended to the synchronization time indicated by the next time stamp, the process proceeds to step S71. In this way, the reproduction streams can be synchronized.

FIGS. 15 to 17 show a method for transmitting and receiving a data stream representing a three-dimensional virtual space according to the present invention.
Steps of the flowchart of FIG. A program for causing a computer to execute the method of the present invention may be recorded on a computer-readable recording medium. For this purpose, for example, as shown in FIG.
An Internet-compatible personal computer PC and a floppy disk FD readable by the personal computer PC may be used as the computer and the recording medium, respectively. Further, the recording medium is not limited to a floppy disk, but may be various optical disks, semiconductor memories, or the like. Also, various programs can be obtained by a personal computer PC via the Internet.

Although the present description has been made on the assumption of a two-way communication system on the assumption of the Internet, transmission and reception of satellite broadcasting, terrestrial broadcasting, and the like can be performed in exactly the same manner by using the system together with the Internet. In the case where bidirectionality is not required, it goes without saying that the present invention can be used only for broadcasting systems such as satellite broadcasting and terrestrial broadcasting.

[0064]

As is apparent from the above description, according to the present invention, shape data of a skeleton structure like a human is transferred as a stream, and the basic operation of the skeleton is sent as a motion stream. By transmitting sound data accompanying the movement of the skeletal structure as a sound stream together with a synchronization method, a compression method, or corresponding information between the streams, a smooth movement of the character and sound information synchronized with the movement are transmitted or received. It is possible to greatly reduce the amount of data at the time of transfer. In addition, a transmission / reception system based on a network enables interactive transmission / reception according to a user's request.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a minimum configuration of an apparatus for transmitting and receiving a data stream representing a three-dimensional virtual space according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating details of a 3DCG data stream transmission unit illustrated in FIG. 1;

FIG. 3 is a block diagram illustrating details of a 3DCG data stream receiving unit illustrated in FIG. 1;

FIG. 4 is a diagram showing an example of a format of a 3DCG data stream according to the present invention.

FIG. 5 is a diagram showing another example of the format of the 3DCG data stream according to the present invention.

FIG. 6 is a schematic diagram showing a packet of an animation stream that is a 3DCG data stream according to the present invention.

FIG. 7 is a schematic diagram showing packets of an operation stream that is a 3DCG data stream according to the present invention.

FIG. 8 is a schematic diagram showing an apparatus of the present invention in which a plurality of clients are connected to a single server.

FIG. 9 is an explanatory diagram showing a basic skeleton structure in a 3DCG data stream according to the present invention.

FIG. 10 is a schematic diagram illustrating a general hierarchical structure of data representing a skeleton structure of an avatar in a 3DCG data stream according to the present invention.

11 is a schematic diagram showing a format showing a data stream representing a motion with respect to the skeleton structure (initial value) shown in FIG.

FIG. 12 is an explanatory diagram showing a description example of a description according to VRML of a three-dimensional scene according to the present invention.

FIG. 13 is a schematic diagram showing a scene structure corresponding to the description example of FIG. 12;

FIG. 14 is an explanatory diagram showing an example of a 3DCG data stream synchronization method according to the present invention.

FIG. 15 is a flowchart illustrating an operation of sharing a three-dimensional space in the multi-user network according to the present invention of FIG. 8;

FIG. 16 is a flowchart illustrating a method for transmitting and receiving three-dimensional virtual space data in the single-user network according to the present invention of FIG. 1;

FIG. 17 shows 3D during 3D CG playback according to the present invention.
It is a flowchart which shows the synchronization process of a CG data stream.

FIG. 18 is a perspective view of a personal computer used with a floppy disk storing a program for causing a computer to execute the method of the present invention having the steps of the flowcharts of FIGS.

[Explanation of symbols]

 Reference Signs List 1 3DCG data stream transmission unit 2 3DCG data stream transmission unit 3 3DCG data stream reception unit 4 3D virtual space generation unit 5 sound generation unit 11 skeletal structure stream transmission unit 12 motion stream transmission unit 13 sound stream transmission unit 14 stream conversion unit 15 State management unit 31 Stream conversion unit 32 Skeletal structure stream reception unit 33 Motion stream reception unit 34 Sound stream reception unit Ss, Ss1 Skeletal data stream Sm, Sm1 Motion data stream Sa, Sa1 Sound data stream Sr Request signal

Claims (19)

[Claims] A transmitting unit (PCS) and a receiving unit (PCC)
And a transmission unit (TR, 2) for connecting the transmitting unit (PCS) and the receiving unit (PCC).
From S), a shape stream (Ss), which is a data stream indicating the shape of the skeletal structure in three-dimensional computer graphics, an operation stream (Sm), which is a data stream indicating motion, and a data stream, which indicates sound synchronized with the motion. The sound stream (Sa) is transmitted to the receiving unit (PCC), and the receiving unit (PCC) transmits the transmitted operation stream (Sm) and the transmitted sound stream (Sm).
An apparatus (100) for transmitting and receiving a data stream representing a three-dimensional virtual space so as to generate three-dimensional computer graphics by synchronizing a), wherein the transmission unit (PCS) includes: , Operation stream (Sm), and the operation stream (Sm)
A sound stream (Sa) synchronized with a plurality of streams (Ss1, Sm1, Sa1; DS) according to a predetermined data stream (DS) format (VRF)
The transmission means (TR) has a plurality of streams (Ss) for transmitting simultaneously as (1).
1, Sm1, Sa1; DS1) from the transmission unit (PCS) to the reception unit (PCC), and the reception unit (PCC) transmits the transmission unit via the transmission means (TR, 2). (PCS)
Receiving the plurality of streams (DS1), determining the type and format (VRML) of the received stream (DS1), receiving the plurality of streams (Sm1, Sa1), and determining each of them. Data stream receiving means (3) for adding necessary restoration processing to the stream (Sm1, Sa1) in accordance with the stream (Sm1, Sa1), and the stream (DS1) received by the receiver (PCC)
Of the skeleton structure or the three-dimensional virtual space in the three-dimensional virtual space using the shape stream (Ss1) or the three-dimensional shape data (DS1, DS) read in advance, and furthermore, the motion stream (Sm1) is generated. A three-dimensional virtual space generating means (4) for moving the shape of the skeletal structure by using the sound data (Sa1) when the stream (DS1) includes the sound data (Sa1). An apparatus (100) for transmitting and receiving a data stream representing a three-dimensional virtual space, comprising: a sound reproducing means (5) for reproducing (Sa1). 2. A transmitting unit (PCS) and a receiving unit (PCC)
And transmission means (TR, 2) for connecting the transmitting section (PCS) and the receiving section (PCC) in two directions, and the shape of the skeleton structure in three-dimensional computer graphics from the transmitting section (PCS). The shape stream (Ss), which is a data stream indicating the motion, the motion stream (Sm), which is a data stream indicating motion, and the sound stream (Sa), which is a data stream indicating sound synchronized with the motion, are transmitted to the receiving unit (PCC). The receiving unit (PCC) transmits and receives a data stream representing a three-dimensional virtual space so as to synchronize the transmitted operation stream (Sm) and the transmitted sound stream (Sa) to generate three-dimensional computer graphics. The transmission unit (PCS) responds to a request (Sr) from the reception unit (PCC). Immediately, the shape stream (Ss), the operation stream (Sm), and the sound stream (Sa) synchronized with the operation stream (Sm) are converted into a predetermined data stream (DS) format (V
RF) and a plurality of streams (Ss1, Sm1, S
a1; DS1), the transmitting means (1) for simultaneously transmitting the plurality of streams (Ss).
1, Sm1, Sa1; DS1) from the transmitting unit (PCS) to the receiving unit (PCC) interactively, and the receiving unit (PCC) transmits via the transmission means (TR, 2). The transmission unit (PCS)
And the type and format (VRML) of the received stream (DS1) are determined, and the plurality of streams (Sm1, Sa1) are received in real time as necessary. Then, each of the determined streams (Sm1, Sm
The restoration processing necessary according to a1) is performed in a stream (Sm1,
Data stream receiving means (3) to be added to Sa1), and the stream (DS1) received by the receiving unit (PCC)
Of the skeleton structure or the three-dimensional virtual space in the three-dimensional virtual space using the shape stream (Ss1) or the three-dimensional shape data (DS1, DS) read in advance, and furthermore, the motion stream (Sm1) is generated. A three-dimensional virtual space generating means (4) for moving the shape of the skeletal structure by using the sound data (Sa1) when the stream (DS1) includes the sound data (Sa1). An apparatus (100) for transmitting and receiving a data stream representing a three-dimensional virtual space, comprising: a sound reproducing means (5) for reproducing (Sa1). 3. The three-dimensional virtual space generation means (4)
For the data stream (DS) restored by the data stream receiving means (3), the transmission means (TR,
On the basis of requests (Sr) from a plurality of receiving units (PCCn) connected via 2), the type and part of data to be transmitted are specified, and the specified data is managed. Means (1) for setting the transmission of the managed data to the receiving unit (PCC), wherein the transmitting unit (PCS, 1) responds to the request (Sr). Receiving unit (PCC) requesting
3. The device (1) according to claim 2, wherein
00). 4. The transmission means (1) transmits a shape stream (S) to a transmission target data stream (DS).
s), motion stream (Sm) and sound stream (Sm).
a) and one or more data streams (D
S), the data stream (DS) is compressed or decomposed into predetermined division units (packets), and discriminating means (ID, ID, MH) and a means (14) for transmitting the stream (DS1) together with the transmitted stream (DS1).
For 1), a shape stream (Ss1), a motion stream (Sm1) and a sound stream (Sa1) are determined,
Based on the discriminating means (ID, MH), means for restoring as one or more streams (DS1) and for simultaneously restoring the original data stream (DS) by connecting predetermined division units (packets) ( 31), and the three-dimensional space generation means (4) generates a shape stream (S) for the reconstructed data stream (DS).
s), motion stream (Sm) and sound stream (Sm).
a) for discriminating a) and a means (3) for reproducing the shape stream (Ss), the motion stream (Sm) and the sound stream (Sa) based on the discriminating means (ID, MH). Apparatus (100) according to claim 2 or 3. 5. The transmission means (1) transmits a shape stream (S) to a transmission target data stream (DS).
s), motion stream (Sm) and sound stream (Sm).
a) is determined, and for a plurality of operations, one or more data streams (DS) are combined with the type and combination of the operation, the shape of the corresponding three-dimensional skeleton structure, and the reproduction condition, and the data stream (DS) is compressed. Alternatively, the stream (DS1) is decomposed into predetermined division units (packets), and the stream (DS1) is transmitted together with the discriminating means (ID, MH) for the shape stream (Ss), the operation stream (Sm), and the sound stream (Sa). (14), and the receiving means (3) includes the transmitted stream (DS)
For 1), a shape stream (Ss1), a motion stream (Sm1) and a sound stream (Sa1) are determined,
For a plurality of actions, the action type and combination
Means for restoring as one or more streams (DS1) based on the shape of the dimensional skeletal structure and the reproduction conditions or for simultaneously reconstructing predetermined division units (packets) to restore the original data stream (DS) ( 31), and the three-dimensional space generation means (4) generates a shape stream (S) for the restored data stream (DS).
s), motion stream (Sm) and sound stream (Sm).
While determining a), for a plurality of actions, a combination of action types and
4. The apparatus according to claim 2, further comprising means (4) for reproducing an operation of the shape of the three-dimensional skeleton structure and the shape of the skeleton structure specified based on the reproduction conditions.
0). 6. The transmission means (1) transmits a shape stream (S) to a transmission target data stream (DS).
s), motion stream (Sm) and sound stream (Sm).
a) is determined, and a plurality of actions and sounds are combined with types of actions and sounds, and the shape of the corresponding three-dimensional skeletal structure;
And one or more data streams (D
S), the data stream (DS) is compressed or decomposed into predetermined division units (packets), and discriminating means (ID, ID) for the shape stream (Ss), the operation stream (Sm), and the sound stream (Sa) MH) and means (14) for transmitting a stream (DS1) together with synchronization information (TS) between the shape stream (Ss), the operation stream (Sm) and the sound stream (Sa). , The transmitted stream (DS
For 1), a shape stream (Ss1), a motion stream (Sm1) and a sound stream (Sa1) are determined,
For a plurality of actions and sounds, based on the type and combination of the actions, the shape of the corresponding three-dimensional skeletal structure, and the reproduction conditions, one or more streams (DS1) are restored or divided into predetermined units (packets). ) At the same time to restore the original data stream (DS) (3)
1), and the three-dimensional space generation means (4) generates a shape stream (S) for the reconstructed data stream (DS).
s), motion stream (Sm) and sound stream (Sm).
Means for recognizing (a), for a plurality of actions and sounds, reproducing the action of the combination of the type of action, the shape of the corresponding three-dimensional skeletal structure, and the shape of the skeletal structure specified based on the playback conditions ( 4) and
3. A means (4) for synchronizing and reproducing the sound data (Sa) and the movement (Sm) at the same time.
Or the apparatus (100) according to 3. 7. A shape stream (Ss), which is a data stream indicating the shape of a skeleton structure in three-dimensional computer graphics, a motion stream (Sm), which is a data stream indicating motion, and synchronization with the motion from the transmitting unit (PCS). A sound stream (Sa) indicating a sound to be played is transmitted to a receiving unit (PCC), and the received operating stream (Sm) and the sound stream (Sm) are transmitted to the receiving unit (PCC).
a) transmitting and receiving a data stream representing a three-dimensional virtual space so as to generate three-dimensional computer graphics by synchronizing the shape stream (Ss) and the operation stream (S
m) and the sound stream (Sa) synchronized with the operation stream (Sm) as a plurality of streams (Ss1, Sm1, Sa1; DS1) according to a predetermined data stream (DS) format (VRF). A transmitting step (S35) for transmitting simultaneously; connecting the transmitting unit (PCS) and the receiving unit (PCC) to connect the plurality of streams (Ss1, Sm1, Sa1; D
(S1) transmitting from the transmitting unit (PCS) to the receiving unit (PCC) (S21 to S37); receiving the transmitted plurality of streams (DS1) and receiving the received stream (DS1) Of the stream (Sm1, Sa1), and receives the streams (Sm1, Sa1) according to the determined streams (Sm1, Sa1).
1, Sa1), a data stream receiving step (S39, S41, S43) for performing restoration processing, and the stream (DS1) received by the receiving unit (PCC).
Of the skeleton structure or the three-dimensional virtual space in the three-dimensional virtual space using the shape stream (Ss1) or the three-dimensional shape data (DS1, DS) read in advance, and furthermore, the motion stream (Sm1) 3D virtual space generation step for moving the shape of the skeletal structure using
73), and when the stream (DS1) includes sound data (Sa1), a sound reproduction step (S61, S61) for reproducing the sound data (Sa1) in synchronization with the movement (Sm1) of the skeletal structure.
S63). A method for transmitting and receiving a data stream representing a three-dimensional virtual space, the method comprising: 8. A shape stream (Ss), which is a data stream indicating the shape of a skeleton structure in three-dimensional computer graphics, a motion stream (Sm), which is a data stream indicating motion, and synchronization with the motion from the transmitting unit (PCS). A sound stream (Sa) indicating a sound to be played is transmitted to a receiving unit (PCC), and the received operating stream (Sm) and the sound stream (Sm) are transmitted to the receiving unit (PCC).
a) transmitting and receiving a data stream representing a three-dimensional virtual space so as to generate three-dimensional computer graphics by synchronizing a) with each other, and in response to a request (Sr) from the receiving unit (PCC), In accordance with a predetermined data stream (DS) format (VRF), the shape stream (Ss), the operation stream (Sm), and the sound stream (Sa) synchronized with the operation stream (Sm) A transmission step (S35) of transmitting simultaneously as streams (Ss1, Sm1, Sa1; DS1), and the transmitting unit (PCS) and the receiving unit (PCC) are bidirectionally connected to each other to form the plurality of streams (Ss1). , Sm1, Sa
1; DS1) from the transmitting section (PCS) to the receiving section (PC
C) a transmission step (S21-S3) for interactive transmission
7) receiving the plurality of transmitted streams (DS1), determining the type and format (VRLM) of the received stream (DS1), and receiving the plurality of streams (Sm1, Sa1) in real time. A data stream receiving step (S39, S41, S43) for performing a restoration process on the stream (Sm1, Sa1) according to each of the determined streams (Sm1, Sa1); and the stream received by the receiving unit (PCC). (DS1)
Of the skeleton structure or the three-dimensional virtual space in the three-dimensional virtual space using the shape stream (Ss1) or the three-dimensional shape data (DS1, DS) read in advance, and furthermore, the motion stream (Sm1) 3D virtual space generation step for moving the shape of the skeletal structure using
73), and when the stream (DS1) includes sound data (Sa1), a sound reproduction step (S61, S61) for reproducing the sound data (Sa1) in synchronization with the movement (Sm1) of the skeletal structure.
S63). A method for transmitting and receiving a data stream representing a three-dimensional virtual space, the method comprising: 9. The three-dimensional virtual space generating step (S5)
1 to S73), the data stream receiving step (S39, S41,
For the data stream (DS) restored in S43), based on the requests (Sr) from the plurality of receiving units (PCCn) connected in the transmitting step (S35),
Specifying a type and a portion of data to be transmitted and managing the specified data, and setting transmission of the managed data to a receiving unit (PCC) that has requested (S35) In response to the request (Sr), the transmitting unit (PCS, 1) transmits the managed data to the receiving unit (PCC)
9. The method according to claim 8, wherein the data is transmitted to a computer. 10. The transmitting step (S35) includes determining a shape stream (Ss), a motion stream (Sm), and a sound stream (Sa) with respect to a data stream to be transmitted (DS) and transmitting one or more data streams. Stream (DS), the data stream (DS) is compressed or decomposed into predetermined division units (packets), and the discriminating means (Ss), motion stream (Sm) and sound stream (Sa) are determined. (S3) transmitting the stream (DS1) together with the ID (MH, ID)
5), wherein the receiving step (S39, S41, S43) determines a shape stream (Ss1), a motion stream (Sm), and a sound stream (Sa1) for the transmitted stream (DS1). Restoring as one or more streams (DS1) based on the discriminating means (ID, MH) and simultaneously restoring the original data stream (DS) by connecting predetermined division units (packets) (S43), and the three-dimensional virtual space generating step (S51 to S73)
For the recovered data stream (DS):
The shape stream (Ss), the motion stream (Sm) and the sound stream (Sa) are discriminated, and the discrimination means (ID, M
10. A step (S59) for reproducing a shape stream (Ss), a motion stream (Sm) and a sound stream (Sa) based on H).
The method described in. 11. The transmitting step (S35) includes determining a shape stream (Ss), a motion stream (Sm), and a sound stream (Sa) for a data stream (DS) to be transmitted, and One or more data streams (DS) together with the type and operation of the operation, the shape of the corresponding three-dimensional skeletal structure, and the reproduction condition, and compress the data stream (DS) or divide the data stream into predetermined division units (packets). Decompose, shape stream (Ss),
The stream (DS) together with the discriminating means (ID, MH) for the operation stream (Sm) and the sound stream (Sa)
1) the step of transmitting (S35), and the step of receiving (S39, S41, S43) includes, for the transmitted stream (DS1), a shape stream (Ss1), a motion stream (Sm1), and a sound. The stream (Sa1) is discriminated, and a plurality of operations are restored or predetermined as one or more streams (DS1) based on the combination of the type of operation, the shape of the corresponding three-dimensional skeleton structure, and the reproduction condition. (S43) restoring the divided units (packets) to the original data stream (DS) at the same time, and further comprises the three-dimensional virtual space generating steps (S51 to S73).
For the recovered data stream (DS):
The shape stream (Ss), the motion stream (Sm), and the sound stream (Sa) are discriminated, and a plurality of motions are specified based on a combination of motion types, a shape of a corresponding three-dimensional skeleton structure, and reproduction conditions. The method according to claim 8 or 9, further comprising a step (S41) of reproducing a motion of the shape of the skeleton structure. 12. The transmitting step (S35) includes determining a shape stream (Ss), a motion stream (Sm), and a sound stream (Sa) with respect to a transmission target data stream (DS), and For the combination of motion and sound type, the shape of the corresponding 3D skeleton structure,
And one or more data streams (D
S), the data stream (DS) is compressed or decomposed into predetermined division units (packets), and discriminating means (ID, ID) for the shape stream (Ss), the operation stream (Sm), and the sound stream (Sa) MH) and the shape stream (Ss), the operation stream (Sm), and the stream (DS1) together with the synchronization information (TS) between the sound streams (Sa). The receiving step (S39, S41) , S43) determines a shape stream (Ss1), a motion stream (Sm1), and a sound stream (Sa1) for the transmitted stream (DS1), and determines the type of motion for a plurality of motions and sounds. And restore or convert as one or more streams (DS1) based on the shape of the corresponding three-dimensional skeletal structure and playback conditions. Or a step of simultaneously connecting predetermined division units (packets) to restore the original data stream (DS) (S43), and further, the three-dimensional virtual space generation steps (S51 to S73).
For the recovered data stream (DS):
The shape stream (Ss), the motion stream (Sm), and the sound stream (Sa) are determined, and a plurality of motions and sounds are combined based on the type of motion, the shape of the corresponding three-dimensional skeleton structure, and reproduction conditions. A step (S59) of reproducing the operation of the shape of the designated skeletal structure, and a step (S61 to S67) of simultaneously reproducing the sound data (Sa) and the movement (Sm) in synchronization. Item 10. The method according to item 8 or 9. 13. A transmitting unit (PCS) and a receiving unit (PC)
C) and transmission means (TR, 2) for connecting the transmitting unit (PCS) and the receiving unit (PCC).
CS), a shape stream (Ss) which is a data stream indicating the shape of the skeletal structure in three-dimensional computer graphics, an operation stream (Sm) which is a data stream indicating motion, and a data stream indicating sound synchronized with the motion. The sound stream (Sa) is transmitted to the receiving unit (PCC), and the receiving unit (PCC) transmits the transmitted operation stream (Sm) and the transmitted sound stream (Sm).
A transmission unit (PCS) of the system (100) for transmitting and receiving a data stream representing a three-dimensional virtual space so as to generate three-dimensional computer graphics by synchronizing a).
A computer (PC) readable recording medium (FD) that stores a program for causing a computer (PC) to execute the transmission function of (a), the transmission function comprising: the shape stream (Ss) and the operation stream (Sm) ,
And a sound stream (Sa) synchronized with the operation stream (Sm) according to a predetermined data stream (DS) format (VRF).
s1, Sm1, Sa1; DS1) and a function of simultaneously transmitting the streams (Ss1, Sm1, Sa1; DS).
(1) transmitting the data from the transmitting unit (PCS) to the receiving unit (PCC).
C) Readable recording medium (FD). 14. A transmitting unit (PCS) and a receiving unit (PC)
C) and transmission means (TR, 2) for connecting the transmitting unit (PCS) and the receiving unit (PCC).
CS), a shape stream (Ss) which is a data stream indicating the shape of the skeletal structure in three-dimensional computer graphics, an operation stream (Sm) which is a data stream indicating motion, and a data stream indicating sound synchronized with the motion. The sound stream (Sa) is transmitted to the receiving unit (PCC), and the receiving unit (PCC) transmits the transmitted operation stream (Sm) and the transmitted sound stream (Sm).
a receiving unit (PCC) of the system (100) for transmitting and receiving a data stream representing a three-dimensional virtual space so as to generate three-dimensional computer graphics by synchronizing a)
A computer (PC) readable recording medium (FD) that records a program for causing a computer (PC) to execute the receiving function of (a), wherein the transmission unit (PCS) includes the shape stream (Ss) and the operation stream. (Sm) and the operation stream (Sm)
A sound stream (Sa) synchronized with a plurality of streams (Ss1, Sm1, Sa1; DS) according to a predetermined data stream (DS) format (VRF)
1), and the transmission means (TR, 2) further transmits the plurality of streams (Ss
1, Sm1, Sa1; DS1) from the transmitting unit (PCS) to the receiving unit (PCC), and the receiving function includes the transmitting unit (PCS) via the transmitting means (TR, 2).
, The type and format (VRML) of the received stream (DS1) are determined, and the plurality of streams (Sm1, Sa1) are received. A function of adding necessary restoration processing to the stream (Sm1, Sa1) according to the stream (Sm1, Sa1), and the stream (DS1) received by the receiving unit (PCC).
Of the skeletal structure or the three-dimensional virtual space in the three-dimensional virtual space using the shape stream (Ss1) or the three-dimensional shape data (DS1, DS) read in advance, and further the motion stream (Sm1) is generated. A function of moving the shape of the skeletal structure using the function, and a function of reproducing the sound data (Sa1) in synchronization with the movement (Sm1) of the skeletal structure when the stream (DS1) includes sound data (Sa1). A computer (PC) readable recording medium (F)
D). 15. A shape stream (Ss), which is a data stream indicating a shape of a skeleton structure in three-dimensional computer graphics, a motion stream (Sm), which is a data stream indicating motion, and synchronization with the motion from the transmitting unit (PCS). A sound stream (Sa) indicating a sound to be played is transmitted to a receiving unit (PCC), and the received operating stream (Sm) and the sound stream (Sm) are transmitted to the receiving unit (PCC).
a) a computer (PC) readable program storing a program for causing a computer (PC) to execute a method of transmitting and receiving a data stream representing a three-dimensional virtual space so as to generate three-dimensional computer graphics by synchronizing a) A recording medium (FD), the method comprising: responding to a request (Sr) from the receiving unit (PCC), in real time, the shape stream (Ss), the operation stream (Sm), and the operation stream ( (S35) transmitting the sound stream (Sa) synchronized with Sm) simultaneously as a plurality of streams (Ss1, Sm1, Sa1; DS1) in accordance with a predetermined data stream (DS) format (VRF). Bidirectionally connecting the transmitting unit (PCS) and the receiving unit (PCC), Ss1, Sm1, Sa
1; DS1) from the transmitting section (PCS) to the receiving section (PC
C) a transmission step (S21-S3) for interactive transmission
7) receiving the plurality of transmitted streams (DS1), determining the type and format (VRLM) of the received stream (DS1), and receiving the plurality of streams (Sm1, Sa1) in real time. A data stream receiving step (S39, S41, S43) for performing a restoration process on the stream (Sm1, Sa1) according to each of the determined streams (Sm1, Sa1); and the stream received by the receiving unit (PCC). (DS1)
Of the skeleton structure or the three-dimensional virtual space in the three-dimensional virtual space using the shape stream (Ss1) or the three-dimensional shape data (DS1, DS) read in advance, and furthermore, the motion stream (Sm1) 3D virtual space generation step for moving the shape of the skeletal structure using
73), and when the stream (DS1) includes sound data (Sa1), a sound reproduction step (S61, S61) for reproducing the sound data (Sa1) in synchronization with the movement (Sm1) of the skeletal structure.
(S63).
C) Readable recording medium (FD). 16. A transmitting unit (PCS) used in a system (100) for transmitting and receiving a data stream representing a three-dimensional virtual space, the system (100) comprising: a receiving unit (PCC); PCS) and transmission means (TR, 2) for connecting the receiving unit (PCC), and a shape stream which is a data stream indicating the shape of a skeletal structure in three-dimensional computer graphics from the transmitting unit (PCS). (Ss), an operation stream (Sm) which is a data stream indicating a motion, and a sound stream (Sm) which is a data stream indicating a sound synchronized with the motion.
a) to the receiving unit (PCC).
Then, the transmitted operation stream (Sm) and the transmitted sound stream (Sa) are synchronized to generate three-dimensional computer graphics, while the transmission unit (PCS) transmits the shape stream (Ss) and the operation stream (Sm). ) And the operation stream (Sm)
A sound stream (Sa) synchronized with a plurality of streams (Ss1, Sm1, Sa1; DS) according to a predetermined data stream (DS) format (VRF)
The transmission means (TR) has a plurality of streams (Ss) for transmitting simultaneously as (1).
1, Sm1, Sa1; DS1) from the transmitting unit (PCS) to the receiving unit (PCC). 17. A transmitting unit (PCS) used in a system (100) for transmitting and receiving a data stream representing a three-dimensional virtual space, the system (100) comprising: a receiving unit (PCC); PCS) and transmission means (TR, 2) for connecting the receiving section (PCC) bidirectionally, and a data stream indicating the shape of the skeletal structure in three-dimensional computer graphics from the transmitting section (PCS). A certain shape stream (Ss), a motion stream (Sm) which is a data stream indicating a motion, and a sound stream (Sm) which is a data stream indicating a sound synchronized with the motion.
a) to the receiving unit (PCC).
Then, the transmitted operation stream (Sm) and the transmitted sound stream (Sa) are synchronized to generate three-dimensional computer graphics, while the transmitting unit (PCS) transmits a request (Sr) from the receiving unit (PCC). ), The shape stream (Ss), the operation stream (Sm), and the sound stream (Sa) synchronized with the operation stream (Sm) are converted in real time into a predetermined data stream (DS) format (V
RF) and a plurality of streams (Ss1, Sm1, S
a1; DS1), the transmitting means (1) for simultaneously transmitting the plurality of streams (Ss).
1, Sm1, Sa1; DS1) from the transmitting unit (PCS) to the receiving unit (PCC) interactively. 18. A receiving unit (PCC) used in a system (100) for transmitting and receiving a data stream representing a three-dimensional virtual space, wherein the system (100) includes a transmitting unit (PCS) and the transmitting unit (PCS). PCS) and transmission means (TR, 2) for connecting the receiving unit (PCC), and a shape stream which is a data stream indicating the shape of a skeletal structure in three-dimensional computer graphics from the transmitting unit (PCS). (Ss), an operation stream (Sm) which is a data stream indicating a motion, and a sound stream (Sm) which is a data stream indicating a sound synchronized with the motion.
a) to the receiving unit (PCC).
Then, the transmitted operation stream (Sm) and the transmitted sound stream (Sa) are synchronized to generate three-dimensional computer graphics, while the transmission unit (PCS) transmits the shape stream (Ss) and the operation stream (Sm). ) And the operation stream (Sm)
A sound stream (Sa) synchronized with a plurality of streams (Ss1, Sm1, Sa1; DS) according to a predetermined data stream (DS) format (VRF)
1), and the transmission means (TR, 2) further transmits the plurality of streams (Ss
1, Sm1, Sa1; DS1) from the transmission unit (PCS) to the reception unit (PCC), and the reception unit (PCC) transmits the transmission unit via the transmission means (TR, 2). (PCS)
Receiving the plurality of streams (DS1), determining the type and format (VRML) of the received stream (DS1), receiving the plurality of streams (Sm1, Sa1), and determining each of them. Data stream receiving means (3) for adding necessary restoration processing to the stream (Sm1, Sa1) in accordance with the stream (Sm1, Sa1), and the stream (DS1) received by the receiver (PCC)
Of the skeletal structure or the three-dimensional virtual space in the three-dimensional virtual space using the shape stream (Ss1) or the three-dimensional shape data (DS1, DS) read in advance, and further the motion stream (Sm1) is generated. A three-dimensional virtual space generating means (4) for moving the shape of the skeletal structure using the sound data (Sa1) when the stream (DS1) includes the sound data (Sa1). And a sound reproducing means (5) for reproducing (Sa1). 19. A receiving unit (PCC) used in a system (100) for transmitting and receiving a data stream representing a three-dimensional virtual space, the system (100) comprising a transmitting unit (PCS) and the transmitting unit (PCS). PCS) and transmission means (TR, 2) for bidirectionally connecting the receiving unit (PCC) with the receiving unit (PCC). A shape stream (Ss), a motion stream (Sm) which is a data stream indicating a motion, and a sound stream (Sm) which is a data stream indicating a sound synchronized with the motion.
a) to the receiving unit (PCC).
Then, the transmitted operation stream (Sm) and the transmitted sound stream (Sa) are synchronized to generate three-dimensional computer graphics, while the transmitting unit (PCS) transmits a request (Sr) from the receiving unit (PCC). ), The shape stream (Ss), the operation stream (Sm), and the sound stream (Sa) synchronized with the operation stream (Sm) are converted in real time into a predetermined data stream (DS) format (V
RF) and a plurality of streams (Ss1, Sm1, S
a1; DS1), and the transmission means (TR, 2) transmits the plurality of streams (Ss
1, Sm1, Sa1; DS1) from the transmitting unit (PCS) to the receiving unit (PCC) interactively, and the receiving unit (PCC) transmits via the transmission means (TR, 2). The transmission unit (PCS)
And the type and format (VRML) of the received stream (DS1) are determined, and the plurality of streams (Sm1, Sa1) are received in real time as necessary. Then, each of the determined streams (Sm1, Sm
The restoration processing necessary according to a1) is performed in a stream (Sm1,
Data stream receiving means (3) to be added to Sa1), and the stream (DS1) received by the receiving unit (PCC)
Of the skeletal structure or the three-dimensional virtual space in the three-dimensional virtual space using the shape stream (Ss1) or the three-dimensional shape data (DS1, DS) read in advance, and further the motion stream (Sm1) is generated. A three-dimensional virtual space generating means (4) for moving the shape of the skeletal structure using the sound data (Sa1) when the stream (DS1) includes the sound data (Sa1). And a sound reproducing means (5) for reproducing (Sa1).