JP2002305758A - Device and system for presence communication - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and a system for presence communication that can build up a communication environment as if one person were working on a place of one's own together with other person and the other person were working on a place of the other person together with the one person even when they are resident in different real spaces and are working at their respective places. SOLUTION: As to people existing in real spaces and objects existing around the people, states between the people and the objects, between the objects and of the object themselves, are respectively logically calculated and by, using information from a database describing rules or the like, space logic information is generated and virtual objects are recognize on the basis of the logical information of the space.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a real-life communication device and a real-life communication system that allow a user of a device to recognize an image three-dimensionally to give a sense of realism.

[0002]

2. Description of the Related Art Japanese Patent Laid-Open Publication No. 20-204702 discloses a realism generating apparatus for performing three-dimensional display of a virtual object by independently detecting the position of a predetermined part of the body and the position of an item of an observer using an item.
No. 00-202164.

According to this known technique, an object to be displayed or an object having a shape close to the object is displayed together with physical information, and an observer feels like operating an object in a virtual space with an item actually held. Is obtained.

[0004]

Consider an environment in which humans existing in real spaces in different situations communicate with each other using real objects existing there.

[0005] In the prior art, it is possible to display an object existing there as it is. However, rather than transmitting the existing objects as they are, rather than extracting the logical information of the transmitting space and converting the model information or the display position for some objects, the information receiver It is desired that the user can easily understand the situation of the other party.

For example, when a teacher of a cooking school stands in a kitchen at home and students also stand in a kitchen at home, there is a difference in the arrangement of cooking utensils in the kitchen and the shape and size of utensils existing in the kitchen. is there. In such a case, in the related art, the position where the teacher stands and the position information of the surrounding objects are transmitted as they are. Also, even if the student has an instrument with the same function, the teacher uses an instrument with a different shape, and the stereoscopic image of the instrument is displayed, and the instrument that the student does not know is displayed. May not be sure.

[0007] The present invention solves these problems, and even if they are present in different real spaces and each other is working at that place, whether each other is working at their own place or not. The purpose is to construct a communication environment like this.

[0008]

Means for Solving the Problems 1) An apparatus for recognizing an individual object in a real space and an apparatus for detecting a change in position and shape are provided. 2) A device for detecting the position and posture of a person is provided. 3) A database is provided which describes the shape and function information of the object and the physical definition applied when the object exists in the real space. 4) A device that converts the relationship between a human and an object in space, the relationship between an object and an object, and the state of the object itself into logical information based on changes in the position and orientation of the human, the movement of the object, and changes in the shape. Is provided. 5) A communication network for connecting between a plurality of real spaces is provided. 6) A device is provided that complements model and coordinate information in order to interpret logical information sent from a communication partner in accordance with the current situation of the real space and generate three-dimensional video information. 7) An image display device is provided in the real space for recognizing the state of the communication partner as a stereoscopic video.

Using the above device, information on the position, posture, position, and state of an object existing in space is detected.
Next, logical information is generated using the detected information and the information described in the database. The generated logical information is transmitted to a terminal existing in another space connected to the communication network.

[0010] The terminal that has received the logical information generates the sent logical information as a stereoscopic image so that there is no inconsistency between the object existing in the space where the terminal is located and the human state.

[0011] By the above-described apparatus and processing, even if they exist in different real spaces and they are working at the same place, they behave as if the other party is at their own place and working. Build a communication environment that can do it.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 This is an example in which the present invention is applied to a teaching system for a method of cooking foodstuffs.

FIG. 1 shows an embodiment system for explaining the teaching of a method of cooking foodstuffs. In the figure, 1 is a teacher's space, 2 is a student's space, 3 is a teacher's data processing terminal,
4 is a network, 5 is a student data processing terminal, 6 is a logical information processing device, 7 is a database of objects and the like, 8 is an input signal processing device, 9 is an output information processing device, 10 is a position sensor, 11 is a marker, Reference numeral 12 denotes a cooking tool, 13 denotes an image of the cooking tool, 14 denotes an avatar, and 15 denotes a real object.

The position sensor 10A detects the position of the teacher and the position of the object (information on the state of cooking utensils) in the space 1 and changes thereof. The input signal processing device 8A informs the logical information processing device 6A of the positions of objects and people in the space 1 and changes in their shapes. In addition, from the database 7A, the function, the shape, the information for complementing the logical processing, and the like of the object existing in the space 1 are transmitted to the logical information processing device 6A. Then, the logical information processing device 6A sends the logical information in the space 1 to the network 4.

The logical information is transmitted to the logical information processing device 6B. The logical information processing device 6B stores the coordinates to be displayed and the logical information added to the model information to be displayed.
The information is transmitted to the output information processing device 9B. Output information processing device 9B
Causes the stereoscopic video information to appear in the space 2 and, for example, the teacher's avatar 14B or the frying pan video 13B on the student's side so as to make the same movement as the teacher or the frying pan operated by the teacher. Let it appear.

The space 1 has a reference numeral 15 as a real object.
There are teachers and cooking utensils indicated by A-1 and 15A-2. For example, for a person who looks at the space 1 using virtual object stereoscopic glasses or the like, a real object existing in the space 1 The body (teacher, cooking utensil, etc.) is superimposed on the virtual object displayed on the virtual object stereoscopic glasses,
It looks as if the real object and the virtual object exist together in the real space. Note that no virtual object exists in the space 1.

In the space 2, a reference numeral 15
For a person who is viewing the space 2 using the virtual object stereoscopic glasses or the like in which a student indicated by B exists, a real object (student) existing in the space 2 and the virtual object It looks as if the avatar 14B of the virtual object (teacher) displayed on the stereoscopic glasses and the image 13B of the cooking utensil exist together.

Consider a situation in which a teacher at a cooking school and a student are located at separate locations, and the teacher teaches the student how to cook while standing in separate kitchens. At this time, the arrangement of the cooking utensils in the kitchen and the tools existing there do not always match. For example, it is conceivable that a teacher and a student cook in two kitchens having different arrangements as shown in FIG. Consider applying the present invention to such a situation.

First, in order to detect body posture information and position information, a plurality of markers 11 are attached to joints of the body, etc., for teachers and students, which exist in the spaces 1 and 2 respectively. By installing two or more cameras (position sensors) 10 in the spaces 1 and 2 and photographing the markers 11 existing in the space,
The three-dimensional position of the marker 11 is detected. Position sensor 10
As long as the three-dimensional position of a human or the like can be detected,
A method such as a magnetic sensor or an ultrasonic wave may be used.

A cooking utensil 12 existing in the space is provided with a position sensor 10 and a marker 11 for detecting a position in advance, as in the case of a human body.

The database 7 describes in advance the functions, models, and rules of objects existing in the space. For example, it is assumed that the following information is described in the database 7. That is, human information includes
When human information is transmitted to a terminal in another place, information indicating that conversion is not possible is described, together with model information displayed as the avatar 14 in the space 2. For stoves, the model information and function information are described, and when transferred to the space where other terminals exist, if there is a stove at the place, it can be converted to the stove at the place Is described. further,
Information indicating that the stove does not move during the work is immovable. In the case of a frying pan, similarly, in addition to the model and function information, information that the frying pan can be converted to a frying pan on the spot and that it can be moved is described in the database.

The operation of the system is as follows: when a position or state of a person or an object changes in either space,
System processing starts. For example, when the sensor detects that the teacher has moved or the body part has moved, the values of the input sensors are used to indicate the position where the teacher stands and the position of the teacher to indicate the posture of the teacher. The respective coordinates are calculated, and the teacher's body orientation, head orientation, hand position, and the like are calculated from the coordinates of the teacher and the coordinates of each part.

Information on objects existing around the teacher is extracted from the database 7A, and ○ exists in the direction in which the teacher's body is facing. △△ exists in the direction in which the teacher's head is facing. Logic information that describes the relationship between the teacher and objects around the teacher, such as the presence of XX at the position of the teacher's hand, is generated.

Some logical information is generated according to a predetermined rule. For example, if there is a cooking stove in the direction in which the teacher's body and head are facing, "the teacher's body is facing the stove" or "the teacher's head is facing the stove. Is generated.

When a movement or a change in shape of an object is detected by a sensor in addition to a human motion, the position, direction, and state of the object are calculated in the same manner. An object in a range affected by the object is calculated from the position of the object, and information on an object that may have an effect or an object that may be affected is extracted from the database 7A.

For example, it is assumed that a change in the position of an object or a frying pan existing in a kitchen is detected. Suppose that it is found from the position that a human hand and a stove exist around the periphery. The rule that the frying pan does not move autonomously is input in advance in the information on the frying pan in the database 7A. In this case, the movement of the frying pan is affected by human hands. As a result, logical information that "the frying pan has been moved by the teacher's hand" is generated.

The generated logical information is transmitted from the logical information processing device 6A to the logical information processing device 6B existing in the student terminal via the network 4. The logical information processing device 6B of the student terminal starts processing upon receiving the logical information.

A device is connected which can detect the arrangement of cooking utensils at the place where the student is located and information on the objects located there in the same way as the teacher's terminal. The database 7B stores information on the position, the state, and the function of the object existing in the space 2 on the student side.

When information such as “the teacher's body is facing the stove” or “the teacher's head is facing the stove” is sent from the teacher, the student terminal It can be seen that there is no teacher in the space 2 and the stove exists in the student's space. Then, since it does not exist in the space, the database 7 exists in the terminal on the teacher side.
A is inquired, and as a result, model information for expressing the teacher is received because it cannot be converted by another object.

The stove treats a real object, which actually exists in space, as a substitute object because an object exists at the place and can be converted by another object.

Next, the logical information expressing the space 2 on the student side and the logical information sent from the teacher side are calculated one by one, and it is determined whether there is any contradiction. If a contradiction occurs, the teacher is requested to select necessary logical information. The teacher makes a selection, and unnecessary logical information that causes inconsistency is deleted.

When the logical information “the frying pan has been moved by the teacher's hand” is sent, the frying pan is
It is possible to convert with the information of the object existing in the student's terminal.
Since the frying pan is movable, it is reproduced in the student's space 2 as a stereoscopic image. The image of the frying pan reproduced at this time is an image in which the teacher holds the frying pan on the student side. The stereoscopic video is output from a stereoscopic video generation device (not shown) connected to the output information processing device 9B of the student terminal. The displayed video is a video that allows the student to intuitively know what to use for cooking.

[Second Embodiment] Consider an example in which the present invention is applied to a dialog between humans existing in different real spaces. FIG.
Shows another embodiment system for explaining interaction between humans existing in different real spaces.

Reference numerals 1 to 14 in the figure respectively correspond to FIG. In the spaces 1 and 2, for example, in the space 2, an image 14B-1 about the user B appears.
At the same time, the avatar 14B-2 for the user A appears. The conversion between the position and the state of these users (generally, the real object 15) is sent to the input signal processing device 8B via the position sensor 10B. Then, the input signal processing device 8B compares the position and the change of the state of the object or the person in the space with
The information is transmitted to the logical information processing device 6B.

On the other hand, the database 7B transmits information for complementing the function, shape, and logical processing of the object in the space to the logical information processing device 6B. Then, as a result of these, the logical information processing device 6B sends the logical information of the space to the network 4.

In the logical information processing device 6A that has received the logical information of the space, the logical information processing device 6A uses the output information processing device 9A to use the space 1 as the movement of the avatar 14A-1 of the user B, for example. The movement of the person B is made to appear.

In the case of FIG. 2 as well, it can be inferred from the description of the case of FIG. 1 that the space 2 has a user B and a chair designated by reference numerals 15B-1 and 15B-2 as real objects. Is present, and for a person looking at the space 2 using the virtual object stereoscopic glasses, the real object 15B existing in the space 2 and the virtual object displayed on the virtual object stereoscopic glasses are displayed. Object (avatar) 14B-
1 and 14B-2 appear to be present together. Similarly, for a person looking at space 1 using virtual object stereoscopic glasses, real object 15 existing in space 1
It looks as if A-1 and 15A-2 and the virtual objects (avatars) 14A-1 and 14A-2 exist together.

It is assumed that a person who wants to have a conversation exists in differently arranged spaces and performs communication. Assuming that the users are A and B, for example, there is a customer service sofa and a chair where the user A sits in the space where the user A is, and there are two chairs in the space where the user B is present. .

When the present invention is used in such a situation, the users A and B are given the same physical position as in the first embodiment.
Attach a sensor to detect the posture. Similarly, a sensor for detecting a change in the position or shape of the chair or sofa is added.

It is assumed that the following data is described in the database 7 regarding the object. That is, with respect to the users A and B, a description is given that the conversion to the object on the spot is impossible together with the model information.

As for the chair, a function of sitting by a person is described as function information together with model information representing the shape. In addition, when the information is transmitted to another space, information indicating that the information can be converted into an object existing at the place and having a function of sitting by a person is described.

It is assumed that the sofa has the same description as the chair. Here, it is assumed that both the chair and the sofa are immovable in the database.

The position of the user A, the position of the chair, and the change in the posture of the user A are sent from the input signal processing device 8A to the logical information processing device 6A. For example, if the position of the user A is less than a certain distance from the position of the chair and the height of the human upper body or the head is less than a certain value, the user A is sitting on the chair. And the information that the user A is sitting on the chair is generated as the logical information. The generated logical information is stored in the logical information processing unit 6 of the terminal of the user B.
B.

When the logical information that the user A is sitting on the chair is transmitted to the user B, information on the user A and the chair is obtained from the database 7B. The user A obtains model information for creating an avatar from the terminal of the user A because the user A cannot convert the object in the space of the user B. Since the chair can be converted into an object in which the person present in the space of the place sits, it is converted into the chair existing in the space of the user B. As a result, the stereoscopic image of the user B appears in the space 2 through the output information processing device 9B.

Similarly, consider a case where logical information that user B is sitting on a chair is generated from the value of the input device and transmitted to the terminal of user A. When transmitted to the user A, it is understood that no chair exists in the room of the user A. Then, an inquiry is made to the database 7A of the user A to obtain information on the object. The terminal on the side of the user A searches the database 7A of the terminal of the user A for an object that can be seated other than the chair, and outputs a sofa present at the place as a search result. As a result, on the user A side, a situation appears in which the user B is sitting on the sofa in the space 1 through the output information processing device 9A.

As an effect, even when there is a great difference between the spaces of the users A and B, both the users A and B talk while obtaining a sense that the other person is in the place where they are. be able to.

FIG. 3 shows a processing flow chart in the case of the sender side. FIG. 4 shows a processing flowchart when the receiver side is set.

On the sender side, step (S1): The position and shape change of the real object are measured by the sensor. Step (S2): It is checked whether or not the position or the shape has changed. Step (S3): The position of the changed object, the direction of the change, and the state of the change are calculated. Step (S4): Calculate the effect that the real object will have on the surrounding objects. Step (S5): Acquire information on the real object from the database. Step (S6): Generate logical information. Step (S7): Transfer the logical information to another terminal device.

On the receiver side, step (S8): It is monitored whether or not logical information has been received. Step (S9): It is checked whether or not there is a reception. Step (S10): It is checked whether or not the object included in the logical information can be converted by the object on the receiver side. Step (S11): If NO in step (S10), information about the object is acquired from the transmission source database. Step (S12): In step (S10), YE
If S, information about the object is obtained from the database on the receiver side. Step (S13): It is checked whether the object is movable. Step (S14): Data of the object is set as computer graphics. Step (S15): The relationship with the state information of the space is calculated. Step (S16): It is checked whether or not a contradiction has occurred. Step (S17): If contradiction occurs, the sender side
Ask them to select the most important information. Then, unnecessary logical information is deleted. Step (S18): The space is reconstructed in a state where there is no contradiction in step (S16).

[0050]

As described above, according to the present invention, for example, changes in the position and the shape of an object existing in the vicinity of the user and the surroundings are detected independently, and the situation is logically determined. Since communication is performed by converting information into information, even if the device is installed in a different real space, it is possible to generate and communicate with a realistic space according to the situation of the real space, and to constrain to the situation of the space Instead, communication using various objects becomes possible.

A teaching system that is easy to understand because a virtual space image is generated in accordance with the space situation, and is converted into an image of an object existing in the user's own space, for example, on the receiving side. Can be built.

[Brief description of the drawings]

FIG. 1 illustrates an example system describing the teaching of a method of cooking foodstuffs.

FIG. 2 shows another embodiment system for explaining interaction between humans existing in different real spaces.

FIG. 3 shows a processing flowchart in a case where the sender side is set.

FIG. 4 shows a processing flowchart in the case of a receiver side.

[Explanation of symbols]

 1: Space 2: Space 3: Data processing terminal 4: Network 5: Data processing terminal 6: Logical information processing device 7: Database 8: Input signal processing device 9: Output information processing device 10: Position sensor 11: Marker 12: Tool 13: Tool image 14: Abata

Continued on the front page (72) Inventor Takashi Kono 2-3-1 Otemachi, Chiyoda-ku, Tokyo Nippon Telegraph and Telephone Corporation (72) Inventor Satoshi Ishibashi 2-3-1 Otemachi, Chiyoda-ku, Tokyo Sun F-term (Reference) within the Telegraph and Telephone Co., Ltd. BD08 BD09 BD16

Claims (2)

[Claims] 1. A means for recognizing a human existing in a real space and an object existing around the real space and detecting a change in position and shape, and for each state of a human and an object, an object and an object, and an object itself. Means for logically calculating a situation; a database describing changes in the shape of the object, actions on objects in the vicinity of the object, and rules when the object exists in space; and information from the means for detecting. A logical information processing device that generates logical information of a space related to a person existing in the real space and an object existing around the human based on the information from the calculating unit and the information from the database, Based on the generated logical information of the space, a virtual object generated corresponding to the human and the object is recognized using a display device. Characteristic presence communication device. 2. A means for recognizing a human existing in a real space and an object existing around the real space and detecting a change in position and shape, and for each state of a human and an object, an object and an object, and an object itself. Means for logically calculating a situation; a database describing changes in the shape of the object, actions on objects in the vicinity of the object, and rules when the object exists in space; and information from the means for detecting. A real information processing device having a logical information processing device for generating logical information of space relating to a person existing in the real space and an object existing therearound based on the information from the calculating means and the information from the database; For the sensation communication device, the presence communication device transmits the generated logical information of the space to one or more other presence communication devices on a network, And, in the received presence communication device, according to the state of the space existing corresponding to the received presence communication device,
A realistic communication system, wherein a virtual object is generated in an appropriate form according to the received logical information of the space.