JP2000218575A - Image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To permit visual display of image with fine shape, color, quality or the like by providing a control means to control a robot applied with coating of recurrent reflective material to follow up movement of an operator, an image picking up means for picking up an image of an operator from a position corresponding to a position of an observer, and a projecting means. SOLUTION: After an operator 11 wears a sensor for detecting movement of the operator 11, e.g. a head position sensor, data glove, data suit or the like, a sensor signal is outputted to a robot applied with coating of recurrent reflective material through a position and posture controller 15B. The operator 11 observes image information outputted from a head camera 24B of the robot 23B through an image processor 16B, by means of an image display device 12B. Also, after an observer 21 wears a similar sensor, a sensor signal is outputted to the robot 13B on the side of the operator 11 to move the robot 13B in almost the similar manner to movement of the observer 21, and image information from the head camera 24B is displayed at the device 12B.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image presenting apparatus used in the field of tele-existence and the like, and in particular, projects an image using an object of an arbitrary shape as a screen and displays the image. The present invention relates to a device used to interactively present an object at a position where an object exists.

[0002]

2. Description of the Related Art Conventionally, when an operator cannot directly operate an object due to being located far away or in a dangerous place, a robot that is controlled to move following the movement of the operator is arranged. In some cases, an operator indirectly operates an object via a robot. Thus, the operator can perform a precise operation as if he were in the environment where the robot is arranged. Such a system is called a telexistence system, a telexistence type robot system, or telerobotics.

[0003] For example, suppose that a doctor cares for a care recipient such as a sick or injured person who is far away. The care recipient is lying on a bed, on the side of which a nursing robot is located. The robot is controlled to follow the doctor's movement. A camera is provided at the position of the eyes of the robot, and the care recipient is photographed by the camera.

[0004] The doctor operates in the cockpit to provide care to the care recipient while viewing the care recipient's image by the technique of virtual reality (virtual reality) or computer visualization (computer visualization). The robot operates according to the operation of the doctor, and the robot performs actual nursing of the care recipient. In this case, the doctor is the operator of the robot and the care recipient is the observer of the robot.

[0005]

As described above, a humanoid robot has been conventionally developed for the purpose of nursing care, communication, and the like. However, although the size and shape of the robot have become closer to humans, their appearance is significantly different from humans.

In order to generate an expression on such a humanoid robot, a CRT is mounted on the head of the robot.
It has been proposed to display a head image on a CRT. However, in this case, since the shape of the face is determined by the shape of the CRT, the expression becomes unnatural except when viewed from the front. In addition, since the display area by the CRT is not large, it is practically difficult to have an expression in a part other than the face.

In addition, mechanically movable parts such as eyes, eyebrows, and mouth are mounted on the face of the robot, and the parts are finely controlled and moved, thereby reproducing a realistic expression. Attempts are being made to research.
However, it has been pointed out that if the mechanical parts or the like are halfway in reproducing the facial expression, it is counterproductive rather than reminiscent of a colder facial expression.

[0008] As described above, when an operator attempts to work on a care recipient (human) in a remote place, by using the teleexistence system, the operator himself is as if he is in a remote place. It is possible to experience a sense of realism like this. However, the robot, which can be said to be an alter ego of the operator, is different from the operator. Therefore, when the robot is viewed from the care recipient who is approached by the robot, it is still a robot, and the operator cannot be observed in the robot.

[0009] Incidentally, there is a device for moving a care receiver from a bed by a device such as a lift at the site of care or the like. However, many caregivers are said to have anxiety about being handled only by machines and have psychological resistance.

The present invention has been made in view of the above-described problems, and has as its object to visually realize fine shapes, colors, textures, and the like that are difficult to express depending on mechanical parts.

[0011]

As shown in FIG. 1, the apparatus according to the first aspect of the present invention comprises a robot 23 coated with a retroreflective material RU and an operator 11
Control means 15 for controlling to move following the movement of
Photographing means 14 for photographing the image of the operator 11 from a position corresponding to the position of the observer 21 observing the robot 23;
And projection means 22 for projecting the image of the operator 11 toward the camera.

Here, the observer 21 may be a robot instead of a human. That is, for example, the robots are configured to face each other, to control one robot to follow the movement of the operator, and to control the other robot to follow the movement of another observer. You may.

As shown in FIG. 2, the apparatus according to the second aspect of the present invention includes a second robot 13B which can be observed by the operator 11 and is coated with a retroreflective material. A second control means 25B for controlling the movement of the observer 21 in accordance with the movement of the observer 21, and an image of the observer 21 from a position corresponding to the position of the operator 11 observing the second robot 13B. A second imaging unit 24B for imaging, and a second projection unit 12B for projecting an image of the observer 21 from the position of the operator 11 toward the second robot 13B.

In the apparatus according to a third aspect of the present invention, the photographing means is provided at a position corresponding to a viewpoint of an observer observing the robot and the second photographing means is provided.
4B is provided at a position corresponding to the viewpoint of the operator 11 observing the second robot 13B, and light of an image projected from the projection unit 22 toward the robot 23 is transmitted to the second imaging unit 24B. Means are provided for preventing light from entering, and means for preventing light of an image projected from the second projecting means 12B toward the second robot 13B from entering the photographing means 14B.

According to a fourth aspect of the present invention, an object 23 coated with a retroreflective material RU, a means 15 for deforming the object 23, and an image corresponding to the deformed shape of the object 23 are generated. Image generating means 26 and the observer 21
And projecting means 22 for projecting the video from the position toward the object 23.

[0016]

DESCRIPTION OF THE PREFERRED EMBODIMENTS [First Embodiment] FIG. 1 is a block diagram showing the configuration of an image presentation device 1 according to a first embodiment of the present invention.

In FIG. 1, the image presentation device 1 is composed of two cockpits CP1 and CP2 that are located apart from each other. An operator 11 is in one cockpit CP1, and an observer 21 is in the other cockpit CP2. Here, it is assumed that the observer 21 is a care recipient and the operator 11 is a nurse such as a doctor. That is, the observer 21 is in a state of being indirectly cared for by the operator 11 (see FIG. 3).

The operator 11 is wearing a sensor for detecting the movement. For example, a head position sensor, a data glove, a data suit, or the like is used as such a sensor. Further, a three-dimensional measuring device that measures the state of the operator 11 in a non-contact manner can be used as the sensor. The output signals of these sensors become control signals by the position / posture control unit 15 and are sent to the robot 23 in the cockpit CP2.

The operator 11 can observe an image (video) presented by the video presentation device 12. As the video presentation device 12, a CAVE type display is preferably used. However, a display similar to the HMP22 or a head mounted display (HMD: He
ad Mounted Display) and other types of video presentation devices are also possible. The image information sent from the head camera 24 of the cockpit CP2 is transmitted to the image processing unit 16
The image of the observer 21 is output to the image presentation device 12 via the.

In front of the operator 11, the mobile camera 1
4 are installed. The moving camera 14 is a known six-degree-of-freedom binocular stereoscopic camera. The moving camera 14 moves according to the position and the posture of the observer 21, but since the observer 21 does not move so much, a normal camera for telexistence is sufficient. The moving camera 14
An image (video) of the operator 11 is photographed in real time, and the image information is sent to the cockpit CP2. Although not shown, the mobile camera 14 is provided with a speaker for outputting sound.

Although not shown, the operator 11
Alternatively, the moving camera 14 is provided with a microphone for detecting sound, and the detected sound information is transmitted to the robot 23 in the cockpit CP2.

On the other hand, the observer 21 is wearing a sensor for detecting the movement. As such a sensor, the same sensor as that used in the operator 11 can be used. However, since the observer 21 moves less than the operator 11, there may be a case where a simple sensor can be used. Output signals of these sensors become control signals by the position / posture control unit 25 and are sent to the mobile camera 14 in the cockpit CP1.

The observer 21 further wears an HMP (head mounted projector) 22. The HMP 22 projects an image (video) forward from a position optically conjugate with the viewpoint of the observer 21 based on the image information output from the image processing unit 26. When a HMP 22 having a position sensor is used, the HMP 22 can be replaced with the sensor described above. Regarding the structure of HMP22, see Kawakami et al., “Research on Object-Oriented Display” (Information Processing Society of Japan Vol. 98, No. 9).
P79-84) describes one example. In addition,
The contents of the image projected from the HMP 22 will be described later.

Further, the head,
A humanoid robot 23 having a shape similar to a human appearance, such as a chest, arms, and legs, and capable of performing a motion similar to a human motion is provided. On the surface of the robot 23,
In particular, the portion observed by the observer 21, that is, HMP2
The retroreflective material RU is applied to a portion where an image is projected by the device 2, such as the face, head, shoulder, and arm of the robot 23. The robot 23 is controlled by a control signal from the position / posture control unit 15 to move following the movement of the operator 11. That is, the robot 23 is controlled to be in the same state as the position and the posture of the operator 11.

A head camera 24 is provided on the head of the robot 23 at the position of the eyes on the face. Head camera 2
4 captures an image in the direction of the robot 23 in real time, and sends the image information to the image processing unit 16 of the cockpit CP1. Usually, the observer 21 is photographed by the head camera 24. When the field of view is wide, the robot 23 itself, for example, its arms, hands, and legs are photographed.

Although not shown, the robot 23 is provided with a speaker for outputting sound. Also,
The observer 21 or the robot 23 is provided with a microphone for detecting sound, and the detected sound information is transmitted to the moving camera 14 in the cockpit CP1.

Next, an image (video) projected from the HMP 22 toward the robot 23 will be described.

The moving camera 14 photographs the operator 11, and the photographing position for the operator 11 has the same relationship as the viewpoint position of the observer 21 with respect to the robot 23, that is, the projection position of the HMP 22. Further, since the robot 23 performs the same operation as the operator 11, when the observer 21 looks at the robot 23, it is as if the operator 11 has been replaced with the robot 23. However, although the approximate shape of the robot 23 is close to that of the operator 11, its appearance is greatly different. Therefore, the image of the operator 11 is projected onto the robot 23 by the HMP 22 to give the robot 23 the same expression and texture as the operator 11.

That is, when the robot 23 is viewed from the observer 21, an image for projecting the robot 23 as if it were the operator 11 itself is projected from the HMP 22. In addition, the image changes according to the movement and expression of the operator 11. This makes it possible to visually realize fine shapes, colors, textures, and the like that are difficult to express depending on mechanical parts.
Can be reproduced on the robot 23. Even when the position of the operator 11 or the observer 21 changes, the respective positions and postures are detected, and an image to be projected is generated from the HMP 22 according to the respective positions and postures. For this purpose, a known image processing technique such as an image-based rendering technique can be used for the content of the image processing performed in the image processing unit 26 itself.

The image projected from the HMP 22 is reflected by the retroreflective material RU applied to the surface of the robot 23, and presents a clear image (video) to the observer 21.
Here, the retroreflective material RU is a material that reflects with high directivity in the direction of the light source, and is realized using a corner cube array, glass beads, or the like. Therefore, the image projected on the robot 23 has high luminance in the direction of the line of sight of the observer 21. The observer 21 can observe the image projected on the robot 23 with extremely high luminance.

Note that the operator 1 is
When photographing 1, there is a portion that cannot be photographed by the moving camera 14 due to a blind spot due to an arm, clothing, or the like. The image of the blind spot cannot be projected by the HMP 22, and therefore the image of
3 is not displayed. However, there is no problem since the blind spot viewed from the mobile camera 14 becomes a blind spot viewed from the observer 21. Regarding the image presentation method described above, reference can be made to Japanese Patent Application No. 10-172722 proposed by the present inventors.

Next, the operation of the image presentation apparatus 1 configured as described above will be described.

The operator 11 looks at the image of the observer 21 presented by the image
Perform the operation for nursing. The operation of the operator 11 becomes the operation of the robot 23 as it is,
Thus, the nursing of the observer 21 is performed.

An image of the operator 11 is captured by the moving camera 14, and the image is projected to the robot 23 by the HMP 22. The voice of the operator 11 is also captured by the microphone, and a sound effect is added to the voice to be emitted from the speaker of the robot 23.

The observer 21 receives nursing care by the robot 23.
Is projected, the robot 23 appears to be the operator 11 itself. That is, the observer 21 can closely observe the expression of the operator 11 and the like. In this way, the observer 21 can read the robot 23
Instead, he / she receives nursing while looking at the operator 11 and talking with the operator 11. Therefore, when observing the robot 23 from the observer 21, it is possible to read the figure of the operator 11 and the movement of the fine expression there, to alleviate the anxiety of receiving nursing by a machine and to reduce psychological resistance. Can be.

It is to be noted that the operator 11 can also provide remote care while naturally transmitting the white movement, which is an advantage of telexistence, and thus has the same effect as going to the site and providing care.

The position / posture control units 15 and 25 and the image processing units 16 and 26 are provided with any of the cockpits CP1 and CP2.
2 and may be arranged in a place other than the cockpits CP1 and CP2. They may be configured as separate devices from each other, or may be configured as one device as a whole. Also, for example,
HMP12B or robot 23
And the image processing unit 16 can be incorporated into the robot 23 or the video presenting device 12. [Second Embodiment] FIG. 2 is a block diagram showing the configuration of an image presentation device 1B according to a second embodiment of the present invention, and FIG. 3 is a block diagram schematically showing the configuration of the image presentation device 1B. In the image presentation device 1B of the second embodiment, the same reference numerals are given to components corresponding to the components of the image presentation device 1 of the first embodiment, and the description thereof will be omitted or simplified. . The same applies hereinafter.

In the image presentation device 1B, a robot 13B similar to the robot 23 is provided in the cockpit CP1. The retroreflective material RU is applied to the surface of the robot 13B. The robot 13B has the observer 21
Position / posture control unit 15B so as to follow the movement of
Is controlled by a control signal from That is, the robot 13B is controlled to be in the same state as the position and the posture of the observer 21.

The head camera 1 incorporated in the head of the robot 13B in place of the mobile camera 14 of the first embodiment
4B is used. The head camera 14B is the robot 13
An image in the direction of B is captured in real time, and the image information is sent to the image processing unit 26B of the cockpit CP2.

The image presentation device 12 according to the first embodiment
Instead, HMP12B is used. Operator 11
Wears the HMP12B on the head and projects an image from the HMP12B toward the robot 13B.

That is, the HMP 12B has the image processing unit 1
Operator 1 based on the image information output from 6B
An image is projected forward from a position optically conjugate to the first viewpoint. Thereby, for the operator 11,
It looks as if the robot 13B is the observer 21 itself.

As described above, in the image presentation device 1B of the second embodiment, the operator 11 cares for the robot 13B having the same posture as the observer 21. Robot 13B
, The image of the observer 21 is projected, so that it feels like nursing the actual observer 21. Therefore, when the operator 11 lifts the observer 21,
Nursing can be performed even more realistically.

In the above-described image presentation apparatus 1B, the operator 11 wears clothes or gloves coated with the retroreflective material RU, and projects the image from the HMP 12B onto his or her clothes or gloves, whereby the robot 23B Can be confirmed.

That is, for example, when the operator 11 looks at his or her own hand, the robot 23B operates to look at the hand in the same manner, so that the hand of the robot 23B is photographed by the head camera 24B and the image is taken. HMP
The image is projected from 12B onto the hand of the operator 11. The operator 11 can check the state of the hand of the robot 23B by looking at the image projected on his own hand.

Thus, the operator 11 can know the temporal or positional deviation between the movement of his / her hand and the movement of the hand of the robot 23B. Can be performed.

When the light of the image projected from the HMP 12B is directly incident on the head camera 14B, the light may exceed the range of brightness that can be photographed by the head camera 14B. Then, the head camera 14B is in a so-called “dazzling” state, and cannot perform normal photographing.
The same applies to the relationship between the HMP 22B and the head camera 24B. In order to prevent this, the light of the image projected from the HMP 12B may be prevented from entering the head camera 14B. There are the following measures for that.

As a first measure, the image processing section 16B sets the image of the portion corresponding to the head camera 14B in the image information output to the HMP 12B as a black image. For this purpose, a process of setting a predetermined area in an image as a black area is performed based on signals input to the position / posture control units 15B and 25B or control signals output therefrom.

In the second measure, a liquid crystal shutter or the like is provided in front of the HMP 12B to selectively block light from entering the head camera 14B. The third measure is H
A polarization filter having a different polarization angle is arranged in front of the MP 12B and in front of the head camera 14B.

When using the first policy and the second policy, the observer's 21 eyes are projected on the robot 13B, that is, the observer's 21 facial expressions are understood. It is necessary to shift the position of the eyes from the position of the eyes of the robot 13B (the position where the head camera 14B is arranged). [Third Embodiment] FIG. 4 is a block diagram showing the configuration of an image presentation device 1C according to a third embodiment of the present invention.

In the image presentation device 1C of the third embodiment,
Three cockpits CP1 to CP3 are provided. The third cockpit CP3 includes two robots 13C and 23C.
Is deployed. Each of the robots 13C and 23C has an HMP18C, in addition to the head cameras 14C and 24C.
28C is mounted. The HMPs 18C and 28C project images to each other's robots 23C and 13C.

In the second cockpit CP1, the operator 11 is photographed by the moving camera 17C. The image of the operator 11 obtained by the moving camera 17C is projected by the HMP 18C via the image processing unit 19C.

In the third cockpit CP3, the operator 21 is photographed by the moving camera 27C. The image of the operator 21 obtained by the moving camera 27C is projected by the HMP 28C via the image processing unit 29C.

The operation of the image presenting apparatus 1C described above is basically the same as that of the image presenting apparatus 1B, except that the images of the robots 13C and 23C in which the opponent operators 11 and 21 are performing teleexistence, respectively. Are presented by the video presenting devices 12C and 22C of the operators 11 and 21, respectively. Therefore, the operators 11 and 21
Can perform telexistence while looking at the opponents performing telexistence by the robots 13C and 23C, respectively, to share the real world and to experience the real world. [Fourth Embodiment] FIG. 5 is a block diagram showing the configuration of an image presentation device 1D according to a fourth embodiment of the present invention.

The image presentation device 1D of the fourth embodiment is basically the same as the image presentation device 1C of the third embodiment, but differs in the following points. That is, cockpit CP1,
2, robots 20D and 30D are also provided. The retroreflective material RU is applied to the surfaces of the robots 20D and 30D. The robots 20D and 30D are connected to the operator 2
The movement is controlled by control signals from the position / posture control units 25D and 15D so as to follow the movements of the movements 1 and 11.

The mobile cameras 17C and 27 according to the third embodiment
Instead of C, head cameras 17D and 27D incorporated in the heads of the robots 20D and 30D are used. Also, instead of the video presenting devices 12C and 22C, HMP12D, 2
2D is used.

In the image presenting apparatus 1D of this embodiment, the images of the robots 13D and 23D whose telecommunication operators 11 and 21 perform teleexistence are displayed by the robots 20D and 30D in front of the operators 11 and 21 respectively.
Projected to The operators 11 and 21 are
Tele-existence is performed while viewing the images projected on D and 30D. [Fifth Embodiment] FIG. 6 is a block diagram showing the configuration of an image presentation device 1E according to a fifth embodiment of the present invention.

In the image presentation device 1E of the fifth embodiment,
The robot 13E is controlled by a robot control unit 42E. At this time, a signal indicating the position and orientation of the robot 13E is fed back to the robot control unit 42E.

The graphic engine 41E generates a CG image (computer graphics image) based on the respective positions and postures of the robot 13E and the operator 11. The generated CG image is projected from the HMP 12E toward the robot 13E.

According to the image presentation device 1E, an arbitrary image generated by the graphic engine 41E can be transferred to the robot 13E.
Can be projected.

The operation and effect of each embodiment described above are as follows.
It can also be expressed as: That is, the operator 11
Of the image observed by the observer 21 and the like, a rough shape is created by a robot,
From the image projected from the camera to increase reality. As a result, it is possible for the operator 11 or the observer 21 to make physical contact with or work with the outside world by using a robot, and to perform mental contact using an image projected on the robot.

This can be viewed as a real-world implementation of texture mapping in computer graphics.

In the above embodiment, the robot is controlled so as to move in real time following the movements of the operator 11 and the observer 21. The robot may be moved in accordance with the reproduction of the video. Further, not only real images of the operator 11 and the observer 21 but also CG images may be superimposed. Although a humanoid robot is used as the robot, the present invention is not limited to this, and may be applied to, for example, a pet robot or a manipulator-type robot in which only an arm extends from a wall.

Further, the present invention can be applied to a mechanical device that is not a robot or an object that is not a mechanical device and that is deformed by various actions or principles. In addition, the configuration, arrangement, number, operation content or operation sequence, operation timing, and the like of the whole or each part of the image presentation devices 1, 1B to E can be appropriately changed in accordance with the gist of the present invention.

[0064]

According to the present invention, fine shapes, colors, textures, etc., which are difficult to express depending on mechanical parts, can be realized visually.

Therefore, for example, when nursing is performed by a robot of the telexistence system, the anxiety of receiving nursing by a machine can be relieved.
Psychological resistance can be reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image presentation device according to a first embodiment of the present invention.

FIG. 2 is a block diagram illustrating a configuration of an image presentation device according to a second embodiment of the present invention.

FIG. 3 is a block diagram schematically illustrating a configuration of an image presentation device according to a second embodiment.

FIG. 4 is a block diagram illustrating a configuration of an image presentation device according to a third embodiment of the present invention.

FIG. 5 is a block diagram illustrating a configuration of an image presentation device according to a fourth embodiment of the present invention.

FIG. 6 is a block diagram illustrating a configuration of an image presentation device according to a fifth embodiment of the present invention.

[Explanation of symbols]

1, 1B-E Image presentation device 11 Operator (observer) 12B, 12D, 12E HMP (projection means, second projection means) 13B-13E Robot (second robot, object) 14 Moving camera (photographing means) 14B Head camera (imaging means, second imaging means) 15 Position / posture control section (control means) 15B-13D Position / posture control section (control means, second control means) 16B-16D Image processing section (prevention means) 17C Moving camera (photographing means, second photographing means) 17D Head camera (photographing means, second photographing means) 21 Observer (operator) 22 HMP (projecting means) 22B, 22D HMP (projecting means, second photographing means) 23B (object) 23B-23D Robot (second robot, object) 24B Head camera (photographing means, second photographing means) 25B 23D Position / posture control unit (control means, second control means) 26B to 26D Image processing unit (prevention means) 27C Moving camera (photographing means, second photographing means) 27D Head camera (photographing means, second photographing means) 41E Graphic engine (image generating means) 42E Robot controller (control means) RU retroreflective material

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Yasuyuki Yanagida 1-2-1405 Midoricho, Tanashi-shi, Tokyo East Otama No.2 Dormitory (72) Inventor Taro Maeda 1-219 Yanaka, Taito-ku, Tokyo No. (72) Inventor Tachi ▲ Susumu ▼ 2-31-14 Umezono, Umezono, Tsukuba City, Ibaraki Prefecture 3F059 AA10 DA01 DA05 DB03 EA00 4C341 LL30 5C054 CE11 CF08 FA00 FD02 HA12

Claims (4)

[Claims] 1. A robot to which a retroreflective material is applied, control means for controlling the robot to move following an operator's movement, and a position corresponding to a position of an observer observing the robot. An image presentation device comprising: a photographing unit that photographs an image of an operator; and a projecting unit that projects an image of the operator from the position of the observer toward the robot. 2. A second robot which can be observed by the operator and is coated with a retroreflective material, and a second robot which controls the second robot to move following the movement of the observer. Control means, and second photographing means for photographing the image of the observer from a position corresponding to the position of the operator observing the second robot, and moving from the position of the operator toward the second robot. The image presentation device according to claim 1, further comprising: a second projection unit configured to project an image of an observer. 3. The photographing means is provided at a position corresponding to a viewpoint of an observer observing the robot, and the second photographing means is provided at a position corresponding to a viewpoint of an operator observing the second robot. Means for preventing light of an image projected from the projection means toward the robot from being incident on the second photographing means, and from the second projection means toward the second robot. The image presentation device according to claim 3, further comprising means for preventing light of the projected image from being incident on the photographing means. 4. An object coated with a retroreflective material, means for deforming the object, image generating means for generating an image corresponding to the deformed shape of the object, and the object from the position of the observer. An image presentation device, comprising: projection means for projecting the video toward.