JP3982664B2 - Image presentation device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image presentation apparatus used in a field such as Tele-Existence, and in particular, an image is projected using an object of an arbitrary shape as a screen, and an object displayed by the image exists as an object. The present invention relates to a device used to interactively present a position.
[0002]
[Prior art]
Conventionally, when an operator cannot directly operate an object because it is far away or in a dangerous place, a robot controlled to move following the movement of the operator is arranged, and the operator moves the robot. Indirect manipulation of the object is performed. As a result, the operator can perform precise work as if he were in the environment where the robot was placed. Such a system is called a teleexistence system, a teleexistence type robot system, or telerobotics.
[0003]
For example, suppose that a doctor cares for a patient such as a sick person or an injured person who is far away. The patient is lying on the bed, and a robot for nursing is arranged on the side. The robot is controlled to move following the movement of the doctor. A camera is provided at the position of the robot's eyes, and the patient is photographed by the camera.
[0004]
In the cockpit, the doctor operates to perform nursing care for the patient while looking at the image of the patient by using the virtual reality or computer visualization technology. The robot operates according to the movement of the doctor, and the actual nursing of the patient is performed by the robot. In this case, the doctor is a robot operator, and the care recipient is a robot observer.
[0005]
[Problems to be solved by the invention]
Thus, humanoid robots have been developed for the purpose of nursing care and communication. However, although the size and shape of robots have become closer to humans, their appearance is very different from humans.
[0006]
Therefore, in order to generate a facial expression on such a humanoid robot, it has been proposed to mount a CRT on the robot's head and display the head image by the CRT. However, in this case, since the shape of the face is determined by the shape of the CRT, the expression becomes unnatural except when viewed from the front. In addition, since the display area by CRT is not large, it is practically difficult to have facial expressions other than the face.
[0007]
In addition, parts such as eyes, eyebrows, or mouth that can be moved mechanically are mounted on the face of the robot, and they are controlled and moved in order to reproduce realistic facial expressions. Research is underway. However, it has also been pointed out that if the machine part is halfway in reproducing a facial expression, it is counterproductive by reminiscent of a cooler facial expression.
[0008]
As described above, when an operator tries to work on a remote nurse (human), using the tele-existence system makes the operator feel as if he is in a remote location. It is possible to experience a sense of reality. However, the appearance of a robot, which can be said to be an operator, is different from that of an operator. Therefore, when the robot is viewed from a nurse to be worked on by the robot, it is still a robot, and the appearance of the operator cannot be observed in the robot.
[0009]
Incidentally, there is a device that moves a cared person from a bed by a device such as a lift at a site such as nursing. However, many care recipients are thought to have anxiety about being treated only by machines and have psychological resistance.
[0010]
The present invention has been made in view of the above-described problems, and an object thereof is to visually realize fine shapes, colors, textures, and the like that are difficult to express depending on mechanical parts.
[0011]
[Means for Solving the Problems]
As shown in FIG. 1, the apparatus according to the invention of claim 1At the position observed from the observer 21A position corresponding to the position of the robot 23 coated with the retroreflective material RU, the control means 15 for controlling the robot 23 so as to follow the movement of the operator 11, and the observer 21 observing the robot 23. Imaging means 14 for taking an image of the operator 11 from above, and projection means 22 for projecting the image of the operator 11 from the position of the observer 21 toward the robot 23.
[0012]
Here, the observer 21 may be a robot instead of a human. In other words, for example, the robots face each other, one robot is controlled to move following the movement of the operator, and the other robot is controlled to move following the movement of another observer. May be.
[0013]
As shown in FIG. 2, the apparatus according to the second aspect of the present invention includes a second robot 13B that can be observed by the operator 11 and coated with a retroreflective material, and the second robot 13B. A second control means 25B that controls to move following the movement of the observer 21 and a second image that captures an image of the observer 21 from a position corresponding to the position of the operator 11 that observes the second robot 13B. 2 imaging means 24B, and second projection means 12B for projecting the image of the observer 21 from the position of the operator 11 toward the second robot 13B.
[0014]
In the apparatus according to the invention of claim 3, the photographing means 14 is provided at a position corresponding to the viewpoint of the observer 21 who observes the robot 23, and the second photographing means 24B is the second robot 13B. Means for preventing the light of the image projected from the projection means 22 toward the robot 23 from entering the second photographing means 24B, and a position corresponding to the viewpoint of the operator 11 who observes Means are provided for preventing the image light projected from the second projection means 12B toward the second robot 13B from entering the photographing means 14B.
[0015]
An apparatus according to the invention of claim 4The control means 15 is configured to control the operation of the robot 23 on the basis of the motion detection signal of the operator 11.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
[First Embodiment]
FIG. 1 is a block diagram showing a configuration of an image presentation device 1 according to the first embodiment of the present invention.
[0017]
In FIG. 1, the image presentation device 1 is configured by two cockpits CP <b> 1 and CP <b> 2 that are separated from each other. One cockpit CP1 has an operator 11 and the other cockpit CP2 has an observer 21. Here, it is assumed that the observer 21 is a nurse and the operator 11 is a nurse such as a doctor. That is, the observer 21 is in a situation where the operator 11 is indirectly nursed (see FIG. 3).
[0018]
The operator 11 wears a sensor for detecting the movement. As such a sensor, for example, a head position sensor, a data glove, or a data suit is used. Moreover, it is also possible to use a three-dimensional measuring device that measures the state of the operator 11 in a non-contact manner as a sensor. The output signals of these sensors become control signals by the position / posture control unit 15 and are sent to the robot 23 in the cockpit CP2.
[0019]
The operator 11 can observe an image (video) presented by the video presentation device 12. As the video presentation device 12, a CAVE type display is preferably used. However, it is also possible to use a display similar to the HMP 22, a head mounted display (HMD), or other types of video presentation devices. Image information sent from the head camera 24 of the cockpit CP2 is output to the image presentation device 12 via the image processing unit 16, and an image of the observer 21 is presented to the operator 11.
[0020]
A moving camera 14 is installed in front of the operator 11. The moving camera 14 is a known binocular stereoscopic camera with six degrees of freedom. Although the moving camera 14 moves according to the position and posture of the observer 21, the observer 21 does not move so much, and therefore a normal tele-distance camera is sufficient. The moving camera 14 takes an image (video) of the operator 11 in real time and sends the image information to the cockpit CP2. Although not shown, the moving camera 14 is provided with a speaker for outputting sound.
[0021]
Although not shown, the operator 11 or the mobile camera 14 is provided with a microphone for detecting voice, and the detected voice information is sent to the robot 23 in the cockpit CP2.
[0022]
On the other hand, the observer 21 wears a sensor for detecting the movement. As such a sensor, a sensor similar to that used in the operator 11 can be used. However, since the observer 21 moves less than the operator 11, a simple sensor may be in time. The output signals of these sensors become control signals by the position / posture control unit 25 and are sent to the moving camera 14 in the cockpit CP1.
[0023]
The observer 21 further wears an HMP (head mounted projector) 22. Based on the image information output from the image processing unit 26, the HMP 22 projects an image (video) forward from a position optically conjugate with the viewpoint of the observer 21. When the HMP 22 with a position sensor is used, it can be replaced with the above-described sensor. An example of the structure of HMP22 is described in Kawakami et al., “Research on Object Oriented Display” (Information Processing Society of Japan Vol. 98, No. 9 P79-84). The contents of the image projected from the HMP 22 will be described later.
[0024]
Further, the cockpit CP2 is provided with a humanoid robot 23 having a shape similar to a human appearance such as a head, a chest, an arm, and a leg and capable of moving similar to a human movement. ing. On the surface of the robot 23, a retroreflecting material RU is applied to a portion observed by the observer 21, that is, a portion where an image is projected by the HMP 22, for example, the face, head, shoulder, arm, etc. of the robot 23. ing. The robot 23 is controlled by a control signal from the position / posture control unit 15 so as to follow the movement of the operator 11. That is, the robot 23 is controlled to be in the same state as the position and posture of the operator 11.
[0025]
A head camera 24 is provided on the head of the robot 23 at the position of the eyes on the face. The head camera 24 takes an image of the direction in which the robot 23 faces in real time, and sends the image information to the image processing unit 16 of the cockpit CP1. Usually, the observer 21 is photographed by the head camera 24. When the field of view is wide, the robot 23 itself, for example, its arms, hands, legs, etc. are photographed.
[0026]
Although not shown, the robot 23 is provided with a speaker for outputting sound. The observer 21 or the robot 23 is provided with a microphone for detecting sound, and the detected sound information is sent to the moving camera 14 in the cockpit CP1.
[0027]
Next, an image (video) projected from the HMP 22 toward the robot 23 will be described.
[0028]
The moving camera 14 photographs the operator 11, but the photographing position with respect to the operator 11 has the same relationship as the viewpoint position of the observer 21 with respect to the robot 23, that is, the projection position of the HMP 22. Further, since the robot 23 performs the same operation as the operator 11, when the robot 23 is viewed from the observer 21, the operator 11 is in a state of being replaced with the robot 23. However, although the approximate shape of the robot 23 is close to that of the operator 11, the appearance is greatly different. Therefore, the image of the operator 11 is projected onto the robot 23 by the HMP 22, and the same expression and texture as the operator 11 are given to the robot 23.
[0029]
That is, when viewing the robot 23 from the observer 21, an image for making the robot 23 appear to be the operator 11 itself is projected from the HMP 22. Moreover, the image changes according to the movement and expression of the operator 11. As a result, fine shapes, colors, textures, and the like that are difficult to express depending on mechanical parts can be visually realized, and the expression of the operator 11 can be reproduced on the robot 23. Even when the position of the operator 11 or the observer 21 is changed, the respective positions and postures are detected, and images to be projected from the HMP 22 are generated according to the respective positions and postures. For this purpose, a known image processing technique such as an image-based rendering method can be used for the content of the image processing performed in the image processing unit 26 itself.
[0030]
The image projected from the HMP 22 is reflected by the retroreflecting material RU applied to the surface of the robot 23 and presents a clear image (video) to the observer 21. Here, the retroreflecting material RU is a material that reflects with a strong directivity in the direction of the light source, and is realized using a corner cube array or glass beads. Therefore, the image projected on the robot 23 has high brightness in the line-of-sight direction of the observer 21. The observer 21 can observe the image projected on the robot 23 with extremely high luminance.
[0031]
It should be noted that when the operator 11 is photographed by the moving camera 14, there are portions that cannot be photographed by the moving camera 14 due to blind spots due to arms or clothes. The image of the part that has become the blind spot cannot be projected by the HMP 22, and therefore the image of that part is not displayed on the robot 23. However, since the blind spot as viewed from the moving camera 14 becomes a blind spot as viewed from the observer 21, there is no problem. Regarding the image presentation method described above, Japanese Patent Application No. 10-172722 proposed by the present inventors can be referred to.
[0032]
Next, the operation of the image presentation device 1 configured as described above will be described.
[0033]
The operator 11 performs an operation for nursing the observer 21 while watching the video of the observer 21 presented by the video presentation device 12. The operation of the operator 11 becomes the operation of the robot 23 as it is, and the observer 21 is cared for by the robot 23.
[0034]
An image of the operator 11 is captured by the moving camera 14, and the image is projected onto the robot 23 by the HMP 22. The voice of the operator 11 is also captured by the microphone, and a sound effect is added and emitted from the speaker of the robot 23.
[0035]
The observer 21 receives nursing care by the robot 23. Since the image of the operator 11 is projected on the robot 23, the robot 23 appears to be the operator 11 itself. That is, the observer 21 can observe the expression of the operator 11 in detail. As described above, the observer 21 receives nursing care while looking at the operator 11 instead of the robot 23 and talking with the operator 11. Therefore, when observing the robot 23 from the observer 21, it is possible to read the figure of the operator 11 and the movement of fine expressions there, to relieve anxiety of receiving nursing by the machine, and to reduce psychological resistance. Can do.
[0036]
The operator 11 can also provide care from a distance while naturally transmitting the white movement that is an advantage of teleexistence, so that the same effect as performing care at the site can be obtained.
[0037]
The position / posture control units 15 and 25 and the image processing units 16 and 26 may be disposed in any cockpit CP1 or 2, or may be disposed in a place other than the cockpit CP1 or 2. They can be configured as separate devices, or can be configured as a single device as a whole. Further, for example, the position / posture control unit 15 can be incorporated into the HMP 12B or the robot 23, and the image processing unit 16 can be incorporated into the robot 23 or the video presentation device 12.
[Second Embodiment]
FIG. 2 is a block diagram showing the configuration of the image presentation device 1B according to the second embodiment of the present invention, and FIG. 3 is a block diagram schematically showing the configuration of the image presentation device 1B. Note that, in the image presentation device 1B of the second embodiment, components corresponding to the components of the image presentation device 1 of the first embodiment are denoted by the same reference numerals, and description thereof is omitted or simplified. . The same applies hereinafter.
[0038]
In the image presentation device 1B, a robot 13B similar to the robot 23 is also provided in the cockpit CP1. A retroreflecting material RU is applied to the surface of the robot 13B. The robot 13B is controlled by a control signal from the position / posture control unit 15B so as to follow the movement of the observer 21. That is, the robot 13B is controlled to be in the same state as the position and posture of the observer 21.
[0039]
Instead of the moving camera 14 of the first embodiment, a head camera 14B incorporated in the head of the robot 13B is used. The head camera 14B takes an image of the direction in which the robot 13B is directed in real time, and sends the image information to the image processing unit 26B of the cockpit CP2.
[0040]
Moreover, it replaces with the image | video presentation apparatus 12 of 1st Embodiment, and HMP12B is used. The operator 11 wears the HMP 12B on the head and projects an image from the HMP 12B toward the robot 13B.
[0041]
That is, the HMP 12B projects an image forward from a position optically conjugate with the viewpoint of the operator 11 based on the image information output from the image processing unit 16B. This makes it appear to the operator 11 that the robot 13B is the observer 21 itself.
[0042]
Thus, in the image presentation device 1B of the second embodiment, the operator 11 cares for the robot 13B that has the same posture as the observer 21. Since the image of the observer 21 is projected on the robot 13B, it feels like nursing the actual observer 21. Therefore, when the operator 11 lifts the observer 21, nursing can be performed more realistically.
[0043]
In the image presentation apparatus 1B described above, the operator 11 wears clothes and gloves coated with the retroreflecting material RU, and projects the image from the HMP 12B onto his clothes and gloves, thereby the position of the robot 23B or The posture can be confirmed.
[0044]
That is, for example, when the operator 11 looks at his / her hand, the robot 23B operates to look at the hand in the same manner, so the hand of the robot 23B is taken by the head camera 24B, and the image is taken from the HMP 12B. 11 projected onto the hand. The operator 11 can confirm the state of the hand of the robot 23B by viewing the image projected on his / her hand.
[0045]
Thereby, the operator 11 can know the temporal or positional deviation between the movement of his / her hand and the movement of the hand of the robot 23B, and perform more precise work by feeding it back on his / her own. Is possible.
[0046]
By the way, if the light of the image projected from the HMP 12B is directly incident on the head camera 14B, it may exceed the range of brightness that can be taken by the head camera 14B. In this case, the head camera 14B is in a so-called “dimmed” state and cannot perform normal photographing. The same applies to the relationship between the HMP 22B and the head camera 24B. In order to prevent this, the light of the image projected from the HMP 12B may be prevented from entering the head camera 14B. To do so, there are the following measures.
[0047]
In the first measure, the image processing unit 16B sets the image of the portion corresponding to the head camera 14B in the image information output to the HMP 12B as a black image. For this purpose, a predetermined region in the image is processed as a black region based on signals input to the position / posture control units 15B and 25B or control signals output from them.
[0048]
In the second measure, a liquid crystal shutter or the like is provided in front of the HMP 12B to selectively block the light of the part that is about to enter the head camera 14B. In the third measure, polarizing filters having different polarization angles are arranged in front of the HMP 12B and in front of the head camera 14B.
[0049]
When using the first policy and the second policy, the position of the eyes of the observer 21 is such that the image of the eyes of the observer 21 is projected onto the robot 13B, that is, the facial expression of the observer 21 can be understood. And the position of the eyes of the robot 13B (position where the head camera 14B is disposed) need to be shifted.
[Third Embodiment]
FIG. 4 is a block diagram showing a configuration of an image presentation device 1C according to the third embodiment of the present invention.
[0050]
In the image presentation device 1C of the third embodiment, three cockpits CP1 to CP3 are provided. Two robots 13C and 23C are arranged in the third cockpit CP3. In addition to the head cameras 14C and 24C, HMPs 18C and 28C are mounted on the robots 13C and 23C, respectively. The HMPs 18C and 28C project images onto the partner robots 23C and 13C.
[0051]
In the second cockpit CP1, the operator 11 is photographed by the moving camera 17C. The image of the operator 11 obtained by the moving camera 17C is projected by the HMP 18C through the image processing unit 19C.
[0052]
In the third cockpit CP3, the operator 21 is photographed by the moving camera 27C. The image of the operator 21 obtained by the moving camera 27C is projected by the HMP 28C through the image processing unit 29C.
[0053]
The operation of the image presentation device 1C described above is basically the same as that of the image presentation device 1B. However, the images of the robots 13C and 23C in which the counterpart operators 11 and 21 are performing tele-distance, respectively, Are presented by the video presentation devices 12C and 22C of the operators 11 and 21. Accordingly, the operators 11 and 21 can perform the tele-distance while watching the opponents performing the tele-distance by the robots 13C and 23C, respectively, and can share the real world and experience it.
[Fourth Embodiment]
FIG. 5 is a block diagram showing a configuration of an image presentation device 1D according to the fourth embodiment of the present invention.
[0054]
The image presentation device 1D of the fourth embodiment is basically the same as the image presentation device 1C of the third embodiment, but differs in the following points. That is, the robots 20D and 30D are also provided in the cockpits CP1 and CP2. A retroreflecting material RU is applied to the surfaces of the robots 20D and 30D. The robots 20D and 30D are controlled by control signals from the position / posture control units 25D and 15D so as to follow the movements of the operators 21 and 11.
[0055]
Instead of the mobile cameras 17C and 27C of the third embodiment, head cameras 17D and 27D incorporated in the heads of the robots 20D and 30D are used. Further, HMPs 12D and 22D are used instead of the video presentation devices 12C and 22C.
[0056]
In the image presentation device 1D of this embodiment, the images of the robots 13D and 23D in which the counterpart operators 11 and 21 perform tele-distance are projected onto the robots 20D and 30D in front of the operators 11 and 21, respectively. The operators 11 and 21 perform tele-distance while viewing images projected on the robots 20D and 30D.
[Fifth Embodiment]
FIG. 6 is a block diagram showing a configuration of an image presentation device 1E according to the fifth embodiment of the present invention.
[0057]
In the image presentation device 1E of the fifth embodiment, the robot 13E is controlled by the robot control unit 42E. At that time, a signal indicating the position and posture of the robot 13E is fed back to the robot controller 42E.
[0058]
The graphic engine 41E generates a CG image (computer graphics image) based on the respective positions and postures of the robot 13E and the operator 11. The generated CG image is projected from the HMP 12E toward the robot 13E.
[0059]
According to the image presentation device 1E, an arbitrary image generated by the graphic engine 41E can be projected onto the robot 13E.
[0060]
The operation and effect of each embodiment described above can also be expressed as follows. That is, of the images observed by the operator 11 and the observer 21, a rough shape is created by the robot, and a fine portion is created by the image projected from the HMP, thereby increasing the reality. This makes it possible for the operator 11 or the observer 21 to make physical contact or work with the outside world using the robot, and to make mental contact using the image projected on the robot.
[0061]
This can also be viewed as a real world implementation of texture mapping in computer graphics.
[0062]
In the embodiment described above, the robot is controlled so as to move in real time following the movement of the operator 11 and the observer 21, but the movement of the operator 11 or the observer 21 is captured in advance, You may make it move a robot according to reproduction | regeneration. Further, not only the actual images of the operator 11 and the observer 21 but also a CG image may be superimposed. Although a humanoid robot is used as the robot, the present invention is not limited to this. For example, the present invention can also be applied to a pet robot or a manipulator robot in which only an arm extends from a wall.
[0063]
further, PaintingThe configuration, arrangement, number, operation content or operation sequence, operation timing, and operation timing of the whole or each part of the image presentation devices 1 and 1B to 1E can be appropriately changed in accordance with the spirit of the present invention.
[0064]
【The invention's effect】
According to the present invention, it is possible to visually realize fine shapes, colors, textures, and the like that are difficult to express depending on mechanical parts.
[0065]
Therefore, for example, when nursing is received by a robot of the telexistence system, the anxiety of receiving nursing by a machine can be relieved and psychological resistance can be reduced.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an image presentation device according to a first embodiment of the present invention.
FIG. 2 is a block diagram showing a configuration of an image presentation apparatus according to a second embodiment of the present invention.
FIG. 3 is a block diagram schematically illustrating a configuration of an image presentation apparatus according to a second embodiment.
FIG. 4 is a block diagram illustrating a configuration of an image presentation apparatus according to a third embodiment of the present invention.
FIG. 5 is a block diagram illustrating a configuration of an image presentation device according to a fourth embodiment of the present invention.
FIG. 6 is a block diagram illustrating a configuration of an image presentation device according to a fifth embodiment of the present invention.
[Explanation of symbols]
1,1B-E Image presentation device
11 Operator (Observer)
12B, 12D, 12E HMP (projection means, second projection means)
13B to 13E Robot (second robot, object)
14 Moving camera (photographing means)
14B Head camera (photographing means, second photographing means)
15 Position / attitude control unit (control means)
15B-13D Position / attitude control unit (control means, second control means)
16B to 16D Image processing unit (means for preventing)
17C moving camera (photographing means, second photographing means)
17D head camera (photographing means, second photographing means)
21 Observer (operator)
22 HMP (projection means)
22B, 22D HMP (projection means, second projection means)
23 Robot (object)
23B-23D Robot (second robot, object)
24B head camera (photographing means, second photographing means)
25B-23D Position / attitude control unit (control means, second control means)
26B-26D Image processing part (means for preventing)
27C Moving camera (photographing means, second photographing means)
27D head camera (photographing means, second photographing means)
41E graphic engine (image generation means)
42E Robot controller (control means)
RU retroreflective material

Claims (4)

A robot retroreflective material applied to the position to be observed from the observer, and a control controlling means to move to follow the robot to the movement of the operator, corresponding to the position of the observer who observes the robot An image presentation apparatus comprising: an imaging unit that captures an image of the operator from a position to be projected; and a projection unit that projects the image of the operator from the position of the observer toward the robot. A second robot that can be observed by the operator and coated with a retroreflecting material; and a second control unit that controls the second robot to move following the movement of the observer; A second imaging means for capturing an image of the observer from a position corresponding to the position of an operator observing the second robot; and an image of the observer from the position of the operator toward the second robot. The image presentation apparatus according to claim 1, further comprising: a second projecting unit that projects. The imaging means is provided at a position corresponding to the viewpoint of an observer observing the robot, and the second imaging means is provided at a position corresponding to the viewpoint of an operator observing the second robot, Means for preventing the light of the image projected from the projection means toward the robot from entering the second photographing means, and the image projected from the second projection means toward the second robot. The image presentation apparatus according to claim 2 , further comprising means for preventing light from entering the photographing means. The image presenting apparatus according to claim 1, wherein the control unit is configured to control an operation of the robot based on a detection signal of the operator's movement .

Images