JP2022147921A - Pseudo contact presentation system, control program and control method - Google Patents

Abstract

To enable providing a new pseudo contact presentation system that presents pseudo contact.SOLUTION: A pseudo contact presentation system (10) comprises: a communication robot 12; and a HMD 20, displays a plurality of virtual agents (70) on the HMD, and displays an animation in which predetermined actions are individually or simultaneously performed. An arm of a robot corresponding to a virtual agent is operated in connection with the predetermined action. In other words, the pseudo contact presentation system presents a visual stimulus to a user 60 wearing the HMD, and presents a contact stimulus associated with this. Thus, the user obtains a body feeling in which he/she is contacting with the virtual agent. In other words, pseudo contact is presented to the user.SELECTED DRAWING: Figure 4

Description

TECHNICAL FIELD The present invention relates to a pseudo contact presentation system, a control program and a control method, and in particular, for example, a pseudo contact presentation system that uses an effector that presents a stimulus by contact to a user and a head-mounted display that presents a stimulus by sight to the user. The present invention relates to a contact presentation system, a control program and a control method.

An example of this type of conventional pseudo-contact presentation system is disclosed in Patent Document 1. According to the pseudo contact presentation system disclosed in Patent Literature 1, a communication robot presents a contact stimulus to a user by, for example, hugging the user with an arm long enough to be wrapped around the user's back. An HMD (Head Mounted Display) worn on the user's head displays a virtual agent whose state changes substantially in synchronization with the hugging motion of the communication robot. A contact stimulus is given to the user by the communication robot hugging the user, and a visual stimulus is given to the user by the virtual agent of the HMD.

JP 2019-45928

In the pseudo contact presentation system disclosed in the above Patent Document 1, one virtual agent is assigned to one communication robot, and the action of the virtual agent is synchronized with the action of the communication robot. In other words, the configuration for presenting the user with a stimulus by contact using the communication robot and the virtual agent is simple. In Patent Document 1, no consideration is given to presenting stimulation to the user by contact of a plurality of virtual agents.

SUMMARY OF THE INVENTION Therefore, a primary object of the present invention is to provide a novel pseudo-touch presentation system, control program and control method.

Another object of the present invention is to provide a simulated contact presentation system, control program and control method capable of providing complex contact stimulation.

A first invention provides a contact stimulus presentation device comprising a plurality of stimulus presentation units that act on a user to provide a contact stimulus to the user, and a virtual An assigning means for assigning agents, and a plurality of virtual agents worn on the user's head and assigned by the assigning means are displayed, and some or all of the plurality of virtual agents are operated to present visual stimuli to the user. The contact stimulus presentation device is provided with a head-mounted display device for displaying two or more stimulus presentation units among the plurality of stimulus presentation units in relation to the actions of some or all of the plurality of virtual agents. It is a pseudo contact presentation system that operates each.

A second invention is according to the first invention, wherein the visual stimulus presentation device displays an animation showing a user's action for some or all of the plurality of virtual agents, the stimulus presentation section includes a movable section, The contact stimulus presenting unit presents the contact stimulus to the user by operating the movable part such that the movement of the movable part changes in relation to the animation of the actions of some or all of the plurality of virtual agents.

A third aspect of the present invention provides a pseudo stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide contact stimuli to the user and a head-mounted display device that is worn on the user's head. A control program executed by a computer of a contact presentation system, the computer controlling a movable section control section for controlling a plurality of stimulus presentation sections of a contact stimulus presentation device to present contact stimuli to a user, and a plurality of stimulus presentation sections. functioning as allocation means for allocating a virtual agent to each of two or more of the stimulus presentation units, and a display unit for displaying a part or all of the plurality of virtual agents allocated by the allocation means; Visual stimuli are presented to the user by displaying them on the head-mounted display device, and a plurality of stimuli are presented in relation to the actions of some or all of the plurality of virtual agents by the movable part control unit. A sensory stimulus presentation program for operating each of two or more stimulus presentation sections of the sections.

A fourth aspect of the present invention provides a pseudo stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide contact stimuli to the user and a head-mounted display device that is worn on the user's head. A control method for a contact presentation system, comprising: controlling a plurality of stimulus presentation units of a contact stimulus presentation device to present a contact stimulus to a user; and displaying some or all of the assigned plurality of virtual agents, the plurality of virtual agents being displayed on a head-mounted display to provide a visual stimulus to the user. and causing the contact stimulus presentation device to operate each of two or more stimulus presentation units among the plurality of stimulus presentation units in relation to the operation of some or all of the plurality of virtual agents; control method.

According to the present invention, some or all of a plurality of virtual agents displayed on a head-mounted display mounted on the user's head are caused to act, and in relation to the operation, the plurality of virtual agents Since the stimulus presentation unit corresponding to some or all of them is operated, it is possible to present the visual stimulus by the action of the virtual agent and, in relation to this, to present the contact stimulus by the stimulus presentation unit. In other words, it is possible to obtain the sensation of being in contact with a virtual agent. Therefore, it is possible to present a new pseudo-contact presentation system, control program, and control method for presenting pseudo-contact.

The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings.

FIG. 1 is a block diagram showing the electrical configuration of a pseudo touch presentation system according to one embodiment of the present invention. FIG. 2 is a front view of the external configuration of the robot shown in FIG. 3 is a block diagram showing an electrical configuration of the robot shown in FIG. 1. FIG. FIG. 4 is a side view of an example of a state in which the pseudo contact presentation system shown in FIG. 1 is used. FIG. 5 is a diagram showing an example of two virtual agents displayed on the HMD shown in FIG. FIG. 6 is a diagram showing an example of three virtual agents displayed on the HMD shown in FIG. FIG. 7 is a diagram showing another example of two virtual agents displayed on the HMD shown in FIG. FIG. 8 is a diagram showing another example of three virtual agents displayed on the HMD shown in FIG. FIG. 9 is a diagram showing an example of a memory map of a RAM built into the server shown in FIG. FIG. 10 is a flowchart showing a part of an example of first pseudo-contact processing of a CPU built in the server shown in FIG. 1; 11 is another part of the first pseudo-contact processing of the CPU built in the server shown in FIG. 1, and is a flowchart subsequent to FIG. 10. FIG. FIG. 12 is a flowchart showing a part of an example of second pseudo-contact processing of a CPU built in the server shown in FIG. 1; 13 is another part of the second pseudo-contact processing of the CPU built in the server shown in FIG. 1, and is a flowchart subsequent to FIG. 12. FIG.

Referring to FIG. 1, a pseudo-tactile presentation system (hereinafter referred to as "system") 10 of this embodiment provides a user (person or participant) with a physical or bodily stimulus, that is, a user's sense of touch. It includes a robot 12 for presenting a stimulus to be given or a stimulus given to the user by contact (hereinafter referred to as "contact stimulus"). In this embodiment, the robot 12 is a communication robot that includes actions that present contact stimuli and, if necessary, generates sounds.

FIG. 2 shows an example of the external configuration of the robot 12 as a contact stimulus presentation device used in the embodiment of FIG. 1, viewed from the front. However, any tactile stimulus presentation device with other appearances and structures can be used.

The robot 12 includes three rod-shaped frames 120a, 120b, and 120c. Frames 120a, 120b, and 120c may be made of metal such as aluminum, or resin such as fiber-reinforced plastic or engineering plastic, and are basically designed with cross-sectional shapes and thicknesses that provide safe strength. For example, a prismatic or tubular frame may be utilized.

The frame 120a constitutes the bottom of the robot 12, extends horizontally and is installed on the floor or floor surface. The frame 120b constitutes the body of the robot 12, is arranged to extend in the vertical direction, and has its lower end fixed to the center or approximately the center of the frame 120a. The frame 120c constitutes the shoulder of the robot 12, extends horizontally, and is fixed to the top of the frame 120b.

Frame 120c is provided with robot arms (hereinafter simply referred to as "arms") 122, 124 and 126 extending from opposite ends thereof. Arms 122, 124 and 126 each have a structure imitating a human arm and have joints corresponding to elbow joints and shoulder joints, respectively. Therefore, in the following description, the names of parts of the human arm may be used for convenience.

Arms 122 and 124 are provided at the left end of frame 120c when robot 12 is viewed from the front, and arm 126 is provided at the right end of frame 120c when robot 12 is viewed from the front. Hereinafter, when the robot 12 and a portion thereof are described using directions, the top, bottom, left, and right when the robot 12 is viewed from the front will be used.

The arm 122 includes a frame 122a corresponding to the forearm and a frame 122b corresponding to the upper arm, a joint (corresponding to the elbow joint) 122c connecting the frame 122a and the frame 122b, and connecting the frame 122b and the frame 120c. It includes a joint (corresponding to the shoulder joint) 122d.

The arm 124 includes a frame 124a corresponding to the forearm and a frame 124b corresponding to the upper arm, a joint (corresponding to the elbow joint) 124c connecting the frame 124a and the frame 124b, and connecting the frame 124b and the frame 120c. It includes a joint (corresponding to the shoulder joint) 124d.

However, as can be seen from FIG. 2, the arm 122 is connected to the lower surface of the left end of the frame 120c, and the arm 124 is connected to the upper surface of the left end of the frame 120c.

The arm 126 includes a frame 126a corresponding to the forearm and a frame 126b corresponding to the upper arm, a joint (corresponding to the elbow joint) 126c connecting the frame 126a and the frame 126b, and connecting the frame 126b and the frame 120c. It includes a joint (corresponding to the shoulder joint) 126d. This arm 126 is connected to the lower surface of the right end of the frame 120c.

As an example, joints 122c, 124c and 126c can each open the forearm up to 90° relative to the upper arm. Thus, by controlling the angles of joints 122c, 124c and 126c, arms 122, 124 and 126 can be bent and extended, respectively.

Also, by way of example, joints 122d, 124d and 126d, respectively, can open the upper arm up to 150° with respect to the plane to which each joint 122d, 124d and 126d is coupled with frame 120c. However, the angle when the frames 122b, 124b and 126b constituting the upper arm are perpendicular to the plane where the joints 122d, 124d and 126d are connected to the frame 120c is 0°. Thus, by controlling the angle of joints 122d, 124d and 126d, arms 122, 124 and 126 can be raised and lowered or extended and retracted, respectively.

Additionally, this embodiment incorporates actuators at joints 122c, 122d, 124c, 124d, 126c and 126d so that robot 12 can shake hands with user 60 (see FIG. 4) at each of arms 122, 124 and 126. .

A contact stimulus presentation device, that is, a robot 12 is connected to a server 16 via a network 14 such as the Internet or a telephone communication line. The server 16 is a general-purpose computer such as a PC, PDA, smart phone, or tablet terminal.

Although not shown, the server 16 includes input devices such as a touch panel and/or keyboard, and an operator can operate such input devices. Programs and data for controlling such operations of the server 16 are stored in a memory (for example, HDD and RAM) 16b built in the server 16, and the overall operation of the server 16 is controlled by a similarly built-in CPU 16a. movement is controlled. This CPU 16a controls the motion and speech of the robot 12 through the CPU 36 (described later) of the robot 12, and further controls the display of a virtual agent by the HMD 20 described later, according to the operator's operation described above. For this reason, the server 16 has a communication module 16c, and the CPU 16a is communicably connected to the network 14, that is, the robot 12 and the HMD 20 via the communication module 16c by wire or wirelessly.

The system 10 of FIG. 1 also includes an HMD 20, which is a head-mounted display device, and this HMD 20 is worn on the head 64 of the user 60 who receives contact stimuli presented by the robot 12 (see FIG. 4). In the embodiment, an Oculus Rift (trade name) manufactured by Oculus is used as the HMD 20, and an Oculus sensor (trade name) is used to accurately detect the position of the HMD 20, that is, the head position of the user 60. The position detection device 18a was used.

That is, the server 16 is provided with the output of the position detection device 18a. The position detection device 18a functions as a head position measurement unit, and the Oculus sensor used as the position detection device 18a tracks the position of the HMD 20 (Oculus Rift) by combining a gyro sensor and acceleration. , the position of the head 64 of the user 60 can be detected with high accuracy.

As described above, the robot 12 is fixed to the floor, and by setting or registering the initial position of the HMD 20 with respect to the robot 12, the position of the HMD 20 in real space, that is, the head position of the user 60 can be detected. Therefore, in this embodiment, the CPU 16a uses the positional relationship between the head positions of the robot 12 and the user 60 in the real space to control the positions of the virtual agent and the viewpoint (virtual camera) in the virtual space. Since the HMD 20 displays a so-called first-person viewpoint image (virtual agent in this embodiment), the position of the virtual camera corresponds to the head position of the user 60 in the virtual space. Also, the gaze point of the virtual camera is set at a predetermined position of the robot 12 in the virtual space. For example, the predetermined position is set at the center of the robot 12 in virtual space. This is an example, and other positions may be adopted as long as virtual agents of various sizes fit within the image displayed on the HMD 20 .

However, the robot 12 itself is not drawn in the virtual space, and the position of the robot 12 is used when arranging the virtual agent in the virtual space or determining the position of the virtual camera in the virtual space. Only. For example, the position of the robot 12 (the position of the center of the frame 120a) is set to the origin in the three-dimensional virtual space, the direction parallel to the frames 120a and 120c of the robot 12 is set to the X-axis direction, and the direction parallel to the frame 120b is set. The direction perpendicular to both the X and Z axes is set to the direction of the Y axis.

Connected to the server 16 are a microphone 18b for collecting environmental sounds and the voice of the user 60, a camera 18c for acquiring environmental images and images of the user 60, and the like. The voice data collected by the microphone 18b is input to the server 16, and the server 16 can perform voice recognition using the voice data. Image data from camera 18c is input to server 16, which can process the image data, for example, for personal identification.

In this embodiment, robot 12 is enabled to interact with user 60 . For this reason, the memory 16b of the server 16 is preset with topics necessary for dialogue with the user 60 and scenarios corresponding to the topics. However, a remote operator may also interact with user 60 through server 16 as previously described.

The HMD 20 described above can display a high-resolution stereoscopic image with a 110° field of view at a refresh rate of 90 Hz, for example.

The HMD 20 displays virtual agents 70 (70a, 70b, 70c) as shown in FIGS. This gives a stimulus (visual stimulus) to the user's 60 vision and/or hearing.

Therefore, the memory 16b of the server 16 is preset with image data of a plurality of virtual agents 70 of different sexes and ages. The image data includes not only data of a still image in which the virtual agent 70 stands upright, but also data of animation when performing predetermined actions (greeting, shaking hands and walking in this embodiment).

Then, after identifying the user 60 who cooperates (interacts) with the robot 12, the server 16 selects a virtual agent 70 whose gender and age match the identified user 60, and transfers the image data to the network 14. to the HMD 20 via the This allows the HMD 20 to display the virtual agent 70 that matches the user 60 . However, virtual agent 70 may be registered by user 60 .

In this embodiment, robot 12 may be assigned multiple virtual agents 70 . In this embodiment, the number of virtual agents 70 assigned to robot 12 is less than or equal to the number of arms of robot 12 . Accordingly, one, two or three virtual agents 70 are assigned corresponding to the three arms 122 , 124 and 126 of the robot 12 .

The user 60 can decide how many virtual agents 70 to assign to the robot 12 . However, when applying this system 10 to an arbitrary application such as a game, it can be fixedly or dynamically determined according to the scenario of the application.

If one virtual agent 70 is assigned to robot 12 , one virtual agent 70 is assigned to arms 122 and 126 . If two virtual agents 70 are assigned to robot 12 , one virtual agent 70 of the two is assigned to arm 122 and the other virtual agent 70 of the two is assigned to arm 126 . If three virtual agents 70 are assigned to robot 12 , one of the three virtual agents 70 is assigned to arm 122 and one of the remaining two virtual agents 70 is assigned to arm 124 . and the other virtual agent 70 of the remaining two is assigned to arm 126 .

However, this is an example and does not need to be limited. Also, the plurality of assigned virtual agents 70 are displayed on the HMD 20 so as not to overlap each other.

It should be noted that the case where one virtual agent 70 is assigned to the robot 12 is not the essential content of the present invention, and is disclosed in Japanese Patent Application Laid-Open No. 2019-45928, which was previously filed by the applicant and has already been published. The detailed description is omitted because it overlaps with the contents described above.

Although not shown in FIG. 2, a speaker and a microphone are attached to the upper part of the robot 12 (for example, the upper end of the frame 120b), and the robot 12 speaks to the user 60 and listens to the speech of the user 60. can be XIMERA (ATR Spoken Language Laboratory: trade name) was adopted as speech synthesis software for speech.

FIG. 3 is a block diagram showing an example of the electrical configuration of the robot 12. As shown in FIG. As shown in FIG. 3, the robot 12 includes a CPU 36 that controls the robot 12 as a whole. CPU 36 is connected to communication module 40 through bus 38 and thus CPU 36 is communicatively connected via communication module 40 to network 14 or server 16, either by wire or wirelessly.

The CPU 36 can also access the memory 42 through the bus 38, and according to the programs and data set in this memory 42, give command values to the actuator control circuit 44 through the bus 38 to control the operation of each actuator A1-An. The actuator control circuit 44 generates the number of pulse powers according to the command value given from the CPU 36, and gives them to the corresponding stepping motors to drive the actuators A1-An.

However, as the actuator, in addition to the one using such a stepping motor, any actuator such as an actuator using a servomotor or a fluid actuator can be used.

Here, actuators A1-An are actuators (stepping motors) of joints 122c, 122d, 124c, 124d, 126c and 126d. Therefore, the actuator control circuit 44 also functions as a joint control section.

A sensor I/F (interface) 46 is connected to the CPU 36 via the bus 38 and receives outputs from the tactile sensor 48 and the camera 50 respectively.

The tactile sensor 48 is, for example, a touch sensor and forms part of the tactile sense of the robot 12 . In other words, the tactile sensor 48 detects whether or not a person or other object has touched the robot 12 . For example, tactile sensors 48 are provided on the forearms (frames 122a, 124a and 126a) of arms 122, 124 and 126 of robot 12, respectively, to detect whether a hand of user 60 touches arms 122, 124 and 126 of robot 12. can do.

The tactile sensor 48 also functions as a pressure sensor. As an example, such a tactile sensor 48 is formed of, for example, a polymer piezo sensor sheet. Such a tactile sensor 48 can detect pressure applied to the forearms (frames 122a, 124a, 126a) of the robot 12 that shake hands with the user 60 .

Output (detected data) from the tactile sensor 48 is given to the CPU 36 via the sensor I/F 46 . Therefore, the CPU 36 can detect that a human or other object has touched the robot 12 (and its strength or pressure) and that the human or other object has left the robot 12 .

Furthermore, pressure data detected by the tactile sensor 48 is output from the CPU 36 and provided to the server 16 via the communication module 40 and the network 14 .

The camera 50 is an image sensor and forms part of the vision of the robot 12 . That is, the camera 50 is used to detect the image or images seen by the eyes of the robot 12 . In this embodiment, data (image data) corresponding to an image (moving or still image) captured by camera 50 is provided to CPU 36 via sensor I/F 46 . CPU 36 not only detects changes in the captured image, but also transmits the image data to server 16 via communication module 40 and network 14 (FIG. 1). The server 16 then outputs the received image data to a monitor (not shown). Therefore, the image captured by the camera 50 is displayed on the monitor.

Also, the speaker 54 and the microphone 56 are connected to the input/output I/F 52 . The speaker 54 outputs sound when the robot 12 speaks. When an operator of the server 16 or an operator (hereinafter sometimes referred to as a “remote operator”) speaks directly, the voice is output through the network 14 , the communication module 40 and the input/output I/F 52 . Specifically, when the remote operator speaks through microphone 56 , corresponding voice data is provided from server 16 to CPU 36 via network 14 . The CPU 36 then outputs the audio data from the speaker 54 via the input/output I/F 52 .

The microphone 56 is a sound sensor and forms part of the robot's 12 sense of hearing. The microphone 56 has directivity and is mainly used to detect the voice of the user 60 (person) who interacts (communicates) with the robot 12 .

The memory 42 of the robot 12 is, for example, RAM and HDD, and is preset with a speech recognition program and a speech synthesis program such as XIMERA described above. The speech recognition program is a program for the CPU 36 to recognize the contents of what a person has said to the robot 12, which is input through the microphone 56. The CPU 36 performs, for example, DP matching and Hidden Markov Model (HMM). recognizes the content of human utterances.

The CPU 36 recognizes speech input through the microphone 56 by the remote operator according to its speech recognition program. For example, an utterance can be detected using an acoustic model such as the above-mentioned HMM method or a deep neural network (DNN).

In addition to outputting the remote operator's voice directly from the speaker 54, the robot 12 can be made to utter synthetic voice. The CPU 36 outputs synthesized speech to the speaker 54 in accordance with the synthesized speech data given from the server 16 when an instruction to speak from the speaker 54 is input from the server 16 using synthesized speech.

Essentially, in this embodiment, all movements of the robot 12 are provided by a server 16 operated by an operator (not shown). Therefore, the CPU 36 supplies sensor data acquired through the sensor I/F 46 and audio data and image data acquired through the input/output I/F 52 to the CPU 16 a of the server 16 from the communication module 40 via the network 14 . The server 16 can obtain those data and know the state of the robot 12 .

However, server 16 can also autonomously control the motion of both virtual agent 70 and robot 12 using position and pressure information from position sensing device 18a.

FIG. 4 shows an example of a side view of a user 60 using the system 10. As shown in FIG. As shown in FIG. 4 , the user 60 wears the HMD 20 and stands in front of the robot 12 . That is, the robot 12 and the user 60 face each other. Robot 12 and user 60 can interact or converse. Also, when the user 60 approaches the robot 12 and the distance d between the robot 12 and the user 60 becomes equal to or less than the predetermined distance df, the robot 12 can touch the user 60 with the arms 122 , 124 and 126 . In this case, user 60 can touch at least one of arms 122 , 124 and 126 of robot 12 . As an example, the predetermined distance df is 70 cm−1 m and is determined by the lengths of arms 122 , 124 and 126 .

As noted above, a robot 12 may be assigned multiple virtual agents 70 . When two virtual agents 70 are assigned to the robot 12 , one virtual agent 70 is positioned to overlap the arm 122 of the robot 12 and the other virtual agent 70 is positioned to overlap the arm 126 of the robot 12 . be done. In the example shown in FIG. 5, virtual agents 70a and 70b are placed in the virtual space such that virtual agent 70a's left arm overlaps arm 122 and virtual agent 70b's right arm overlaps arm 126. FIG. That is, when two virtual agents 70 are assigned to robot 12, arm 122 of robot 12 corresponds to the left arm of virtual agent 70a and arm 126 of robot 12 corresponds to the right arm of virtual agent 70b.

Also, when three virtual agents 70 are assigned to the robot 12, one of the three virtual agents 70 is placed in a position overlapping the arm 122 of the robot 12, and one of the remaining two The virtual agent 70b is placed at a position overlapping the arm 126 of the robot 12, and the other virtual agent 70c of the remaining two is placed at a position overlapping the arm 124 of the robot 12. FIG. In the example shown in FIG. 6, the right arm of the virtual agent 70a is positioned to overlap the arm 122 of the robot 12, the right arm of the virtual agent 70b is positioned to overlap the arm 126 of the robot 12, and the left arm of the virtual agent 70c is positioned to overlap the arm 126 of the robot. It is arranged in a position overlapping twelve arms 124 . That is, when three virtual agents 70 are assigned to robot 12, arm 122 of robot 12 corresponds to the right arm of virtual agent 70a, arm 124 of robot 12 corresponds to the left arm of virtual agent 70c, and robot 12 corresponds to the left arm of virtual agent 70c. corresponds to the right arm of the virtual agent 70b.

Therefore, the CPU 16a of the server 16 causes each of the plurality of virtual agents 70 to individually perform a first predetermined action on the user 60 (person or participant) 60 approaching the robot 12. Also, in synchronism with the virtual agent 70 performing a first predetermined action on the user 60, a process of operating the arm 122, 124 or 126 to which the virtual agent 70 is assigned (hereinafter referred to as "first pseudo-contact process ) can be executed. However, the movement of arm 122, 124 or 126 is movement according to the first predetermined movement.

In addition, the CPU 16a of the server 16 brings the plurality of virtual agents 70 closer to the user 60 in the virtual space, and the plurality of virtual agents 70 performs the second predetermined action on the user 60 at the same time. A process (hereinafter referred to as a "second pseudo-contact process") for operating arms 122 and 124 or arms 122, 124 and 126 to which a plurality of virtual agents 70 are assigned can be executed. However, the movements of arms 122, 124 and 126 are movements according to a second predetermined movement.

In this example, the first predetermined action and the second predetermined action are handshakes. As an example, the first pseudo-contact process is executed in a situation where the user 60 individually shakes hands with a plurality of idle virtual agents 70 at a handshake meeting venue set in the virtual space. On the other hand, the second pseudo-contact process is executed in a situation where the user 60 is idle, surrounded by a plurality of virtual agents 70 who are fans, and each virtual agent 70 requests a handshake.

Therefore, which of the first pseudo-contact process and the second pseudo-contact process is to be executed is determined by the operator of the server 16 or the remote operator, or determined by the scenario (context) of the application to which the system 10 is applied. or determined by However, the application scenario may be variably determined depending on which option the user 60 chooses.

For example, in the first pseudo-contact process, the user 60 interacts with one, two, or three virtual agents 70 assigned to the robot 12, and when the user 60 approaches a predetermined distance df at which the robot 12 can shake hands, the server The CPU 16a of 16 sends a signal to both the virtual agent 70 and the robot 12 to cause the HMD 20 to display the handshake animation of the virtual agent 70 and to cause the robot 12 to start shaking hands. That is, a contact action is performed on the user 60 . Accordingly, the user 60 can grasp the arm 122, 124 or 126 of the robot 12 to shake hands and get the sensation of shaking hands with the virtual agent 70 assigned to the grasping arm 122, 124 or 126. In other words, a pseudo contact is presented to the user 60 . However, among the plurality of virtual agents 70, the handshake animation is displayed only for the virtual agents 70 assigned to the arms 122, 124, or 126 closest to the user 60. FIG.

For example, when two virtual agents 70 are assigned to the robot 12, when the user 60 approaches the virtual agent 70a, as shown in FIG. synchronously), the arm 122 corresponding to the virtual agent 70a is operated. That is, the robot 12 uses the arm 122 to shake hands. Therefore, the user 60 holds the arm 122 of the robot 12 and shakes hands while watching the handshake of the virtual agent 70a on the HMD 20 . That is, the user 60 is presented with a visual stimulus by the animation of the virtual agent 70 displayed on the HMD 20 , and a contact stimulus by the action of the robot 12 is given to the user 60 .

Although illustration is omitted, by enabling the movement of the user's hand to be detected from the image of the camera 18c, etc., the image of the user's hand can also be displayed on the HMD 20 when the user shakes hands. However, even if the movement of the user's hand is not detected, it can be detected from the output of the tactile sensor 48 that the user's hand or the like has touched the arm of the robot 12. The image of the user's hand may be displayed on the HMD 20 when it is detected that the is touched. Hereinafter, the same applies to the case where the user shakes hands.

Also, although illustration is omitted, in the first pseudo-contact process, when three virtual agents 70 are assigned to the robot 12, similarly, the virtual agent 70 to which the user 60 (viewpoint) is closest in the virtual space handshake, and the robot 12 uses the arm 122 , 124 or 126 corresponding to the virtual agent 70 to perform the handshake motion.

In the first pseudo-contact process, when one virtual agent 70 is assigned to the robot 12, when the user 60 (viewpoint) approaches the virtual agent 70 in the virtual space, the virtual agent 70 shakes hands. , and the robot 12 uses the arm 122 or 124 to which the virtual agent 70 corresponds to perform a handshake action.

Also, in this embodiment, the degree of intimacy between the user 60 and each virtual agent 70 is set. Increased intimacy. For example, the familiarity level of the virtual agent 70 that has never shaken hands is set to 0, and the familiarity level is incremented by 1 every time handshake occurs. For example, familiarity can be used as a condition for selecting the virtual agent 70, and can be used to determine what to say when greeting. When selecting a virtual agent 70, a virtual agent 70 with high or low familiarity can be selected. In addition, it is possible to greet a user 60 with a degree of familiarity of 0 with "Nice to meet you", and greet a user 60 with a degree of familiarity of 1 or higher with "Hello" or "Long time no see". These are just examples, and affinity may be used for other purposes.

Also, in the first pseudo-contact process, at least one virtual agent 70 among the plurality of virtual agents 70 greets the user 60, and the plurality of virtual agents 70 interacts with the user 60 individually. User 60 is guided to approach robot 12 by virtual agent 70 greeting user 60 . Further, when the user 60 interacts with a plurality of virtual agents 70 individually, as an example, the plurality of virtual agents 70 speak to the user 60 in order, and when the plurality of virtual agents 70 finishes speaking, the user 60 speaks.

In this case, the purpose of this case is to allow the user 60 to simulate the motion of the virtual agent 70 through the motion of the robot 12, that is, to present a pseudo contact. You don't have to interact.

In the first pseudo-contact process, the user 60 moves away from the robot 12, waves a hand toward the robot 12, performs a specific action such as the user 60 removing the HMD 20, or when the user 60 says "Goodbye" or " When the user utters a specific content such as "bye bye", the first pseudo-contact process is terminated. This is the same when ending the second pseudo-contact process. It should be noted that removal of the HMD 20 by the user 60 can be detected based on the image captured by the camera 18c.

In the second pseudo-contact process, one, two, and three virtual agents 70 assigned to the robot 12 approach the user 60, and the user 60 interacts with one, two, or three virtual agents assigned to the robot 12. Shake hands with Agent 70. When the user 60 approaches the robot 12 within a predetermined distance df at which they can shake hands, the CPU 16a of the server 16 starts the second pseudo-contact process. This is because in the second pseudo-contact process, the virtual agent 70 is brought close to the user 60 (viewpoint) in the virtual space to shake hands, so the user 60 needs to be close enough to the robot 12 to shake hands. It is from.

In the second pseudo-contact process, the virtual agent 70 is moved closer to the user (viewpoint). Therefore, in this embodiment, the server 16 performs one or more virtual When drawing the agent 70, one or a plurality of virtual agents 70 are placed (drawn) at positions farther from the viewpoint than the positional relationship between the robot 12 and the user 60 in the real space.

When the second pseudo-contact process is started, one, two or three virtual agents 70 assigned to the robot 12 are gradually brought closer to the user 60 (virtual camera). A signal is transmitted to each of one, two or three virtual agents 70 assigned to the robot 12 and the robot 12 to cause the HMD 20 to display the handshake animation of all the virtual agents 70 assigned to the robot 12. , causing the robot 12 to initiate a handshake. That is, a contact action is performed on the user 60 . However, in the second pseudo-contact process, all the virtual agents 70 assigned to the robot 12 shake hands at the same time. An arm is operated. That is, if one virtual agent 70 and two virtual agents 70 are assigned to robot 12, arms 122 and 126 are actuated. Also, if three virtual agents 70 are assigned to robot 12, arms 122, 124 and 126 are actuated. Thus, the user 60 can grasp and shake any one or two of the arms 122, 124 and 126 of the robot 12 and all one, two or three of the arms 122, 124 and 126 assigned to the robot 12 to get the feeling of shaking hands with a virtual agent 70 of In other words, a pseudo contact is presented to the user 60 .

For example, when three virtual agents 70 are assigned to the robot 12, when the three virtual agents 70 approach the user 60 (viewpoint) in the virtual space, as shown in FIG. Agents 70 (70a, 70b, 70c) shake hands, and associated (or synchronized) arms 122, 124 and 126 corresponding to each of the three virtual agents 70 are moved. That is, robot 12 uses arms 122, 124 and 126 to shake hands. Therefore, the user 60 shakes hands by grasping one or two of the arms 122, 124 and 126 of the robot 12 while watching the three virtual agents 70 shaking hands on the HMD 20. FIG. That is, the user 60 is presented with a visual stimulus by the animation of the virtual agent 70 displayed on the HMD 20 , and a contact stimulus by the action of the robot 12 is given to the user 60 .

Although not shown, in the second pseudo-contact process, when two virtual agents 70 are assigned to the robot 12, the two virtual agents 70 also approach the user 60 (viewpoint) in the virtual space. Then, the two virtual agents 70 shake hands, and the robot 12 uses the arms 122 and 126 corresponding to the two virtual agents 70 to perform the handshake action.

In the second pseudo contact process, when one virtual agent 70 is assigned to the robot 12, when the one virtual agent 70 approaches the user 60 (viewpoint) in the virtual space, the one virtual agent The virtual agent 70 shakes hands, and the robot 12 uses the arm 122 to which the one virtual agent 70 corresponds to perform the handshake action.

Although detailed description is omitted, the robot 12 includes the tactile sensors 48 on the frames 122a, 124a, and 126a of the arms 122, 124, and 126, respectively. is shaking hands, the arms 122, 124, and 126 may be repeatedly moved up and down upon detecting that the user 60's hand or the like has touched them.

Also, the movements of the arms 122, 124, and 126 change in relation to the movements of the virtual agent 70, but the server 16 stores the movements (state transitions) of the robot 12 so that such relevance can be further enhanced. ) and the operation (state transition) of the virtual agent 70 on the HMD 20 are set.

In that program, the joint angles of the joints 122c, 122d, 124c, 124d, 126c, and 126d of the robot 12 (actuators A1-An controlling each joint are detected and fed back to the motor control) is controlled.

However, the motion of the robot 12 and the motion of the virtual agent 70 do not necessarily have to be synchronized. It has been confirmed that there is no problem even if the motion of the robot 12 and the motion of the virtual agent 70 displayed on the HMD 20 are slightly behind each other as long as there is a certain degree of lag.

When synchronizing the action of the robot 12 and the action of the virtual agent 70, the delay time of the action of the robot 12 is detected in advance, and the animation of the virtual agent 70 is reproduced in consideration of the delay time. good.

FIG. 9 shows an example of a memory map 300 of the memory (here, RAM) 16b built in the server 16. As shown in FIG. As shown in FIG. 9, memory 16b includes program storage area 302 and data storage area 304 . A program storage area 302 stores the control program of this embodiment.

The control programs include a communication program 302a, an allocation program 302b, a sensor value detection program 302c, an image generation program 302d, a robot control program 302e, and the like.

The communication program 302a is a program for communicating with an external device, the robot 12 and the HMD 20 in this embodiment, by wire or wirelessly.

The allocation program 302b is a program for allocating one or more virtual agents 70 to the robot 12 according to the number of virtual agents 70 set according to user's operation or the like.

The sensor value detection program 302c is a program for detecting outputs from various sensors (the tactile sensor 48, the camera 50 and the microphone 56 in this embodiment).

The image generation program 302d is a program for generating image data to be displayed on the HMD 20. FIG. In this embodiment, image data is generated for still and animated images of one or more virtual agents 70 assigned to robot 12 .

The robot control program 302e causes one or more virtual agents 70 assigned to the robot 12 to perform predetermined first and second actions (in this embodiment, the first and second actions are shaking hands). 122 and 124, or 122, 124 and 126) corresponding to each virtual agent 70 in synchronization with predetermined first and second motions; This is a program for transmitting voice data corresponding to the utterance content of the robot 12 to the robot 12 in response.

Although not shown, the program storage area 302 also stores other programs for controlling the robot 12 and the virtual agent 70 .

The data storage area 304 also stores sensor value data 304a, arm control data 304b, speech data 304c, and image data 304d.

The sensor value data 304a is data output from various sensors and detected according to the sensor value detection program 302c. The arm control data 304b is data for controlling the arms 122, 124 and 126 of the robot 12, specifically, data for driving the actuators.

The utterance data 304c is synthesized speech data of the content uttered by the virtual agent 70 assigned to the robot 12. FIG. However, utterance (dialogue) content can use content set in advance in a database (for example, memory 16b of server 16) or use an API (Application Programming Interface) on the web.

The image data 304d is image data for displaying the virtual agent 70 on the HMD 20, and is generated according to the image generation program 302d. However, the image data 304 d includes still image data of the virtual agent 70 and animation data of the action of the virtual agent 70 .

Although illustration is omitted, the data storage area 304 stores other data necessary for controlling the robot 12 and the virtual agent 70, and provides timers (counters) and flags.

10 and 11 are flowcharts of the first pseudo-contact processing of the CPU 16a incorporated in the server 16 shown in FIG. 12 and 13 are flow charts of the second pseudo-contact processing of the CPU 16a incorporated in the server 16 shown in FIG. The first pseudo-contact process and the second pseudo-contact process will be described below, but the same processes as those already described will be briefly described. Also, in this embodiment, the operations of the first pseudo-contact and the second pseudo-contact are executed by the CPU 16a of the server 16 according to a control program set in the memory 16b. In other words, the server 16 indirectly controls the operation of the robot 12 by sending a command signal from the CPU 16a of the server 16 to the CPU 36 of the robot 12 (FIG. 3).

As shown in FIG. 10, when the CPU 16a starts the first pseudo-contact process, in step S1, the CPU 16a detects image data from the camera 18c. identify who the

However, the personal identification of the user 60 may be performed using other means such as RFID (radio frequency identifier) instead of the image data from the camera 18c. In this case, the CPU 16a detects the RFID in step S1.

For users 60 who have not been registered in advance and whose identities could not be identified in step S1, it is possible to continue the process by provisionally assigning an anonymous ID.

In the following step S5, the number of virtual agents 70 is set. As described above, the number of virtual agents 70 can be set according to the user's operation within the number of arms provided in the robot 12, and is set to any one of 1-3 in this embodiment.

In the following step S7, it is determined whether or not the number of virtual agents 70 is one. If "YES" in step S7, that is, if the number of virtual agents 70 is 1, in step S9, one virtual agent 70 is displayed on the HMD 20, and the process proceeds to step S17 shown in FIG.

However, in step S9, an image of one virtual agent 70 selected according to the user 60 identified in step S3 is drawn at a position where the right arm overlaps the arm 122 and the left arm overlaps the arm 124. Data is sent to the HMD 20 .

In this case, the server 16 may randomly select from virtual agent candidates 70 of the same sex or the opposite sex as the user 60, or manually select a virtual agent 70 that the user 60 has pre-registered for himself/herself. You may do so.

However, a specific virtual agent 70 may be manually selected for a user 60 with an anonymous ID as described above.

When the virtual agent 70 is manually selected in this manner, for example, a selection switch (not shown) is provided on the HMD 20, and the server 16 sequentially selects images of candidates for the virtual agent 70 according to the operation of the selection switch. When the desired virtual agent 70 is displayed on the HMD 20, the operation of the selection switch may be stopped.

The distance d between the robot 12 and the user 60 is calculated based on the predetermined position of the robot 12 and the position information from the position detection device 18a. A distance of 12 is set to a distance corresponding to the distance d in the real space, and the virtual agent 70 is drawn. Also, as described above, the gaze point of the virtual camera is the center of the robot 12 in the virtual space. The same applies to the case where the virtual agent 70 is rendered to generate image data.

On the other hand, if "NO" in step S7, that is, if the number of virtual agents 70 is not one, it is determined whether or not the number of virtual agents 70 is two in step S11.

If "YES" in step S11, that is, if the number of virtual agents 70 is two, two virtual agents 70 are displayed on the HMD 20 in step S13, and the process proceeds to step S17.

However, in step S13, the two virtual agents 70 selected according to the user 60 identified in step S3 overlap the left arm of one of them with the arm 122, and the other one of them Image data in which the right arm overlaps the arm 124 and is drawn side by side is transmitted to the HMD 20 .

On the other hand, if "NO" in step S11, that is, if the number of virtual agents 70 is three, three virtual agents 70 are displayed on the HMD 20 in step S15, and the process proceeds to step S17.

However, in step S15, one of the three virtual agents 70 selected according to the user 60 identified in step S3 has its right arm overlapped with the arm 122, and one of the remaining two The right arm of one of the remaining two overlaps the arm 124, and the left arm of the other one of the remaining two overlaps the arm 126, and the image data drawn side by side is transmitted to the HMD 20.

As shown in FIG. 11, in step S17, a topic is selected and dialogue is started. At this time, the user 60 may select a dialogue theme, or a specific theme may be automatically selected based on the dialogue history of each user 60 stored in the server 16 . Also, in step S17, when starting the dialogue, synthetic speech data of the utterance content of the greeting is transmitted to the robot 12 and the HMD 20. FIG. However, instead of the greeting, the content of the utterance may bring the user 60 closer to the robot 12 or induce a dialogue. Furthermore, when a plurality of virtual agents 70 are assigned to the robot 12 , synthetic voice data of the utterance contents of each of the plurality of virtual agents 70 are transmitted to the robot 12 and HMD 20 in a predetermined order. However, this is only an example, and only the synthetic voice data of the utterance content of any one virtual agent 70 may be transmitted. However, the pitch (frequency) of the synthesized speech is changed according to the type of virtual agent 70 .

Subsequently, in step S19, all arms 122, 124 and 126 of robot 12 are returned to their initial positions. That is, the CPU 16a transmits to the robot 12 control data for returning the arms 122, 124 and 126 to their initial positions.

Subsequently, in step S21, it waits for a certain period of time (for example, 2 to 3 seconds). Here, the movements and actions of the robot 12 and all virtual agents 70 are stopped for a certain period of time. Then, in step S23, it is determined whether or not the user 60 (viewpoint) has approached the distance at which the virtual agent 70 can shake hands. However, if a plurality of virtual agents 70 are assigned to the robot 12, it is determined whether or not the robot 12 has come close enough to shake hands with any one of the virtual agents 70. FIG. That is, it is determined whether or not the distance between the user 60 and the arms 122, 124, 126 of the robot 12 to which the virtual agent 70 is assigned is equal to or less than the predetermined distance df in the real space. Specifically, the CPU 16a determines whether or not the user 60 has come close enough to shake hands with at least one of the arms 122, 124 and 126 based on the image data from the camera 50 received from the robot 12 through the network 14. do.

However, the operator may check the image data from the camera 50 on a monitor (not shown) and make a decision.

In the embodiment, the image data of the camera 50 is analyzed for the determination of step S23. User 60 position sensing systems such as laser rangefinders (Oculus Sensors), matte sensors, etc. previously described may be utilized.

If "NO" in step S23, that is, if the user 60 is not close enough to shake hands with the virtual agent 70, the process returns to step S21. On the other hand, if "YES" in step S23, that is, if the user 60 is close enough to shake hands with the virtual agent 70, the arm of the virtual agent 70 closest to the user 60 is brought to the user 60 (viewpoint) in step S25. Get close to and shake hands. Here, the CPU 16a reproduces an animation in which the arm of the virtual agent 70 closest to the user 60 approaches the direction of the viewpoint in the virtual space and handshake.

In the subsequent step S27, the arm 122, 124 or 126 of the robot 12 corresponding to the virtual agent 70 closest to the user 60 is brought close to the user 60 to shake hands. In step S27, the CPU 16a controls the actuators of the joints (122c and 122d, 124c and 124d, or 126c and 126d) through the actuator control circuit 44 to move the robot 12 corresponding to the virtual agent 70 closest to the user 60. Arm 122, 124 or 126 can be positioned at the angle of the forearm (frame 122a) relative to the upper arm (frame 122b), the angle of the forearm (frame 124a) relative to the upper arm (frame 124b), or the angle of the forearm (frame 126a) relative to the upper arm (frame 126b). With the angle set to 70°, raise (or move forward) slightly in the direction where the user 60 is.

Subsequently, in step S29, the familiarity of the virtual agent 70 is improved, and in step S31, dialogue corresponding to the topic is executed. Here, the CPU 16a executes dialogue according to the topic selected in step S17. However, by storing the conversation history, when the selected topic has been discussed in the past, non-overlapping statement contents are selected.

In addition, when the virtual agent 70 performs an action along with the speech, the arm 122, 124 or 126 to which the virtual agent 70 is assigned also operates in synchronization with the action of the virtual agent 70. FIG.

Then, in step S33, it is determined whether or not the user 60 wishes to end the dialogue. Here, when the CPU 16a detects a specific action of the user 60 as described above or an utterance about a specific content of the user 60 as described above, the CPU 16a detects that the user 60 wishes to end the dialogue. It is determined that This also applies to the process of step S81, which will be described later.

If "NO" in step S33, that is, if the user 60 does not wish to end the dialogue, the process returns to step S31 to continue the dialogue. On the other hand, if "YES" in step S33, that is, if the user 60 wishes to end the dialogue, the first pseudo-contact process is ended.

Further, as shown in FIG. 12, when the second pseudo-contact process is started, the CPU 16a detects image data from the camera 18c in step S51, and based on the image data detected in step S51 in step S53, Identify who the user 60 is.

However, in the second pseudo-contact process, an image is displayed on the HMD 20 so that the virtual agent 70 approaches the user 60. Therefore, as described above, this second pseudo-contact process enables the user 60 to shake hands with the robot 12. It is started on the assumption that the position is approached.

At subsequent step S55, the number of virtual agents 70 is set, and at step S57, it is determined whether or not the number of virtual agents 70 is one. If "YES" in step S57, one virtual agent 70 is displayed on the HMD 20 in step S59, and the process proceeds to step S67 shown in FIG.

On the other hand, if "NO" in step S57, it is determined whether or not the number of virtual agents 70 is two in step S61. If "YES" in step S61, two virtual agents 70 are displayed on the HMD 20 in step S63, and the process proceeds to step S67. On the other hand, if "NO" in step S61, three virtual agents 70 are displayed on the HMD 20 in step S65, and the process proceeds to step S67.

In the second pseudo contact process, as described above, the CPU 16a draws one or more virtual agents 70 in steps S59, S63 and S65 in order to move the virtual agent 70 closer to the user (viewpoint). In this case, one or more virtual agents 70 are arranged (drawn) at positions farther from the viewpoint than the positional relationship between the robot 12 and the user 60 in the real space.

As shown in FIG. 13, in step S67, all arms 122, 124 and 126 of robot 12 are returned to their initial positions. Subsequently, in step S69, the system waits for a certain period of time. In the next step S71, all the virtual agents 70 are made to approach the user 60. FIG. For example, the CPU 16 a generates image data when all the virtual agents 70 are moved toward the viewpoint by a predetermined distance (for example, 70 cm) in the virtual space, and transmits the image data to the HMD 20 .

Then, in step S73, it is determined whether or not all the virtual agents 70 are close enough to shake hands with the user 60 or not. Here, in the virtual space, it is determined whether or not the distances between all the virtual agents 70 and the viewpoint are equal to or less than a predetermined distance df.

If "NO" in step S73, that is, if none of the virtual agents 70 are close enough to shake hands with the user 60, the process returns to step S69. On the other hand, if "YES" in step S73, that is, if all the virtual agents 70 are close enough to shake hands, then in step S75 the arms of all the virtual agents 70 are brought close to the user 60 to shake hands. Here, the CPU 16a reproduces an animation in which the arms of all the virtual agents 70 are brought closer to the direction of the viewpoint in the virtual space and handshake. However, when there is only one virtual agent 70, the animation of the virtual agent 70 is reproduced. When there are two virtual agents 70, the animation of each of the two virtual agents 70 is reproduced. When there are three virtual agents 70, each animation of the three virtual agents 70 is reproduced.

In the subsequent step S77, the arms of the robots 12 corresponding to all the virtual agents 70 are brought close to the user 60 to shake hands. In step S77, the CPU 16a raises the arms of the robots 12 corresponding to all the virtual agents 70 slightly in the direction in which the user 60 exists (or moves them forward) while setting the angle of the forearm to the upper arm at 70°. ). However, when there is one virtual agent 70, the movement of the arm 122 is controlled. When there are two virtual agents 70, the movements of arms 122 and 124 are controlled. When there are three virtual agents 70, the movements of arms 122, 124 and 126 are controlled.

Subsequently, in step S79, it waits for a certain period of time (for example, 2 to 3 seconds), and in step S81, it is determined whether or not the user 60 wishes to end the handshake. If "NO" in step S81, that is, if the user 60 does not wish to end the handshake, the process returns to step S79. On the other hand, if "YES" in step S81, that is, if the user 60 wishes to end the handshake, the second pseudo-contact process ends.

According to this embodiment, an HMD is used to present arbitrary visual stimuli using a plurality of virtual agents, while contact stimuli are presented from a plurality of arms of a robot to which a plurality of virtual agents are assigned. You can get the feeling that you are in contact with the agent. That is, a new pseudo-touch presentation system can be provided.

Further, according to this embodiment, the visual stimulus presentation device, that is, the HMD, displays the motions of a plurality of virtual agents, and by moving the movable parts of the contact stimulus presentation device, that is, the plurality of arms of the robot, in relation to the motions. Since the contact stimulus is presented to the user, the sense of unity between the visual stimulus and the contact stimulus can be enhanced.

In the second pseudo-contact process of this embodiment, when three virtual agents are assigned to the robot, the three virtual agents are brought closer to the user (viewpoint) to move. Two of the three virtual agents may be brought closer to the user to perform the action. In this case, two of the three virtual agents are selected according to a predetermined rule such as random or by the user.

Also, in this embodiment, the arm of the robot is configured to have joints corresponding to elbow joints and shoulder joints, but may be configured to have only elbow joints or shoulder joints. If only the elbow joint is provided, the handshake can be performed by moving the upper arm, and if only the shoulder joint is provided, the handshake can be performed by moving the entire arm.

Furthermore, in this embodiment, three arms of the robot are provided, but four or more arms can be provided as long as the number of arms is two or more. If four or more arms are provided, more virtual agents can be displayed on the HMD and interacted with each virtual agent individually or simultaneously.

Furthermore, in this embodiment, the case where the user shakes hands with the virtual agent has been described, but the predetermined action need not be limited to handshakes, and may be touches.

In the above-described embodiment, one server controls each element of the system, but the robot, server, and HMD may be controlled by separate computers, or any number of computers may be used. can manage and control the system.

As the virtual agent displayed on the HMD shown in the above embodiment, any character may be used as long as it is a 3D computer graphics agent. Therefore, appearance, gender, clothing, race (person, animal, anthropomorphic character, etc.), and realism (animation style, live-action style, etc.) may be changed according to the user.

It should be noted that the specific numerical values such as the angles given in the above examples are merely examples, and can be appropriately changed as necessary.

DESCRIPTION OF SYMBOLS 10... Pseudo contact presentation system 12... Communication robot 16... Server 20... HMD
60 ... user 70 ... virtual agent 122, 124, 126 ... robot arm

Claims (4)

A contact stimulus presentation device comprising a plurality of stimulus presentation units that act on a user and provide contact stimuli to the user;
assigning means for assigning a virtual agent to each of the two or more stimulus presentation units among the plurality of stimulus presentation units; and mounted on the user's head to display the plurality of virtual agents assigned by the assignment means. and a head-mounted display device that operates some or all of the plurality of virtual agents to present visual stimuli to the user;
The contact stimulus presentation device operates each of two or more of the plurality of stimulus presentation units in relation to the motion of some or all of the plurality of virtual agents. contact presentation system. The visual stimulus presentation device displays an animation showing actions for the user for some or all of the plurality of virtual agents,
The stimulus presenting unit includes a movable unit, and the contact stimulus presenting unit is movable so as to change in association with animation of actions of some or all of the plurality of virtual agents by movement of the movable unit. 2. The pseudo contact presentation system according to claim 1, wherein a part is operated to present the contact stimulus to the user. A pseudo contact presentation system comprising: a contact stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide contact stimuli to the user; and a head-mounted display device that is worn on the head of the user. A control program executed by a computer, comprising:
a movable unit control unit that controls the plurality of stimulus presentation units of the contact stimulus presentation device to present contact stimuli to the user;
allocation means for allocating a virtual agent to each of two or more of the plurality of stimulus presentation units; and a display for displaying part or all of the plurality of virtual agents allocated by the allocation means. function as a department,
Visual stimuli are presented to the user by displaying the plurality of virtual agents on the head-mounted display device, and some or all of the plurality of virtual agents are displayed by the movable part control unit. A sensory stimulus presentation program that causes each of the two or more stimulus presentation sections among the plurality of stimulus presentation sections to operate in relation to motion. A pseudo contact presentation system comprising: a contact stimulus presentation device having a plurality of stimulus presentation units that act on a user to provide contact stimuli to the user; and a head-mounted display device that is worn on the head of the user. A control method comprising:
controlling the plurality of stimulus presentation units of the contact stimulus presentation device to present contact stimuli to the user;
a step of assigning a virtual agent to each of two or more of the plurality of stimulus presentation units; and a step of displaying some or all of the plurality of virtual agents assigned by the assignment means. including
presenting a visual stimulus to the user by displaying the plurality of virtual agents on the head-mounted display device, and displaying some or all of the plurality of virtual agents on the contact stimulus presentation device; A control method for operating each of the two or more stimulus presentation units among the plurality of stimulus presentation units in relation to motion.

Images