JP6615716B2 - Robot and enclosure - Google Patents

Description

  The present invention relates to a robot and a robot casing.

  Head mounted displays (HMD) are used in various fields. By providing the HMD with a head tracking function and updating the display screen in conjunction with the posture of the user's head, a sense of immersion in the video world can be enhanced.

Japanese Patent Laying-Open No. 2015-95045

  In recent years, a technology called tele-existence has appeared that uses a robot placed in a remote location as its alternation. A remote robot sends surrounding image data and audio data to the user and reproduces it on the user side, so that the user can communicate with the surrounding people with a sense of presence like being in the robot's location. It becomes possible.

  The present inventor has paid attention to the possibility of the cooperation between the telexistence and the HMD, and has developed a technique for enhancing the convenience and usefulness of the telejeg resistance system.

  The present invention has been made in view of these problems, and its purpose is to make effective use of the structure of a remotely operated robot, technology for processing viewing data acquired by the robot, and viewing data acquired by the robot. Is to provide the technology.

  In order to solve the above problems, a robot according to an aspect of the present invention includes an actuator device and a housing whose posture can be changed by the actuator device. The robot of this aspect includes a movable member that is operatively supported by the housing, a motor provided in the housing, a transmission member that transmits the rotation of the motor to the movable member, and a drive control unit that controls the rotation of the motor. A state information acquisition unit that acquires facial expression information and / or emotion information of the user wearing the head-mounted display device. The drive control unit controls the rotation of the motor based on facial expression information and / or emotion information.

  Another aspect of the present invention is a housing that includes a movable member that is operatively supported by the housing, a motor, and a transmission member that transmits rotation of the motor to the movable member. The transmission member is a member that undergoes torsional elastic deformation.

  Note that any combination of the above components, the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium on which the computer program is recorded so as to be readable, a data structure, etc. are also included in the present invention It is effective as an embodiment of

  ADVANTAGE OF THE INVENTION According to this invention, the structure of the robot operated remotely can be provided.

It is a figure which shows the structural example of the information processing system in an Example. It is a figure which shows the example of the usage scene of a robot. It is a figure which shows the example of the external appearance shape of HMD. It is a figure which shows the functional block of HMD. It is a figure which shows the external appearance structure of a robot. It is a figure which shows the structure of an insertion member. It is a figure which shows the cross section of a robot. It is a figure which shows the example of the attitude | position of the housing | casing in a robot. It is a figure which shows the example of the attitude | position of the housing | casing in a robot. It is a figure which shows the functional block of a robot. It is a figure which shows the circuit structure of the phase difference amplifier with which an audio | voice processing part is provided. It is a figure for demonstrating the phase difference of a signal waveform. It is a figure for demonstrating the principle which amplifies the phase difference of an input signal waveform. It is a figure which shows the functional block of the robot for implement | achieving applied technology. It is a figure which shows the functional block of a processing apparatus. It is a figure for demonstrating an omnidirectional panoramic image. It is a figure for demonstrating the picked-up image data currently recorded on the image recording part. It is a figure which shows the relationship between the frame image which an image generation part produces | generates, and image data. It is a figure which shows schematic structure inside a housing | casing. It is a figure which shows the functional block of an input system. It is a figure which shows the position of a protective cover. It is a figure which shows the example of the holding | maintenance content of an operation table.

  FIG. 1 shows a configuration example of an information processing system 1 in the embodiment. The information processing system 1 includes a robot 10 and a head mounted display device (HMD) 100 that a user A wears on the head. The HMD 100 includes a display panel 102 for both eyes, an earphone 104 for both ears, and a microphone 106. Here, the earphone 104 is used as the sound output means, but a headphone shaped to touch the ear may be used. The HMD 100 is connected to the network 4 via an access point (AP) 2. The AP 2 has functions of a wireless access point and a router, and the HMD 100 is connected to the AP 2 by a known wireless communication protocol, but may be connected by a cable.

  The robot 10 includes an actuator device 12 and a housing 20 that is driven by the actuator device 12 so that the posture can be changed. The housing 20 is equipped with a right camera 14a, a left camera 14b, a right microphone 16a, a left microphone 16b, and a speaker 18. Hereinafter, the right camera 14a and the left camera 14b will be referred to as “camera 14” unless otherwise distinguished, and the right microphone 16a and the left microphone 16b will be referred to as “mic 16” unless otherwise distinguished. In the embodiment, the camera 14 and the microphone 16 are provided in the housing 20 driven by the actuator device 12, but the speaker 18 may be provided in the hemispherical housing 36 of the actuator device 12, for example. The robot 10 is connected to the network 4 via an access point (AP) 3. The robot 10 is connected to the AP 3 by a known wireless communication protocol, but may be connected by a cable.

  In the information processing system 1, the HMD 100 and the robot 10 are communicably connected via the network 4. When the HMD 100 and the robot 10 exist in the vicinity, they may be connected to each other so as to be able to communicate directly or wirelessly without using an AP. In the information processing system 1, the robot 10 operates as a so-called alternate user A. The movement of the HMD 100 worn by the user A is transmitted to the robot 10, and the actuator device 12 moves the housing 20 in conjunction with the movement of the HMD 100. For example, when the user A swings his / her neck back and forth, the actuator device 12 moves the casing 20 to swing back and forth, and when the user A swings his / her neck left and right, the actuator device 12 moves the casing 20 to swing left and right. . Thus, a person around the robot 10 can communicate with the user A with a sense that the user A is on the spot.

  The right camera 14a and the left camera 14b are arranged on the front surface of the housing 20 with a predetermined interval in the lateral direction. The right camera 14a and the left camera 14b constitute a stereo camera. The right camera 14a captures a right-eye image at a predetermined cycle, and the left camera 14b captures a left-eye image at a predetermined cycle. The captured right-eye image and left-eye image are transmitted to the user A's HMD 100 in real time. The HMD 100 displays the received right-eye image on the right-eye display panel, and displays the received left-eye image on the left-eye display panel. As a result, the user A can view the video in the direction in which the housing 20 of the robot 10 is facing in real time.

  The right microphone 16a and the left microphone 16b are arranged in the housing 20 with a predetermined interval in the lateral direction. The right microphone 16a and the left microphone 16b constitute a stereo microphone, and are arranged at a predetermined interval in the horizontal direction, so that the time for the sound to reach varies depending on the position of the sound source. The difference in the arrival time of sound appears as a phase difference between sound signals generated by the right microphone 16a and the left microphone 16b. In order to increase the phase difference between the audio signals of the right microphone 16a and the left microphone 16b, the right microphone 16a and the left microphone 16b may be arranged as far apart as possible, specifically on both sides of the housing 20. preferable.

  The audio signals generated by the right microphone 16a and the left microphone 16b are processed as described later, and are transmitted to the user A's HMD 100 in real time as audio data for the right ear and audio data for the left ear. The HMD 100 outputs the received right ear audio data from the right ear earphone 104, and outputs the received left ear audio data from the left ear earphone 104. Thereby, the user A can hear the sound around the robot 10 in real time.

  It is known that humans perceive the position of a sound source in the left-right direction based on the difference in arrival time of sound waves to both ears, but in reality, not only the difference in arrival time but also the shape of the pinna that collects sound waves, The position of the sound source is perceived depending on the shape of the ear canal that conveys the sound to the middle ear. Also, when the sound source is on the right or left side of the human front, the face is located in the path for the sound wave to reach the pinna on the far side compared to the pinna on the near side. The difference in arrival time of sound waves becomes larger than the difference in distance from the sound source.

  On the other hand, the front surface of the housing 20 has a flat shape, and the microphone 16 does not have a shape corresponding to an auricle or an external auditory canal, so the difference in sound arrival time is substantially the difference between the sound source and both microphones. It will correspond to. In the embodiment, the right microphone 16a and the left microphone 16b are arranged on both side surfaces of the housing 20 and are provided at positions farthest from each other. However, the sound signal generated by the right microphone 16a and the left microphone 16b are generated. The inventor's experiment has revealed that even if the audio signal is amplified and output from the right earphone and the left earphone, the position of the sound source in the left-right direction cannot be perceived well.

  In other words, it has been experimentally found that the phase difference between the audio signals generated by the right microphone 16a and the left microphone 16b is small to perceive the left-right direction as compared with the sound that humans are usually accustomed to listening to. Therefore, the robot 10 has a mechanism for amplifying the phase difference between the audio signals of the right microphone 16a and the left microphone 16b and providing the HMD 100 with audio data that is closer to the sound that can be heard by both human ears. This mechanism will be described later.

  In the HMD 100, the microphone 106 generates an audio signal emitted by the user A. The voice data by the user A is transmitted to the robot 10 in real time, and the robot 10 outputs the received voice data from the speaker 18. Thereby, the person around the robot 10 can hear the voice uttered by the user A in real time.

  As described above, in the information processing system 1, the robot 10 is remotely operated by the user A to reproduce the movement and sound of the face of the user A, and the user A can view images and sounds around the robot through the HMD 100. And people around the robot 10 can communicate in real time. Such an information processing system 1 is usefully used in various environments.

  FIG. 2 shows an example of a usage scene of the robot 10. In this example, a meeting is held in a room, and a robot 10 that is a user A's alternate is placed on a table. In this example, the robot 10 faces the front four people, and the camera 14 photographs the front four people within the angle of view. The robot 10 transmits the captured image of the camera 14 to the user A's HMD 100 in real time. The user A participates in the conference while viewing the room status through the display panel 102 of the HMD 100 and speaks to transmit the voice of the user A to the robot 10 in real time, and the robot 10 outputs the voice of the user A from the speaker 18.

  As described above, the robot 10 transmits the audio data obtained by amplifying the phase difference between the audio signals generated by the left and right microphones 16 to the HMD 100 in real time. Thereby, the user A can perceive whether the person who has spoken in the room is located on the right side, the left side, or the front side with respect to the direction in which the housing 20 is facing. When the user A feels that the person on the right side speaks, the user A turns the neck to the right and turns to the right side. At this time, since the housing 20 of the robot 10 also faces the right side in conjunction with the movement of the neck of the user A, the camera 14 photographs the participant sitting on the right side.

  In this way, the user A can join the conference as if he was in a room while being at a remote place by having the alternate robot 10 interlocked with the movement of the user A. In addition, participants who are actually in the room can communicate with the user A from the voice of the user A and the movement of the housing 20 without feeling uncomfortable. Note that the usage scene shown in FIG. 2 is an example, and the user A can obtain viewing data from the robot 10 while in a remote location even in other usage scenes.

  FIG. 3 shows an example of the external shape of the HMD 100. In this example, the HMD 100 includes an output mechanism unit 110 and a mounting mechanism unit 112. The wearing mechanism unit 112 includes a wearing band 108 that goes around the head when worn by the user and fixes the HMD 100 to the head. The wearing band 108 is made of a material or a structure whose length can be adjusted according to the user's head circumference.

  The output mechanism unit 110 includes a housing 114 shaped to cover the left and right eyes when the user wears the HMD 100, and includes a display panel 102 in a position facing the eyes. The display panel 102 may be a liquid crystal panel or an organic EL panel. Inside the housing 114, a pair of left and right optical lenses that are positioned between the display panel 102 and the user's eyes and expand the viewing angle of the user are provided.

  The HMD 100 further includes an earphone 104 that is inserted into the user's ear when worn. The earphone 104 is an example of an audio output unit, and the HMD 100 may include a headphone. At this time, the HMD 100 and the headphones may be configured integrally, but may be separate.

  The HMD 100 transmits the sensor information detected by the posture sensor and the voice data obtained by encoding the voice signal from the microphone 106 to the robot 10 and receives the image data and voice data generated by the robot 10 to display the display panel. 102 and the earphone 104.

  3 shows an immersive (non-transmissive) display device that completely covers both eyes, but may be a transmissive display device. The shape may be a hat shape as shown in the figure, but may also be a glasses shape. Note that the HMD 100 may include not only a dedicated head-mounted display device but also a terminal device having a display panel, a microphone, and a speaker, and a housing that fixes the display panel of the terminal device to a position in front of the user's eyes. Good. The terminal device may have a relatively small display panel such as a smartphone or a portable game machine.

  FIG. 4 shows functional blocks of the HMD 100. The control unit 120 is a main processor that processes and outputs various signals and data such as image signals, audio signals, sensor information, and commands. The storage unit 122 temporarily stores data, commands, and the like that are processed by the control unit 120. The attitude sensor 124 detects attitude information such as the rotation angle and inclination of the HMD 100 at a predetermined cycle. The posture sensor 124 includes at least a triaxial acceleration sensor and a triaxial gyro sensor. The microphone 106 converts a user's voice into an electrical signal and generates an audio signal.

  The communication control unit 126 transmits and receives signals and data to and from the robot 10 by wired or wireless communication via a network adapter or an antenna. The communication control unit 126 receives, from the control unit 120, the posture information detected by the posture sensor 124 and the voice data obtained by encoding the voice signal from the microphone 106, and transmits them to the robot 10. In addition, the communication control unit 126 receives image data and audio data from the robot 10 and supplies them to the control unit 120. Upon receiving image data and audio data from the robot 10, the control unit 120 supplies the image data to the display panel 102 for display, and supplies the audio data to the earphone 104 for audio output.

  FIG. 5 shows an external configuration of the robot 10. The housing 20 accommodates the camera 14, the microphone 16, and the speaker 18. The camera 14 and the speaker 18 are provided on the front surface of the housing, and the microphone 16 is provided on the side surface of the housing. The camera 14, the microphone 16, and the speaker 18 operate by being supplied with electric power from a power supply device housed in the housing 36 via a power line (not shown).

  The housing 20 has a protective cover 19. When the robot 10 is not used, that is, when the power of the robot 10 is turned off, the protective cover 19 is disposed at a closed position that covers the front of the housing, and the camera 14 and The speaker 18 is protected. The protective cover 19 has a rotation shaft projecting inward at both ends in the longitudinal direction, and is attached by inserting each rotation shaft into a pair of shaft holes on the side surface of the housing. Thus, the protective cover 19 is attached to the housing 20 so as to be rotatable about the rotation axis. In the state shown in FIG. 5, the protective cover 19 is disposed at the open position rotated approximately 180 degrees from the closed position, and the camera 14 is exposed so that the surroundings can be photographed. The protective cover 19 preferably has a stopper mechanism that is fixed in the open position.

  In a state where the robot 10 is used, the protective cover 19 may be driven and controlled according to the emotion of the user wearing the HMD 100 and the tilt of the neck. In this case, a motor as a drive unit is provided in the housing 20, and the operation of the protective cover 19 can be controlled by connecting the motor shaft to the rotating shaft of the protective cover 19. In this case, the stopper mechanism is not provided, and the protective cover 19 may be rotatable within a range of approximately 270 degrees from the closed position.

  The housing 20 is supported by the actuator device 12 so that the posture can be changed. The actuator device 12 includes a leg portion 40, a hemispherical housing 36 supported on the upper portion of the leg portion 40, and a drive mechanism 50 for driving the housing 20. The drive mechanism 50 includes a first arcuate arm 32 having a first through hole 32a formed in the longitudinal direction, a second arcuate arm 34 having a second through hole 34a formed in the longitudinal direction, A pedestal 30 that rotatably supports the first arc-shaped arm 32 and the second arc-shaped arm 34 in a state where the first arc-shaped arm 32 and the second arc-shaped arm 34 intersect each other is provided. The upper side of the pedestal 30 is covered with a cover 38, and motors for rotating the first arcuate arm 32 and the second arcuate arm 34 are arranged in the space covered with the cover 38, respectively. The pedestal 30 is rotatably supported with respect to the housing 36, and a motor for rotating the pedestal 30 is disposed in the housing 36.

  The first arc-shaped arm 32 and the second arc-shaped arm 34 are formed in a semicircular shape, and both ends are supported by the pedestal 30 so as to have the same center of rotation. The diameter of the semicircular first arc-shaped arm 32 is slightly larger than the diameter of the semicircular second arc-shaped arm 34, and the first arc-shaped arm 32 is located on the outer peripheral side of the second arc-shaped arm 34. Be placed. The first arc-shaped arm 32 and the second arc-shaped arm 34 may be arranged so as to be orthogonal to each other on the pedestal 30. In the embodiment, a line connecting both ends of the first arcuate arm 32 supported by the pedestal 30 and a line connecting both ends of the second arcuate arm 34 supported by the pedestal 30 are orthogonal to each other. The insertion member 42 is inserted into the first through long hole 32a and the second through long hole 34a, and is disposed at the intersection of the first through long hole 32a and the second through long hole 34a. The insertion member 42 slides in the first through long hole 32a and the second through long hole 34a by the rotation of the first arc-shaped arm 32 and the second arc-shaped arm 34.

  FIG. 6 shows the configuration of the insertion member 42. The insertion member 42 has a first restriction portion 42a wider than the first through long hole 32a and the second through long hole 34a so as to maintain the insertion state of the first through long hole 32a and the second through long hole 34a. And a wider second restricting portion 42b. The first restricting portion 42a is disposed above the first through long hole 32a, the second restricting portion 42b is disposed below the second through long hole 34a, and the insertion member 42 is the first through long hole 32a. Further, it is prevented from dropping off from the second through long hole 34a. When the insertion member 42 is attached to the first through long hole 32a and the second through long hole 34a, either the first restricting portion 42a or the second restricting portion 42b is formed separately from the shaft portion 42c, and the shaft A structure may be adopted in which the portion 42c is fixed to the end portion of the shaft portion 42c in a state where the portion 42c is inserted into the first through long hole 32a and the second through long hole 34a.

  The shaft portion 42c is a portion that is inserted into the first through long hole 32a and the second through long hole 34a, and is always located at the intersection of the first through long hole 32a and the second through long hole 34a. The rotation of the shaft portion 42c is restricted in the first through long hole 32a and the second through long hole 34a. In the embodiment, the shaft portion 42c has a rectangular cross section having a width slightly narrower than the widths of the first through long hole 32a and the second through long hole 34a, and the inside of the first through long hole 32a and the second through long hole Although the rotation is restricted within 34a, the rotation of the shaft portion 42c may be restricted by other means. For example, a rail is provided on the inner peripheral surface of the second arc-shaped arm 34, a rail groove is provided in the second restricting portion 42b, and the rotation of the shaft portion 42c is restricted by fitting the rail and the rail groove. Also good. The housing 20 is attached to the first restricting portion 42a, and the rotation of the shaft portion 42c is restricted, so that the housing 20 can be maintained in a desired posture.

  The shaft portion 42c has a narrower width than the first through long hole 32a and the second through long hole 34a, so that it can slide in the first through long hole 32a and the second through long hole 34a. To do. Accordingly, the insertion member 42 can move along the first through long hole 32a and can move along the second through long hole 34a by the rotation of the first circular arc arm 32 and the second circular arc arm 34.

  FIG. 7 shows a cross section of the robot 10. FIG. 7 mainly shows a drive system of the robot 10, and illustration of a control circuit board, a memory, wiring, and the like is omitted. FIG. 7A shows a cross section cut along the second arcuate arm 34 in a state where the first arcuate arm 32 and the second arcuate arm 34 stand 90 degrees with respect to the pedestal 30. (B) shows a cross-section cut along the first arc-shaped arm 32 in a state where the first arc-shaped arm 32 and the second arc-shaped arm 34 stand 90 degrees with respect to the pedestal 30.

  The first motor 52 is provided for rotating the first arcuate arm 32, and the second motor 54 is provided for rotating the second arcuate arm 34. The first motor 52 and the second motor 54 are arranged on the pedestal 30, and when the pedestal 30 rotates, the first motor 52 and the second motor 54 also rotate together with the pedestal 30. The third motor 56 is provided to rotate the pedestal 30 and is disposed in the housing 36. The first motor 52, the second motor 54, and the third motor 56 are rotated by power supplied from a power supply device (not shown).

  The first motor 52 rotates the first arc-shaped arm 32, the second motor 54 rotates the second arc-shaped arm 34, and the third motor 56 rotates the pedestal 30. The orientation and posture of the housing 20 attached to 42 can be changed.

8 and 9 are diagrams illustrating examples of the posture of the housing 20 in the robot 10.
8A and 8B show an example in which the housing 20 is tilted in the left-right direction. FIGS. 9A and 9B show an example in which the housing 20 is tilted in the front-rear direction. Thus, the drive mechanism 50 of the robot 10 can cause the housing 20 to take an arbitrary posture. The attitude of the housing 20 is controlled by adjusting the driving amounts of the first motor 52 and the second motor 54, and the orientation of the housing 20 is controlled by adjusting the driving amount of the third motor 56. .

  FIG. 10 shows functional blocks of the robot 10. The robot 10 includes an input system 22 that receives and processes input from the outside, and an output system 24 that processes output to the outside. The input system 22 includes a reception unit 60, a sensor information acquisition unit 62, a motion detection unit 64, a line-of-sight direction determination unit 66, an actuator control unit 68, an audio data acquisition unit 70, and an audio processing unit 72. The output system 24 includes an image processing unit 80, an audio processing unit 82, and a transmission unit 90.

  In FIG. 10, each element described as a functional block for performing various processes can be configured by a circuit block, a memory, and other LSIs in terms of hardware, and loaded in the memory in terms of software. Realized by programs. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  As described above, the HMD 100 transmits to the robot 10 the audio data obtained by encoding the sensor information detected by the posture sensor 124 and the audio signal generated by the microphone 106, and the receiving unit 60 receives the sensor information and the audio data. The audio data acquisition unit 70 acquires the received audio data, the audio processing unit 72 performs audio processing, and outputs it from the speaker 18. Thereby, the robot 10 reproduces the voice of the user A in real time, and the people around the robot 10 can hear the voice of the user A.

  The sensor information acquisition unit 62 acquires posture information detected by the posture sensor 124 of the HMD 100. The motion detection unit 64 detects the posture of the HMD 100 attached to the user A's head. The line-of-sight direction determination unit 66 determines the line-of-sight direction of the camera 14 of the housing 20 according to the attitude of the HMD 100 detected by the motion detection unit 64.

  The motion detection unit 64 performs head tracking processing that detects the posture of the head of the user wearing the HMD 100. The head tracking process is performed in order to link the visual field displayed on the display panel 102 of the HMD 100 to the posture of the user's head. In the head tracking process of the embodiment, the rotation angle with respect to the horizontal reference direction of the HMD 100 and the horizontal plane The tilt angle is detected. The horizontal reference direction may be set, for example, as a direction facing when the power of the HMD 100 is turned on.

  The line-of-sight direction determination unit 66 determines the line-of-sight direction according to the posture of the HMD 100 detected by the motion detection unit 64. This line-of-sight direction is the line-of-sight direction of the user A, and by extension, the line-of-sight direction (optical axis direction) of the camera 14 of the robot 10 that is a substitute.

  In order to link the viewing direction (optical axis direction) of the camera 14 with the viewing direction of the user A, the reference posture of the robot 10 needs to be set in advance. FIG. 5 shows a state in which the first arc-shaped arm 32 and the second arc-shaped arm 34 stand up by 90 degrees with respect to the pedestal 30. This state is set as the horizontal direction, and the power supply of the robot 10 The direction in which the front surface of the housing 20 faces when is turned on may be set as the horizontal reference direction. Note that the robot 10 may have a posture sensor as in the HMD 100 so that the horizontal direction can be set autonomously.

  With the reference postures of the HMD 100 and the robot 10 set, the line-of-sight direction determination unit 66 determines the rotation angle and tilt angle detected by the motion detection unit 64 as the line-of-sight direction (optical axis direction) of the camera 14 as it is. Good. When the motion detection unit 64 detects the rotation angle and the inclination angle of the HMD 100, the line-of-sight direction determination unit 66 determines the line-of-sight direction of the HMD 100 as a vector (x, y, z) of three-dimensional coordinates. The line-of-sight direction of the camera 14 may be determined as the same (x, y, z), or may be determined as (x ′, y ′, z ′) with some correction.

  The actuator control unit 68 controls the orientation of the camera 14 so that the line-of-sight direction determined by the line-of-sight direction determination unit 66 is obtained. Specifically, the actuator control unit 68 adjusts the power supplied to the first motor 52, the second motor 54, and the third motor 56 so that the movement of the housing 20 follows the movement of the HMD 100. The motor drive control by the actuator control unit 68 is performed in real time, and therefore the direction of the housing 20 is moved in the same way as the direction of the line of sight of the user A.

  According to the actuator device 12 of the embodiment, the housing 20 is driven with reference to the rotation centers of the first arc-shaped arm 32 and the second arc-shaped arm 34, and this movement shows the same movement as the human neck. . The actuator device 12 reproduces the movement of the neck of the user A with a simple structure in which two semicircular arms are crossed.

  A person communicates his intention by moving his neck. For example, in Japan, if the head is swung vertically, positive is expressed, and if it is swung horizontally, negative is expressed. However, since the actuator device 12 moves the housing 20 in the same manner as the movement of the neck of the user A, people around the robot 10 Can sense the intention of the user A also by the movement of the housing 20. Therefore, the ability to reproduce the movement of the neck of the user A with a simple structure is very useful in the teleexistence technology.

Next, the output system 24 will be described.
In the output system 24, the right camera 14a and the left camera 14b are directed in the directions controlled by the actuator device 12 and shoot the respective angles of view. The right camera 14a and the left camera 14b may be arranged apart from each other, for example, so as to be an average distance between eyes of an adult. The right-eye image data captured by the right camera 14a and the left-eye image data captured by the left camera 14b are transmitted from the transmission unit 90 to the HMD 100 and displayed on the right half and the left half of the display panel 102, respectively. These images form parallax images viewed from the right eye and the left eye, and can be displayed stereoscopically by displaying the display panel 102 in an area divided into two. In order for the user A to view the display panel 102 through the optical lens, the image processing unit 80 may generate image data in which optical distortion caused by the lens is corrected in advance and supply the image data to the HMD 100.

  The right camera 14a and the left camera 14b capture images at a predetermined cycle (for example, 1/60 seconds), and the transmission unit 90 transmits image data to the HMD 100 without delay. As a result, the user A can see the situation around the robot 10 in real time, and can also see the desired direction by changing the direction of the face.

  The right microphone 16a and the left microphone 16b convert sound around the robot 10 into an electric signal to generate a sound signal. Hereinafter, the audio signal generated by the right microphone 16a is referred to as a “first audio signal”, and the audio signal generated by the left microphone 16b is referred to as a “second audio signal”. As described above, since the right microphone 16a and the left microphone 16b are disposed laterally apart from each other in the housing 20, the first audio signal generated by the right microphone 16a and the second audio signal generated by the left microphone 16b. Causes a phase difference.

  When the present inventor encodes the first audio signal and the second audio signal with the same phase difference and provides them to the HMD 100, the user cannot recognize the direction of the sound source, that is, whether the audio can be heard from the right side. Or, it was found through experiments that it was difficult to determine whether the sound was heard from the left side. In the experiment, the lateral width of the housing 20 is set to about the adult human face width (16 cm). However, since the sound transmission structure in the human ear cannot be reproduced by the microphone 16, It was concluded that only the phase difference between the two audio signals is insufficient for humans to perceive the direction of the sound source.

  As a means for solving this, it is conceivable to increase the lateral width of the casing 20 to increase the phase difference between the first audio signal and the second audio signal. It becomes heavier and it becomes necessary to increase the output of the motor used in the actuator device 12. Further, when the lateral width of the housing 20 is increased, the interval between the right microphone 16a and the left microphone 16b becomes wider than the interval between both ears of a human, so that an audio signal that is different from the sense that a person actually hears sound is generated. Will be acquired.

  Therefore, the present inventor has devised to solve this problem by amplifying the phase difference between the first audio signal and the second audio signal. As will be described below, the audio processing unit 82 has a function of amplifying the phase difference between the first audio signal generated by the right microphone 16a and the second audio signal generated by the left microphone 16b. Since the robot 10 needs to transmit the microphone sound to the HMD 100 in real time, the sound processing unit 82 realizes the phase difference amplification function by a hardware circuit.

FIG. 11 shows a circuit configuration of the phase difference amplifying device 82a included in the audio processing unit 82. Phase difference amplifier 82a is an analog circuit device for amplifying and outputting the phase difference between the second audio signal v _L of the first audio signal v _R and the left microphone 16b to the right microphone 16a was formed was produced.

The first amplifier 84a, when the right microphone 16a is inputted to the first audio signal _{v R,} the first and positive-phase signal _V ^{R +} obtained by amplifying the first voice signal _{v R,} inverting amplifying the first voice signal _{v R} The first negative phase signal V _R ⁻ is output. The first amplifier 84a may be composed of an operational amplifier that amplifies and outputs the positive phase component of the input signal and an operational amplifier that amplifies and outputs the negative phase component of the input signal. May be composed of an operational amplifier having two output terminals for outputting.

The second amplifier 84b is inverted and left microphone 16b is input to the second audio signal _{v L,} the second positive phase signal _V ^{L +} obtained by amplifying the second audio signal _{v L,} the second audio signal _{v L} The amplified second negative phase signal V _L ⁻ is output. Similarly to the first amplifier 84a, the second amplifier 84b may be composed of two operational amplifiers that output the positive phase component and the negative phase component, respectively, and one output that outputs both the positive phase component and the negative phase component. You may comprise from an operational amplifier.

The first adder 86a adds a signal obtained by multiplying the first positive phase signal V _R ⁺ by a first coefficient (α times) and a signal obtained by multiplying the second negative phase signal V _L ⁻ by a second coefficient (β times). The output signal V _rOUT is output. Here, α and β are values greater than 0 and 1 or less. Note that α and β are set to be different, and α> β in this example. The output signal V _rOUT is expressed by the following equation.
V _rOUT = α × V _R ⁺ + β × V _L ⁻

The first adder 86a adds the output of the voltage dividing circuit that divides the first positive phase signal V _R ⁺ by α times and the output of the voltage dividing circuit that divides the second negative phase signal V _L ⁻ by β times. However, it may be an operational amplifier that adds a voltage signal obtained by multiplying the first positive phase signal V _R ⁺ by α and a voltage signal obtained by multiplying the second negative phase signal V _L ⁻ by β.

The second adder 86b adds the signal obtained by multiplying the second positive phase signal V _L ⁺ by the first coefficient (α times) and the signal obtained by multiplying the first negative phase signal V _R ⁻ by the second coefficient (β times). The output signal _VlOUT is output. The output signal _VlOUT is expressed by the following equation.
V _lOUT = α × V _L ⁺ + β × V _R ⁻

The second adder 86b adds the output of the voltage dividing circuit that divides the second positive phase signal V _L ⁺ by α times and the output of the voltage dividing circuit that divides the first negative phase signal V _R ⁻ by β times. However, it may be an operational amplifier that adds a voltage signal obtained by multiplying the second positive phase signal V _L ⁺ by α and a voltage signal obtained by multiplying the first negative phase signal V _R ⁻ by β.

The third amplifier 88a is an output signal _{V ROUT} of the first adder 86a third factor multiplication (gamma times) and outputs a _{V ROUT,} fourth amplifier 88b is an output signal _{V LOUT} of the second adder 86b V _LOUT is output by multiplying by the third coefficient (γ times). In the audio processing unit 82, the output signals V _ROUT and V _LOUT from the phase difference amplifying device 82a are respectively audio encoded and transmitted from the transmission unit 90 to the HMD 100 as audio data for right ear and audio data for left ear. .

FIG. 12 is a diagram for explaining a phase difference between signal waveforms. 12 (a) shows a relationship between the second audio signal _{v L} waveform first audio signal _{v R} and the left microphone 16b to the right microphone 16a is generated is produced. Here, for convenience of explanation, the relationship between the first positive phase signal V _R ⁺ and the second positive phase signal V _L ^{+ obtained} by amplifying the first audio signal v _R and the second audio signal v _L to the same magnification is shown. Yes. In this input waveform, the sound source is arranged on the right side when viewed from the housing 20 of the robot 10, and the phase of the first positive phase signal V _R ⁺ slightly advances from the second positive phase signal V _L ^+. Also, the amplitude of the first positive phase signal V _R ⁺ is higher.

FIG. _12B shows the relationship between the waveforms of the output signal V _rOUT of the first adder 86a and the output signal V _lOUT of the second adder 86b. Compared with the phase difference of the input waveform shown in FIG. 12A, it can be seen that the phase difference of the output waveform of the adder shown in FIG. 12B is widened (amplified).

FIG. 13 is a diagram for explaining the principle of amplifying the phase difference of the input signal waveform. FIG. 13A shows the first positive phase signal V _R ⁺ and the first negative phase signal V _R ⁻ , the second positive phase signal V _L ⁺ and the second negative phase signal V _L ⁻ in a two-dimensional coordinate system. doing. The phase difference between the first positive phase signal V _R ⁺ and the second positive phase signal V _L ⁺ is θ.

Figure 13 (b) shows the output signal _{V ROUT} of the first adder 86a and the output signal _{V LOUT} of the second adder 86b. As described above, V _rOUT and V _lOUT are
V _rOUT = α × V _R ⁺ + β × V _L ⁻
V _lOUT = α × V _L ⁺ + β × V _R ⁻
It is expressed as In FIG. 13B, α = 1.0 and β = 0.6 are set.

As shown in FIG. _13B , the phase difference between V _rOUT and V _1OUT is θ ′, which is larger than the phase difference θ shown in FIG. Thus, the phase difference amplifying device 82a amplifies the phase difference between the two input audio signals.

  As a result of simulation by the present inventor, when the phase difference of the input signal is 15 degrees, the phase difference of the output signal is 4 times 60 degrees, and when the phase difference of the input signal is 30 degrees, the phase difference of the output signal is 3 It was found that when the input signal phase difference was 45 degrees, the output signal phase difference was approximately 2.7 times 120 degrees when the input signal was 90 degrees.

According to this simulation result, the smaller the phase difference, the greater the amplification factor. In the actual case 20, the phase difference of the input signal is about 5 to 20 degrees, and the phase difference amplifying device 82a can increase the amplification factor in this range, so that the phase difference of the output signal can be determined by the user. It is widened to the extent that you can hear the direction. The output signals V _ROUT and V _LOUT from the phase difference amplifying device 82a are each audio-encoded and transmitted from the transmission unit 90 to the HMD 100 as audio data for the right ear and audio data for the left ear.

  In the HMD 100, right ear audio data is output as audio from the right ear earphone 104, and left ear audio data is output as audio from the left ear earphone 104. User A recognizes the direction of the sound source by listening to the sound with the amplified phase difference from both ears. If the user A feels that a voice has been heard from the right side, the user A turns his face to the right side. At this time, since the housing 20 of the robot 10 faces the right side in conjunction with the movement of the face of the user A (see FIG. 2), the camera 14 of the robot 10 captures the environment on the right side and captures the captured image data in real time. Send to HMD100. As a result, the user A can speak while looking at the face of the person who speaks, and can realize an unprecedented excellent user interface.

  In the above example, α = 1.0 and β = 0.6 are set. However, it is preferable that the values of α and β are appropriately set by experiments. As shown in FIG. 5, the right microphone 16 a and the left microphone 16 b are provided at positions where the side surface of the housing 20 is recessed and at the back side when viewed from the front. Since the sound wave transmission structure in the microphone 16 depends on the shape of the side surface of the housing, the ratio of α and β is preferably determined optimally through experiments.

  In FIG. 5, the microphone 16 is disposed inside the rear plate 17 in the lateral direction. This is because the rear plate 17 has a function of making the frequency characteristics of the sound wave from the front and the sound wave from the rear different from each other and reducing the high frequency component from the rear. In other words, the rear plate 17 has a function like a human auricle with respect to the microphone 16, and a sound wave from the rear wraps around the rear plate 17 and reaches the microphone 16. In order to make the frequency characteristics of the sound wave from the front and the sound wave from the rear different, the rear plate 17 may be formed to be further expanded in the vertical direction and the horizontal direction. By forming a sound wave shield such as the rear plate 17 behind the microphone 16, the user A can also recognize the position of the sound source in the front-rear direction.

  As described above, in the information processing system 1, the user A can freely communicate with the people around the robot 10 in real time using the robot 10 that is his or her own character. Below, the technique which further raises the usability of the information processing system 1 is proposed.

  2. Description of the Related Art Conventionally, a technique for generating an omnidirectional panoramic image by stitching (sewing) images taken while changing the tilt of a camera is known. Recently, a dedicated pan / tilt camera has also been sold, and individuals can shoot spherical panoramic images.

  In the information processing system 1, the robot 10 photographs the surroundings by directing the camera 14 in the line-of-sight direction according to the movement of the user A's head. As the user A faces various directions, the camera 14 captures various directions. It is possible to generate a virtual omnidirectional panoramic image by adding a three-dimensional vector representing the line-of-sight direction to this captured image and recording it.

  FIG. 14 shows a modification of the functional block of the robot 10. This functional block is based on the functional block shown in FIG. 10, and shows that the determined visual line direction is supplied from the visual line direction determining unit 66 to the image processing unit 80.

  During the use of the robot 10 by the user A, the transmission unit 90 transmits the image data for both eyes and the sound data for both ears (hereinafter sometimes collectively referred to as “viewing data”) via the network 4 to the user A. To the HMD100. At this time, the transmission unit 90 transmits the same viewing data to the processing device 200 via the network 4 via the router 5, and the processing device 200 records the viewing data of the user A.

  The processing device 200 generates the omnidirectional panoramic image in real time based on the image data of the user A while recording the viewing data of the user A, and displays an image corresponding to the viewing direction of the user B different from the user A. B has a function provided to the HMD 100a. The HMD 100a has the same configuration as the HMD 100 described so far. The processing device 200 may be configured by a single server, for example, but may be configured by a server group that provides a cloud service.

  In order to enable the processing device 200 to generate an omnidirectional panoramic image, the image processing unit 80 includes vector information indicating the line-of-sight direction supplied from the line-of-sight direction determination unit 66 and the start of shooting for each frame image data. Shooting time information indicating the elapsed time from the point is added. The vector information indicates the line-of-sight direction of the camera 14 of the robot 10. The shooting time information only needs to express the time from the shooting start point, and may be a frame number indicating the order of shooting, for example.

  In this technique, while the robot 10 is being used by the user A, the user B wears the HMD 100a and provides the HMD 100a with image data and audio data generated based on the viewing data of the user A supplied from the robot 10. To do. If the viewing data of the user A is simply reproduced as it is, the processing device 200 may simply stream the received viewing data to the HMD 100a of the user B as it is. An image based on the line-of-sight direction of the user B is reconstructed from the panoramic image formed based on the image data, and can be provided to the user B's HMD 100a. The audio data is streamed to user B's HMD 100a.

  FIG. 15 shows functional blocks of the processing apparatus 200. The processing device 200 includes a reception unit 202, a sensor information acquisition unit 204, a motion detection unit 206, a line-of-sight direction determination unit 208, an image determination unit 210, an audio determination unit 212, a viewing data provision unit 214, a transmission unit 216, and a recording unit 218. Prepare. The recording unit 218 includes an image recording unit 220 and an audio recording unit 222. When the receiving unit 202 receives the viewing data transmitted from the robot 10, the image recording unit 220 sequentially records the received image data, and the audio recording unit 222 sequentially records the received audio data. The image data is added with vector information and shooting time information at the time of shooting for each frame image.

  User B transmits an instruction to reproduce viewing data of user A to processing apparatus 200 through HMD 100a. When receiving the playback instruction, the processing device 200 starts playback processing of viewing data. The audio determination unit 212 determines audio data to be provided to the user B, and immediately reads out the audio data recorded in the audio recording unit 222 from the audio recording unit 222 and provides it to the viewing data providing unit 214. That is, the sound determination unit 212 distributes the sound data provided from the robot 10 to the HMD 100a in a streaming manner. Therefore, the user B can hear the same sound as the sound that the user A is listening from the earphone 104 of the HMD 100a.

  During the reproduction process by the processing device 200, the reception unit 202 receives sensor information transmitted from the HMD 100a worn by the user B, and the sensor information acquisition unit 204 acquires the received sensor information. This sensor information is posture information obtained by the posture sensor 124 detecting the posture of the HMD 100a. The motion detection unit 206 detects the posture of the HMD 100a mounted on the user B's head. The line-of-sight direction determination unit 208 determines the line-of-sight direction of the virtual camera in the omnidirectional panoramic image according to the attitude of the HMD 100 a detected by the motion detection unit 206. The image determination unit 210 determines image data to be provided to the user B, and uses the plurality of image data recorded in the image recording unit 220 to synthesize an image photographed by the virtual camera toward the determined line-of-sight direction. To generate image data.

  The viewing data providing unit 214 provides viewing data that combines the image data determined by the image determining unit 210 and the sound data determined by the sound determining unit 212 from the transmitting unit 216 to the HMD 100a of the user B.

  In FIG. 15, each element described as a functional block for performing various processes can be configured by a circuit block, a memory, and other LSIs in terms of hardware, and loaded in the memory in terms of software. Realized by programs. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The processing device 200 generates an omnidirectional panoramic image. Therefore, when the user B turns his / her neck left or right and rotates the horizontal line of sight to the left or right, the panorama image in the left or right direction is displayed on the display panel 102 of the HMD 100a, and the user B By tilting up or down and tilting the line of sight in the vertical direction, a panoramic image in the upward or downward direction is displayed on the display panel 102 of the HMD 100a.

  FIG. 16 is a diagram for explaining an omnidirectional panoramic image generated by the processing device 200. In this technology, a virtual environment in which a visible image is changed is realized by the user B being positioned at the center of the sphere and changing the direction of the line of sight. The image determining unit 210 stitches the image data recorded in the image recording unit 220 to generate an omnidirectional panoramic image.

  In the embodiment, for simplicity of explanation, the robot 10 does not zoom the camera 14 and acquires image data at a constant enlargement ratio. Therefore, the image determination unit 210 forms an omnidirectional panoramic image by pasting the image data onto the inner peripheral surface of the omnidirectional sphere based on the vector information added to the image data. It should be noted that a portion where a plurality of image data overlap is overwritten with the latest image data, so that an omnidirectional panoramic image close to a real-time situation can be constructed.

  Note that the actual image generation processing of the image determination unit 210 does not always reconstruct the omnidirectional panoramic image in order to reduce the processing load, but the frame image 7 taken from the center point 9 where the user B is located. Is a process of dynamically generating. At this time, the image determination unit 210 preferably sets the shooting range (view angle) of the virtual camera 8 so as to correspond to the actual shooting range (view angle) of the camera 14 of the robot 10. As a result, at the timing when the line-of-sight direction of user A and the line-of-sight direction of user B match, user B can see the same image as user A.

  In this manner, the image determination unit 210 performs image stitching processing using the vector information set as metadata in the image data, and generates the frame image 7 in the shooting range determined from the line-of-sight direction of the user B. The motion detection unit 206 detects the rotation angle and inclination of the head of the user B (actually, the HMD 100a) by performing the head tracking process of the user B. Here, the rotation angle of the HMD 100a is a rotation angle with respect to the reference direction of the horizontal plane, and the reference direction may be set as a direction facing when the power of the HMD 100a is turned on, for example. The inclination of the HMD 100a is an inclination angle with respect to the horizontal plane. A known technique may be used as the head tracking process, and the motion detection unit 206 detects the rotation angle and inclination of the HMD 100a from the sensor information detected by the attitude sensor of the HMD 100a.

  The line-of-sight direction determination unit 208 determines the posture of the virtual camera 8 in the virtual sphere according to the detected rotation angle and inclination of the HMD 100a. The virtual camera 8 is arranged so as to photograph the inner peripheral surface of the virtual sphere from the center point 9 of the virtual sphere, and the line-of-sight direction determination unit 208 changes the direction of the optical axis of the virtual camera 8 to the camera 14 of the robot 10. It may be determined so as to coincide with the optical axis direction.

  In the robot 10, when the line-of-sight direction determination unit 66 determines the line-of-sight direction of the HMD 100 of the user A as a three-dimensional coordinate vector (x, y, z), the line-of-sight direction of the camera 14 of the robot 10 is the same (x, y, It was explained that z) may be determined. Also in the processing apparatus 200, when the line-of-sight direction determination unit 208 determines the line-of-sight direction of the HMD 100a of the user B as a vector (x, y, z) of three-dimensional coordinates, the line-of-sight direction of the virtual camera 8 is the same (x, y, z) may be determined. When the gaze direction determining unit 66 determines the gaze direction of the camera 14 by correcting the gaze direction of the HMD 100 with a predetermined conversion formula, the gaze direction determination unit 208 also corrects the gaze direction of the HMD 100a with the same conversion formula. Then, the line-of-sight direction of the virtual camera 8 may be obtained. In this way, by handling each three-dimensional coordinate system, the user B can see the same image as the user A at the timing when the line-of-sight direction of the user A matches the line-of-sight direction of the user B. .

  When the image determination unit 210 generates the frame image 7 of the virtual camera 8, the image determination unit 210 performs optical distortion correction for the optical lens, and supplies image data to the viewing data providing unit 214. In FIG. 16, one virtual camera 8 is shown. Actually, however, two virtual cameras 8 for the left eye and the right eye are arranged, and each image data is provided for the left eye provided from the robot 10. It is generated based on the image data and the right-eye image data.

  FIG. 17 is a diagram for describing captured image data recorded in the image recording unit 220. Here, for convenience of explanation, a plurality of image data for one eye is shown, and image data in a state in which an appropriate affine transformation is applied to the user B's line-of-sight direction is arranged on a two-dimensional plane. Note that the line-of-sight direction of the user B will be described later.

  The image determination unit 210 has a function of generating an omnidirectional panoramic image by connecting overlapping portions of the captured images. As a technique for joining photographed images, for example, a known technique may be used as described in Japanese Patent No. 5865388 by the same applicant. Hereinafter, a method for selecting which captured image data to use from among a plurality of captured image data recorded in the image recording unit 220 will be described.

  FIG. 17 shows five image data I1 to I5. (X, y, z) included in each image data represents the viewing direction (vector information) of the camera 14 at the time of photographing, and “t” represents photographing time information. Here, the image data I1 has vector information (x1, y1, z1) and shooting time information t1 as additional information. Similarly, the image data I2 has vector information (x2, y2, z2) and shooting time information t2 as additional information, and the image data I3 has vector information (x3, y3, z3) and shooting time information t3 as additional information. The image data I4 includes vector information (x4, y4, z4) and shooting time information t4 as additional information, and the image data I5 includes vector information (x5, y5, z5) and shooting time information t5. It has as additional information.

  Note that the shooting time information t1 to t5 as additional information expresses the elapsed time from the shooting start point (time 0) and has a relationship of t1 <t2 <t3 <t4 <t5. Therefore, among the image data I1 to I5, the image data I1 is photographed first and the image data I5 is photographed last. The image determination unit 210 selects image data for generating a composite image based on the shooting time information and the visual line direction of the virtual camera 8 determined by the visual line direction determination unit 208.

  Specifically, the image determination unit 210 captures a shooting range (virtual camera) cut from the panoramic image from the gaze direction of the virtual camera 8 determined by the gaze direction determination unit 208, that is, the direction in which the user B wearing the HMD 100a faces. 8) and image data including an image included in the shooting range is extracted based on vector information added to the image data.

  FIG. 18 is a diagram illustrating the relationship between the frame image 7 to be generated by the image determination unit 210 and the image data. 17 and 18, each image data I1 to I5 is mapped on a two-dimensional plane orthogonal to the viewing direction (X, Y, Z) of the virtual camera 8 based on each vector information. The positions of the data I1 to I5 are defined by four vertex coordinates on the two-dimensional plane. The image determination unit 210 determines the position (shooting range) of the field angle of the virtual camera 8 in the panoramic image based on the line-of-sight direction (X, Y, Z) of the virtual camera 8, and is a two-dimensional plane orthogonal to the line-of-sight direction. The four vertex coordinates of the frame image 7 are determined. The image determination unit 210 extracts image data included in the frame image 7 from the image data recorded in the image recording unit 220. As illustrated, the image data I1 to I5 include images included in the frame image 7, and thus are extracted as image data included in the shooting range of the virtual camera 8.

  For an area where a plurality of image data overlaps, the image determination unit 210 generates a composite image by preferentially using image data having late shooting time information. In the example shown in FIG. 18, the frame image 7 is combined with the frame image 7 so that the image data is written in order from the image data with the earliest shooting time, that is, in order from the image data I1.

  As described above, the image determination unit 210 generates a composite image using image data having shooting time information closer to the current time for overlapping regions of the plurality of image data. For example, in the image included in the shooting range, when there is an overlapping portion between the image data I4 and the image data I5, the image data I5 having a later shooting time is embedded in the overlapping portion. As a result, a composite image can be generated using image data close to the current time, and a composite image close to the current time can be provided to the user B.

  In this image reproduction application, depending on the direction in which the user B is facing, image data may be insufficient and it may be difficult to generate the frame image 7. In particular, immediately after the robot 10 starts shooting, the number of image data is small in the first place, so that the image determination unit 210 may not be able to generate the frame image 7 according to the user B's line-of-sight direction. Although this cannot actually occur, if the user A does not move the HMD 100 during the shooting period, the vector information of the image data recorded in the image recording unit 220 is all the same. When facing the direction opposite to the user A, there is no image data included in the shooting range of the virtual camera 8 in the line-of-sight direction.

  In such a case, the image determination unit 210 generates image data in which a message indicating that an image in the line-of-sight direction of the user B cannot be generated is superimposed on the received image data of the user A, and the viewing data providing unit 214 sends the image data to the HMD 100a. May be provided. For example, when the image of the user B's line-of-sight direction cannot be synthesized by a predetermined ratio (for example, 30%) or more, the image determination unit 210 does not generate a synthesized image and provides viewing data for the image data viewed by the user A together with the message It may be supplied to the unit 214.

  Further, since the image determination unit 210 synthesizes the frame image 7 from a plurality of image data, the generated frame image 7 becomes a stitched image, and visibility may be deteriorated. Therefore, for example, when an image of a predetermined ratio (for example, 50%) within the shooting range cannot be formed from one image data, as described above, the image determination unit 210 cannot generate an image in the line-of-sight direction of the user B. The image data may be generated by superimposing the above message on the image data of the user A.

  In the example described above, it has been described that the image determination unit 210 generates a composite image by using image data having late shooting time information preferentially. However, a frame can be generated by using image data having earlier shooting time information. When a predetermined ratio or more of the image 7 can be configured, image data having earlier shooting time information may be used.

  Further, as time elapses, the environment in which the robot 10 is shooting changes, so it may be undesirable to provide the user B with a composite image using past image data. Therefore, the image determination unit 210 may perform image extraction processing so that image data older than a predetermined time is not included in the composite image.

  The above shows an example in which the user B uses the viewing data of the user A in real time. The applied technology will be described below. In the applied technology, the processing device 200 records the viewing data for the secondary use, not for the purpose of real-time reproduction of the viewing data of the user A.

  For secondary use of viewing data, in the robot 10, the image processing unit 80 adds shooting time information and vector information to each of the frame image data, and the audio processing unit 82 adds the recording start point to the audio data. Recording time information indicating the elapsed time from is added. Note that since shooting (recording) by the camera 14 and recording by the microphone 16 are started at the same timing, the shooting start point and the recording start point indicate the same timing. The shooting time information and the recording time information may be time information generated by a clock generation unit in the robot 10. Any format may be used for adding the additional information to the image data and the audio data as long as the processing device 200 can refer to the viewing data for reproduction.

  In this applied technology, after the user A finishes using the robot 10, another user B (which may be the user A) wears the HMD 100 a and the viewing data of the user A recorded in the processing device 200 is stored. Originally generated image data and audio data are provided to the HMD 100a. At this time, as described in the embodiment, the processing device 200 constructs an omnidirectional panoramic image based on the viewing data of the user A, and regenerates an image based on the viewing direction of the user B from the omnidirectional panoramic image. It is configured so that it can be provided to the HMD 100a of the user B. In this usage environment, the robot 10 is not used.

  Referring to FIG. 15, the image recording unit 220 records image data transmitted from the robot 10, and the audio recording unit 222 records audio data transmitted from the robot 10. In this applied technology, the image recording unit 220 and the audio recording unit 222 are in a state where all the viewing data transmitted from the robot 10 to the user A has been recorded. The image data is added with shooting time information and vector information at the time of shooting, and the audio data is added with recording time information.

  User B transmits an instruction to reproduce viewing data of user A to processing apparatus 200 through HMD 100a. When receiving the playback instruction, the processing device 200 starts playback processing of viewing data. Note that when the recording unit 218 records viewing data for one hour, the user B may be allowed to start playback from any time within the range of one hour. In this case, the receiving unit 202 receives time designation from the user B and supplies it to the image determining unit 210 and the sound determining unit 212.

  The audio determination unit 212 reads audio data having recording time information corresponding to the reproduction time information indicating the elapsed time from the reproduction start point from the audio recording unit 222 and provides the audio data to the viewing data providing unit 214. The playback start point means a playback start point of viewing data, and therefore indicates the same timing as the shooting start point and the recording start point. The audio determination unit 212 reads out audio data whose recording time information matches the reproduction time information from the audio recording unit 222 and provides it to the viewing data providing unit 214.

  During the reproduction process by the processing device 200, the reception unit 202 receives sensor information transmitted from the HMD 100a worn by the user B, and the sensor information acquisition unit 204 acquires the received sensor information. This sensor information is posture information obtained by the posture sensor 124 detecting the posture of the HMD 100a. The motion detection unit 206 detects the posture of the HMD 100a mounted on the user B's head. The line-of-sight direction determination unit 208 determines the line-of-sight direction of the virtual camera according to the posture of the HMD 100a detected by the motion detection unit 206. The image determination unit 210 uses the plurality of image data recorded in the image recording unit 220 to synthesize an image photographed by the virtual camera oriented in the determined line-of-sight direction. The viewing data providing unit 214 provides viewing data, which is a combination of the image data combined by the image determining unit 210 and the audio data read by the audio determining unit 212, from the transmitting unit 216 to the HMD 100a.

  The image determination unit 210 stitches the images viewed by the user A before the reproduction time of the viewing data by the user B, and dynamically generates a frame image 7 photographed from the center point 9 where the user B is located. To do.

  An image viewed by the user A before the viewing data playback time by the user B will be described. When image data for one hour from the shooting start point is recorded in the image recording unit 220, the playback time from the playback start point by the user B is specified at some timing within one hour. For example, if the playback time is 15 minutes from the start of playback, an image to which shooting time information within 15 minutes has been added, that is, an image shot until 15 minutes have passed since the start of shooting is displayed before the playback time. The image viewed by the user A is displayed. That is, if the playback is performed at the time point of 15 minutes from the start of playback, the image determination unit 210 generates the frame image 7 using the image data to which shooting time information within 15 minutes from the start of shooting is added, and plays back. If the time point of 45 minutes from the start is being played back, the image determination unit 210 generates the frame image 7 using image data to which shooting time information within 45 minutes from the start of shooting is added.

  Referring to FIG. 18, the image determination unit 210 extracts image data to which shooting time information before the playback time information is added, and extracts image data to which shooting time information after the playback time information is added. Do not extract. For example, if the time information for reproducing the frame image 7 is after the time t3 and before the time t4, the image determination unit 210 extracts the image data I1 to I3 and does not extract the image data I4 and I5. . As described above, by generating a composite image using image data to which shooting time information before playback time information is added, the image determination unit 210 does not show the user B an image shot after the playback time. Like that.

  Since the viewing data providing unit 214 transmits audio data having recording time information corresponding to the reproduction time to the HMD 100a, the user B is listening to audio synchronized with the reproduction time. Therefore, the situation before the reproduction time is generally known, and if the provided image data is synthesized from the image data before the reproduction time, it is possible to grasp what kind of situation is displayed. However, if the provided image data is synthesized from image data after the reproduction time, the user B will be shown an image that he / she does not know, and it is expected that the user B will be confused. Therefore, the image determination unit 210 prevents the user B from seeing an image taken after the reproduction time.

  Note that the image determination unit 210 generates a composite image by using image data having shooting time information close to reproduction time information for overlapping portions of the plurality of image data. For example, in the case where an image included in the imaging range includes an overlapping portion between the image data I1 and the image data I2, the image data I2 captured later is embedded in the overlapping portion. As a result, a composite image can be generated using image data close to the reproduction time information, and an image synthesized from image data with the latest reproduction time can be provided to the user B.

  In the above, this invention was demonstrated based on the Example. It is to be understood by those skilled in the art that the embodiments are exemplifications, and that various modifications can be made to combinations of the respective components and processing processes, and such modifications are within the scope of the present invention.

  In the embodiment, it has been described that the image determination unit 210 performs the image stitching process and generates the frame image 7 in the shooting range determined from the user B's line-of-sight direction. In the modification, the image determination unit 210 performs the user based on the line-of-sight direction of the virtual camera 8 and the vector information added to the image data recorded in the image recording unit 220 without performing the image stitching process. The image data to be provided to B is determined.

  In this modification, the image determination unit 210 determines the image data to which the vector information corresponding to the viewing direction of the virtual camera 8 is added as the image data to be provided to the user B. The vector information corresponding to the line-of-sight direction of the virtual camera 8 includes vector information that matches the line-of-sight direction of the virtual camera 8 and vector information that can be considered to substantially match the line-of-sight direction of the virtual camera 8. Specifically, when the visual line direction of the virtual camera 8 and the vector information are within a predetermined angle (for example, 10 degrees), the image determination unit 210 substantially determines that the visual line direction of the virtual camera 8 and the vector information are substantially the same. It may be determined that they match.

  When synchronously reproducing the viewing data of the user A, the image determination unit 210 converts the image data having the latest shooting time information from among the image data to which the vector information corresponding to the line-of-sight direction of the virtual camera 8 is added. Determined as image data to be provided. As a result, an image close to the current time can be provided to the user B.

  When the image data to which the vector information corresponding to the line-of-sight direction of the virtual camera 8 is not recorded in the image recording unit 220, the image determination unit 210 removes the component in the height direction (z-axis direction) (x , Y) Image data to which vector information that can be regarded as substantially matching components may be determined as image data to be provided to the user B. The vector information that can be regarded as matching is vector information in which the (x, y) component is within a predetermined angle (for example, 7 degrees). By looking at the approximation of only the (x, y) component, the image determination unit 210 makes it easy to find the image data to which the vector information corresponding to the line-of-sight direction of the virtual camera 8 is added. It will be possible to avoid situations that cannot be provided.

  When the user A's viewing data is secondarily used, the image determination unit 210 determines image data to be provided to the user B from images viewed by the user A before the playback time of the viewing data by the user B. That is, the image determination unit 210 determines image data to which vector information corresponding to the line-of-sight direction of the virtual camera 8 is added from among image data to which shooting time information before reproduction time information is added. At this time, when there are a plurality of corresponding image data, the image determination unit 210 preferably selects image data having shooting time information close to reproduction time information.

  A case where image data when the user A turns his / her neck in the horizontal direction is recorded in the image recording unit 220 will be considered. When the user B is slightly delayed from the user A and turns his head in the opposite direction to the user A, an image viewed by the user A may be reversely reproduced on the HMD 100a. In this case, the time series of the image data is reversed, which may cause the user B to feel uncomfortable. Therefore, when the user B continuously changes the line-of-sight direction, when the image data of the user A is reversely reproduced, the image determination unit 210 fixes the image data and changes the provided image data. It may not be allowed to.

  In order to enhance the usefulness of the information processing system 1, the robot 10 may further include an input sensor that accepts an external input, such as a tactile sensor or a vibration sensor. In the functional block shown in FIG. 10, the input sensor is provided in the output system 24, and sensor information of the input sensor is transmitted from the transmission unit 90 to the HMD 100. The HMD 100 may include an output unit that outputs sensor information, and may convert the sensor information into vibration or the like and transmit it to the user A.

  In the information processing system 1, it has been described that the robot 10 interlocks the housing 20 with the movement of the neck of the user A. However, the robot 10 may further include means for transmitting a state such as the facial expression of the user A. For example, the HMD 100 includes a sensor that detects the movement of the user A's eyes and eyebrows, a means for analyzing the tone of the voice, and the like. The movements of the eyes and eyebrows express the user's facial expressions and emotions, and the tone of the voice expresses the user's psychological state and emotions. The state information regarding the movement of the eyes and eyebrows and / or the tone of the voice is transmitted from the HMD 100 to the robot 10, and the robot 10 drives the facial expression unit provided in the housing 20 to reproduce the facial expression and emotion of the user A. May be. The facial expression unit may be a movable member (for example, imitating the shape of an eyebrow) formed on the upper surface of the camera 14 on the front surface of the housing 20, and the movable member is driven based on information transmitted from the HMD 100. The The protective cover 19 that is rotatably attached to the upper part of the camera 14 may be used as an expression unit imitating a human eyebrow. A motor that moves the protective cover 19 is provided in the housing 20, and the protective cover 19 is provided. You may move. The facial expression unit may be a display that expresses the facial expression and psychological state of the user A with colors, and may express the facial expression and emotion of the user A by changing the display color.

  Hereinafter, the movable member provided in the robot 10 will be described. As described above, the movable member is provided to create the facial expression of the robot 10. The movement of the movable member may be controlled so as to express the facial expression and emotion of the user A wearing the HMD 100, but may be controlled based on the attitude of the housing 20. The robot 10 not only moves the housing 20 but also moves the movable member attached to the housing 20 to provide a person around the robot 10 as if the robot 10 is the user A. .

  As described so far, the posture of the housing 20 can be changed by the actuator device 12, and the actuator device 12 interlocks the posture of the housing 20 with the movement of the HMD 100 worn by the user A. The housing 20 and the actuator device 12 constitute the robot 10, and the housing 20 constitutes the face of the robot 10. Here, “to configure the face of the robot” is to form the casing 20 so that when a person around the robot 10 looks at the robot 10, the casing 20 can be recognized as a face. . The point to be recognized as a face is that the housing 20 is located at the top of the robot 10 and includes the camera 14 corresponding to the human eye, and the posture of the housing 20 is linked to the movement of the user's neck. By holding these points, the housing 20 is recognized as the face of the robot 10.

  FIG. 19A shows a schematic structure of the front surface of the housing 20, and FIG. 19B shows a schematic structure of the top surface of the housing 20. The housing 20 includes a front plate that exposes the lens of the camera 14, a bottom plate that is connected to the front plate and to which the insertion member 42 is attached, a rear plate 17 that is connected to the bottom plate and faces the front plate, and the rear plate 17 and the front plate. And an upper plate that is connected to face the bottom plate, and further includes a front plate, a bottom plate, a rear plate 17 and a pair of side plates that close both ends of the upper plate. 9 and the like show a state in which the rear plate 17 protrudes rearward. However, since the shape of the rear plate 17 is not particularly related to the following description, the housing 20 is shown in FIGS. 19 (a) and 19 (b). Is shown as having a rectangular parallelepiped that is long in the lateral direction. 19A shows a state in which the front plate is removed, and a schematic top view in FIG. 19B shows a state in which the upper plate has been removed. 14, the wiring including the microphone 16 and the power line and the control signal line of the drive motor 142 are not shown.

  A structure for fixing various components is provided in the housing 20. The microphone storage portions 11a and 11b are provided on the inner walls of the side plates and form spaces for storing the right microphone 16a and the left microphone 16b, respectively. The right microphone 16a and the left microphone 16b are inserted and fixed to the microphone storage portions 11a and 11b from the front. The right camera 14a and the left camera 14b are fixed with screws 15 to a camera support portion 13 that protrudes forward from the inner wall of the rear plate 17. A motor fixing unit 140 is formed between the right camera 14 a and the left camera 14 b, and the drive motor 142 is fixed to the motor fixing unit 140.

  The protective cover 19, which is a movable member, is a rectangular plate member and is operably supported by the housing 20. The protective cover 19 has rotating shafts 19a and 19b projecting inward at both ends in the longitudinal direction. Shaft holes are formed in the vicinity of the front edges of the upper portions of the pair of side plates of the housing 20, the rotation shaft 19a is inserted into the shaft hole of the right side plate, and the rotation shaft 19b is inserted into the shaft hole of the left side plate. The protective cover 19 is rotatably supported by the housing 20.

  At least one of the rotation shaft 19a and the rotation shaft 19b is inserted into the housing beyond the inner wall of the side plate, and is connected to the motor shaft 142a of the drive motor 142 inside the housing. In FIGS. 19A and 19B, both of the rotation shafts 19a and 19b have entered the inner side of the inner wall of the housing side plate, and the rotation shaft 19b is fixed to one end of the transmission member 144.

  The transmission member 144 is a member that transmits the rotation of the drive motor 142 to the protective cover 19 that is a movable member. One end of the transmission member 144 is fixed to the rotary shaft 19 b, and the other end is fixed to the motor shaft 142 a of the drive motor 142. The other end of the transmission member 144 may be directly connected to the motor shaft 142a, but may be fixed to an output shaft of a speed reduction mechanism that decelerates motor rotation. The protective cover 19 is connected to the drive motor 142 by the transmission member 144 and is moved between a closed position for protecting the camera 14 and other positions. The protective cover 19 protects the camera 14 while the power of the robot 10 is turned off. On the other hand, the protective cover 19 operates as an expression unit while the power is turned on, so that the protective cover 19 can be effectively used.

  Here, the transmission member 144 is formed as a member that undergoes torsional elastic deformation. In this example, the transmission member 144 is a coil spring, and the rotation input of the motor shaft 142a is delayed in time by the coil spring being twisted and rotated and output to the rotation shaft 19b. The torsion generated in the coil spring becomes a restoring force, and is delayed from the rotation input of the motor shaft 142a to rotate the coil spring in a direction to cancel the torsion.

  Moreover, it is preferable that the transmission member 144 has bending elasticity with respect to the axial direction. As shown in the drawing, since various components are accommodated in the housing 20, the space for arranging the drive motor 142 is limited. Since the transmission member 144 has bending elasticity with respect to the axial direction, even if the motor shaft 142a and the rotation shaft 19b are not on the same axis, the transmission member 144 can be coupled in a state where the transmission member 144 is bent in the axial direction. FIGS. 19A and 19B show a state in which the transmission member 144 connects the motor shaft 142a and the rotating shaft 19b in a bent state so as to absorb the misalignment between the motor shaft 142a and the rotating shaft 19b.

  The reason for adopting the transmission member 144 having torsional elasticity is to delay the rotation of the rotating shaft 19b with respect to the rotation of the motor shaft 142a. As will be described later, the drive motor 142 drives the protective cover 19 according to the posture of the housing 20 and the emotion information of the user, etc., but the protective cover 19 is on the eyes of the robot (camera 14). It becomes like an eyebrow. When the protective cover 19 reciprocates at high speed, the movement of the facial expression unit is not ideal because it feels unsatisfactory as seen by the surrounding people and is different from the movement of the human eyebrows.

  Although it is possible to perform filter processing for gently limiting the rotation acceleration by calculating and controlling the rotation of the drive motor 142 with software, it is inevitable that the load on the calculator increases. Therefore, by transmitting the rotation of the drive motor 142 to the rotating shaft 19b of the protective cover 19 using the transmission member 144 that is torsionally elastically deformed, the transmission response is naturally delayed without performing a soft calculation, and the high speed of the protective cover 19 is achieved. Can be suppressed.

  If the torsional rigidity of the transmission member 144 is high, it is difficult to cause a delay in transmission response to the rotating shaft 19b. Therefore, it is preferable to set the torsional rigidity low. Also depends on weight. Therefore, in order to increase the response delay, the weight of the protective cover 19 may be increased. Currently, coil springs having various characteristics are on the market, and it is advantageous from the viewpoint of cost to use the transmission member 144 as a coil spring. For example, the transmission member 144 is a solid rod-shaped member made of a rubber material. It is good also as a member. By forming the transmission member 144 from a rubber material, it is possible to provide torsional elasticity and bending elasticity.

  FIG. 20 shows functional blocks of the input system 22 in the robot 10. The functional blocks shown in FIG. 20 are added to the input system 22 shown in FIG. 10, and FIG. 20 shows only the configuration for driving and controlling the protective cover 19. The input system 22 includes a receiving unit 60, an actuator control unit 68, a state information acquisition unit 74, a drive control unit 76, and an operation table 78.

  In FIG. 20, each element described as a functional block for performing various processes can be configured by a circuit block, a memory, and other LSIs in terms of hardware, and loaded into the memory in terms of software. Realized by programs. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  Hereinafter, an example of a control method of the protective cover 19 will be described. First, as a premise, when the power supply of the robot 10 is turned on, the actuator control unit 68 stands up 90 degrees with respect to the base 30 with respect to the first arcuate arm 32 and the second arcuate arm 34. The first motor 52 and the second motor 54 are driven. The rotation angle φx of the first motor 52, the rotation angle φy of the second motor 54, and the rotation angle φz of the third motor 56 are set to reference values (for example, 0), with the posture of the housing 20 at this time as a reference posture. That is, (φx, φy, φz) is set to (0, 0, 0) in the reference posture.

21A to 21C show the positional relationship between the housing side plate and the protective cover 19.
While the power of the robot 10 is off, the protective cover 19 is disposed at a closed position that protects the camera 14. FIG. 21A shows a state in which the protective cover 19 covers the front surface of the housing 20 and protects the camera 14.

  When the power supply of the robot 10 is turned on, the drive control unit 76 controls the rotation of the drive motor 142 to move the protective cover 19 to the initial operation position. The initial operation position is a position where the protective cover 19 is rotated 90 degrees upward from the closed position. FIG. 21B shows a state where the protective cover 19 has been moved to the initial operation position. When the protective cover 19 moves to the initial operation position, the camera 14 can capture the front direction. Note that the initial operation position may be set to a position that does not fall within the viewing angle of the camera 14.

After the power of the robot 10 is turned on, the protective cover 19 is operable between the initial operation position and the maximum operation position, but may be operable between the closed position and the maximum operation position.
FIG. 21C shows a state where the protective cover 19 has been moved to the maximum operating position. The maximum operation position may be set at a rotation limit position of the protective cover 19 where the protective cover 19 and the upper plate of the casing come into contact, or may be set before that.

Hereinafter, a method for controlling the rotation of the drive motor 142 based on the attitude of the housing 20 determined by the actuator device 12 will be described.
The actuator control unit 68 controls the rotation angles of the first motor 52, the second motor 54, and the third motor 56 according to the sensor information from the HMD 100. The actuator control unit 68 provides rotation angle information (φx, φy, φz) indicating the attitude of the housing 20 to the drive control unit 76, and the drive control unit 76 uses the rotation angle information (φx, φy, φz). Thus, the rotation of the drive motor 142 is controlled.

Specifically, the drive control unit 76 obtains the rotation angle Φ of the drive motor 142 according to the following calculation formula. The rotation angle Φ at the initial operation position is set to zero.
Φ = a × | φx | + b × | φy | + c × | φz |
Here, | φx |, | φy |, and | φz | express the absolute value of the rotation angle. “a”, “b”, and “c” are weighting coefficients for the respective rotation angles, and the operating characteristics of the protective cover 19 are determined according to preference by appropriately setting the weighting coefficients.

  The drive control unit 76 calculates the rotation angle Φ based on the rotation angle information (φx, φy, φz), and supplies power to the drive motor 142 so that the rotation angle of the drive motor 142 becomes Φ. The behavior of the protective cover 19 linked to the posture of the body 20 can be realized.

Another example of the rotation control method based on the attitude of the housing 20 will be shown.
In the above example, the behavior of the protective cover 19 is linked to the posture of the housing 20. However, for example, when the posture of the housing 20 does not fluctuate for a predetermined time, control may be performed such that the protective cover 19 is raised. When the posture of the housing 20 does not change, it is assumed that the user A is concentrating on some object and staring. At this time, the drive control unit 76 controls the drive motor 142 so that the protective cover 19 is raised. In addition, it is preferable that the person around the robot 10 knows in advance that the standing of the protective cover 19 expresses that the user A is concentrated. Thus, by estimating the situation where the user A is concentrated from the attitude of the housing 20, the situation of the user A can be expressed by the operation of the protective cover 19.

Next, a method for controlling the rotation of the drive motor 142 based on the facial expression information and / or emotion information of the user detected by the HMD 100 will be described.
In this example, the HMD 100 includes a sensor that detects the movement of a specific part of the face of the user A. The HMD 100 includes a camera inside and outside the housing 114, and includes a facial expression monitoring unit that monitors facial expressions of the user, specifically, movements of eyes, eyebrows, nose, lips, and the like. Various studies have been made on the relationship between emotions and facial expressions, and the following (1) to (5) are the relationships between emotions and facial expressions that have been announced so far.
(1) Happiness: The upper end of the lips is raised and pulled back.
(2) Surprise: Eyebrows go up and eyes wide open.
(3) Disgust: Raises upper lip and wrinkles nose.
(4) Anger: Vertical lines are formed between the eyebrows, and the mouth is strongly closed.
(5) Sadness: Both ends of the lips are lowered and the line of sight is lowered.

  In the HMD 100, the relationship between emotions and facial expressions is stored in a database, and the facial expression monitoring unit is configured to read emotion information corresponding to facial expressions by referring to the database when the facial expression of the user A is identified from the camera image. Yes. Since the emotion of a person changes every moment, the facial expression monitoring unit sends the emotion information extracted from the database to the robot 10 every time it determines that a change in emotion has occurred from a change in a specific part included in the camera image. Send.

  For example, when the facial expression monitoring unit determines from the camera image that “the upper end of the lips is raised and pulled backward”, the emotion information (“happiness”) associated with the facial expression information is read and transmitted to the robot 10. To do. Thereafter, when the facial expression monitoring unit determines from the camera image that “the eyebrows are raised and the eyes are wide open”, the emotion information (“surprise”) associated with the facial expression information is read and transmitted to the robot 10. To do. If facial expression information detected by the facial expression monitoring unit is not registered in the database, the facial expression monitoring unit transmits emotion information indicating that there is no user emotion to the robot 10. The sent emotion information only needs to be able to identify an emotion, and may be an information ID of emotion information such as No. (1) or No. (2), for example.

  Instead of the emotion information, the facial expression monitoring section itself displays facial expression information, for example, facial expression information indicating that the upper end of the lips is raised and pulled backward, or that the eyebrows are raised and the eyes are wide open. The facial expression information shown may be transmitted to the robot 10. The facial expression monitoring unit may transmit both emotion information and facial expression information to the robot 10. Hereinafter, such facial expression information and / or emotion information is collectively referred to as state information.

  Referring to FIG. 20, reception unit 60 receives state information, and state information acquisition unit 74 acquires state information. The drive control unit 76 controls the rotation of the drive motor 142 based on the state information. At this time, the drive control unit 76 refers to the operation table 78.

The operation table 78 holds facial expression information and / or emotion information in association with the operation mode of the protective cover 19.
FIG. 22 shows an example of the contents held in the operation table. In the operation table 78 shown in FIG. 22, emotion information and the operation mode of the protective cover 19 are recorded in association with each other. The drive control unit 76 drives the drive motor 142 so that the protective cover 19 operates in a corresponding operation mode based on the emotion information acquired from the HMD 100. For example, when the emotion information is “happiness”, the drive control unit 76 reciprocates 30 degrees back and forth (with an amplitude of 60 degrees) from the state where the protective cover 19 is raised 90 degrees from the initial operation position. The rotation of the drive motor 142 is controlled. Thereby, a person around the robot 10 can recognize that the user A feels happiness by seeing the movement of the protective cover 19. For example, when the facial expression monitoring unit in the HMD 100 can determine to the extent that the emotion is large or small, the drive control unit 76 may set the reciprocating speed according to the degree.

  In FIG. 22, the operation table 78 holds emotion information and the operation mode of the protective cover 19 in association with each other. However, the facial expression information and the operation mode of the protection cover 19 may be stored in association with each other.

  In the above-described example, the state information acquisition unit 74 acquires the facial expression information and / or emotion information of the user detected by the HMD 100 worn by the user. In this way, facial expression information and / or emotion information of the user wearing the HMD 100 may be detected by the HMD 100, but may be detected by an information processing apparatus provided outside the HMD 100. That is, the facial expression monitoring unit may be provided in the HMD 100, but may be provided in an information processing apparatus different from the HMD 100. At this time, the information processing apparatus specifies the facial expression or emotion of the user based on the camera image captured by the camera that captures the user, and transmits it to the robot 10. In any case, the state information acquisition unit 74 acquires facial expression information and / or emotion information of the user wearing the HMD 100, and the drive control unit 76 uses the facial expression information and / or emotion information of the user. Thus, the rotation of the drive motor 142 is controlled.

DESCRIPTION OF SYMBOLS 1 ... Information processing system, 10 ... Robot, 12 ... Actuator apparatus, 14a ... Right camera, 14b ... Left camera, 16a ... Right microphone, 16b ... Left microphone, 20 ... Case, 22 ... Input system, 24 ... Output system, 30 ... Pedestal, 32 ... First arc arm, 32a ... First through slot, 34 ... Second arcuate arm, 34a, second through hole, 36, housing, 38, cover, 40, leg, 42, insertion member, 42a, first regulating portion , 42b... Second restricting portion, 42c... Shaft portion, 50... Driving mechanism, 52... First motor, 54. ..Reception unit, 62... Sensor information acquisition unit, 64... Motion detection unit, 66. 8 ... Actuator control unit, 70 ... Audio data acquisition unit, 72 ... Audio processing unit, 74 ... Status information acquisition unit, 76 ... Drive control unit, 78 ... Operation table, 80 ... Image processing unit, 82 ... Audio processing unit, 82a ... Phase difference amplifying device, 84a ... First amplifier, 84b ... Second amplifier, 86a ... First adder, 86b ... Second adder, 88a ... Third amplifier, 88b ... Fourth amplifier, 90 ... Transmitter, 92 ... Image recording device, 100 ... HMD, 102 ... Display Panel, 104 ... Earphone, 106 ... Microphone, 108 ... Mounting band, 110 ... Output mechanism, 112 ... Mounting mechanism, 114 ... Case, 120 ... Control unit 122, storage unit, 124, posture sensor, 126, communication Control unit, 142... Drive motor, 144... Transmission member, 200... Processing device, 202 .. reception unit, 204... Sensor information acquisition unit, 206. ..Gaze direction determination unit, 210... Image determination unit, 212... Audio determination unit, 214 .. viewing data providing unit, 216... Transmission unit, 218. Image recording unit, 222... Voice recording unit.

Claims (8)

A robot including an actuator device and a housing whose posture can be changed by the actuator device, wherein the housing includes at least a camera, a microphone, and a speaker, and the actuator device is attached by a user. The attitude of the housing is linked to the movement of the head mounted display device,
A movable member operably supported in said housing,
A motor provided in the housing;
A transmission member for transmitting rotation of the motor to the movable member;
A drive control unit that controls the rotation of the motor based on the attitude of the housing;
A robot characterized by that. A robot including an actuator device and a housing whose posture can be changed by the actuator device, wherein the housing includes at least a camera, a microphone, and a speaker, and the actuator device is attached by a user. The attitude of the housing is linked to the movement of the head mounted display device,
A movable member operably supported in said housing,
A motor provided in the housing;
A transmission member for transmitting rotation of the motor to the movable member;
E Bei and a drive control unit for controlling the rotation of said motor,
The transmission member is a member that undergoes torsional elastic deformation,
A robot characterized by that.   The transmission member is a coil spring.
  The robot according to claim 1 or 2, wherein The housing constitutes the face of the robot;
The robot according to any one of claims 1 to 3, wherein: A housing having a camera, a microphone, and a speaker,
A movable member operatively supported by the housing;
A motor,
A transmission member that transmits the rotation of the motor to the movable member ,
The transmission member is a member that undergoes torsional elastic deformation,
A housing characterized by that. The transmission member is a coil spring.
The housing according to claim 5 . The movable member is moved between a closed position that protects the camera and other positions.
The housing according to claim 5 or 6 . The movable member is rotatably supported by the housing.
The housing according to any one of claims 5 to 7 , wherein:

Images