JP2009090420A - Robot remote control system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To properly adapt to a request by a colloquist for robots by enabling the robots to be remotely controlled by one or more operators meeting the requirement requested by the colloquist for the robots. <P>SOLUTION: A robot remote control system 10 comprises the robots 12a, 12b, a central control device 14, and operation terminals 16a-16c connected to each other through a network 100. When the robots 12a, 12b are difficult to adapt by autonomous control only, the request for calling operators is sent to the center control device 14. When receiving the request for calling, the center control device 14 selects one or more operators meeting the requirement requested by the colloquist according to a requirement list prepared from the voice of the colloquist for the robots 12a, 12b. Then, the one or more operators selected remotely control the robots 12a, 12b by using the operation terminal 16. Since the robots are remotely controlled by one or more operators meeting the requirement requested by the colloquist for the robots, the system can properly adapt to the requirement. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a robot remote control system, and more particularly to a robot remote control system for remotely controlling a robot, for example.

An example of this type of conventional robot remote control system is disclosed in Patent Document 1. The robot remote operation system of Patent Document 1 includes a plurality of robots, a central control device, and a plurality of operation terminals connected via a network. Each of the plurality of operation terminals is operated by an operator. The robot is an autonomous robot that can communicate with humans, and calls an operator as necessary when the situation becomes difficult to deal with by autonomous control alone. The central control device selects the most appropriate operation terminal (operator) and requests the operation terminal to perform remote operation. The robot that has called the operator controls its own operation based on the operation command from the operation terminal.
JP2007-190659 [B25J 13/00, B25J 5/00]

  However, in the background art shown in Patent Document 1, since it is a single operator who remotely controls the robot, a high knowledge level is required for each operator if all the conditions required by the person to be communicated are to be satisfied. Is required. Such a robot remote control system is difficult to construct and is not realistic. In addition, one operator may not be able to cope with it sufficiently.

  Therefore, the main object of the present invention is to provide a novel robot remote control system.

  Another object of the present invention is to provide a robot remote control system capable of appropriately responding to a request of a partner communicating with a remotely operated robot.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate the corresponding relationship with the embodiments described in order to help understanding of the present invention, and do not limit the present invention.

  According to a first aspect of the present invention, there is provided a robot that performs communication with humans through at least one of body movement and voice by remote operation or autonomous control, a plurality of operation terminals that perform remote operation of the robot via a network, and a network This is a robot remote control system including a central control device that mediates between the robot and the operation terminal via the. The robot determines whether it is difficult to execute communication actions with humans by autonomous control, and notifies the central controller of a request for remote operation when the determination result of the determination means is affirmative A central processing unit including a notification means, a request notification receiving means for receiving a notification of a remote operation request from the notification means, and a remote operation of the robot in response to the request notification receiving means receiving a notification of a remote operation request. Created by a condition list creating means for creating a condition list for conditions to be satisfied by an operator who performs the operation, a terminal information storage means for storing terminal information including attribute information of the operator for each of a plurality of operation terminals, and a condition creating means A comparison means for comparing the prepared condition list with the terminal information stored in the terminal information storage means, and a condition list based on the comparison result of the comparison means Selection means for selecting one or a plurality of operation terminals having terminal information satisfying all or a part of the conditions as an operation terminal for remotely operating the robot, and one or a plurality of operation terminals selected by the robot and the selection means; It is characterized by comprising a connection control means for enabling communication via a network.

  In the first invention, the robot remote control system (10) includes a robot (12a, 12b) that performs a communication action with a human by at least one of body motion and voice by remote control or autonomous control, and a network (100). ) Through a plurality of operation terminals (16a to 16c) that perform remote operation of the robot, and a central control device (14) that mediates between the robot and the operation terminal through a network. In the robot, the determination means (60, S403, S505 to S517) determines whether or not it is difficult to execute a communication action with a human by autonomous control. The notification means (60, 62, 82, 84, 100, S405, S601) is operated remotely by the central control unit when the determination result of the determination means is affirmative, that is, when communication with humans cannot be performed by autonomous control. Notify the request. In the central control unit, the request notification receiving means (S5) receives the notification of the request for remote operation from the notification means of the robot. Then, the condition list creation means (60, 62, 52, 54, 70, 82, 84, 100, S35, S65, S71, S101 to S113) that the request notification receiving means has received the notification of the remote operation request. In response to this, a condition list is created for conditions to be satisfied by an operator who performs remote operation of the robot. The terminal information storage means (S1) stores terminal information including operator attribute information for each of the plurality of operation terminals. Therefore, the comparison means (S67, S75, S201 to S229) compares the condition list created by the condition creation means with the terminal information stored in the terminal information storage means. The selection means (S7, S69, S77, S79, S81, S83) selects one or a plurality of operation terminals having terminal information satisfying all or a part of the condition list based on the comparison result of the comparison means. Select as an operation terminal for remote operation. The connection control means (S85 to S89) connects the robot and one or a plurality of operation terminals selected by the selection means so that they can communicate with each other via a network.

  For example, it is assumed that there are five conditions (requests) in a question from a person who communicates with a robot, and there is no operator who can satisfy all the five requests. In such a case, the central control unit selects one operator's operation terminal capable of responding to three requests and another operator's operation terminal capable of responding to the remaining two requests. Therefore, two operators can respond to questions (five requests) from a person who communicates with the robot by remotely operating the robot.

  According to the first invention, one or a plurality of operators that can respond to a request from a person who communicates with the robot is selected, and the robot and the operation terminal operated by the selected operator are communicably connected. Remote control can satisfy human demands. That is, it is possible to respond appropriately to the request of the other party who communicates with the robot and to respond smoothly.

  A second invention is according to the first invention, and the robot transmits a voice detection means for detecting a voice signal corresponding to a human voice, and a voice signal detected by the voice detection means to at least the central controller. A transmission means is further provided. The central control unit includes: a voice receiving unit that receives a voice signal from the transmission unit; a voice recognition unit that recognizes a human voice based on the voice signal received by the voice receiving unit; and a recognition result of the voice recognition unit. Language specifying means for specifying a language spoken by a human is further provided. The condition list includes language information as a condition.

  In the second invention, the voice detection means (54, 60, 62, 70) of the robot detects a voice signal corresponding to a human voice. The transmission means (60, 62, 82, 84, 100, S605) transmits the audio signal detected by the audio detection means to at least the central controller. However, when the robot and the operation terminal are communicably connected, the audio signal is transmitted to the operation terminal. The voice receiving means (S39) of the central control unit receives the voice signal from the sending means. The voice recognition means (S41, S45, S49) recognizes human voice based on the voice signal received by the voice reception means. The language specifying means (S43, S47, S51 to S63) specifies a language spoken by a human based on the recognition result of the voice recognition means.

  For example, the condition list includes language information spoken by a person communicating with the robot. Therefore, the central control device selects at least an operation terminal operated by an operator who can speak the same language as a human being communicating with the robot.

  According to the second invention, since an operation terminal operated by an operator who can speak the same language as the person who communicates with the robot can be selected, the person who communicates with the robot and the operator have a direct conversation. be able to. Thus, the operator can accurately understand and respond to the human request.

  A third invention is dependent on the second invention, and when the selection means satisfies a part of the condition list by the terminal information of one operation terminal, the selection means satisfies the remaining conditions excluding the part of the condition Including other terminal selection means for selecting one or a plurality of other operation terminals having terminal information to be executed, and the other terminal selection means performs other operations until a condition exceeding a certain ratio with respect to all conditions in the condition list is satisfied. Continue selecting terminals.

  In the third invention, when the other terminal selection means (S69, S75, S77, S79) satisfies a part of the conditions in the condition list by one operation terminal, the other conditions are satisfied except for the part of the conditions. One or more other operation terminals having terminal information to be selected are selected. The other terminal selection means continues to select other operation terminals until a condition exceeding a certain ratio with respect to all conditions in the condition list is satisfied.

  For example, when the fixed ratio is 80% and there are five requests from a person communicating with the robot, the central control device selects another operation terminal until at least four requests are satisfied. In this case, the robot is remotely operated by a plurality of operators.

  Here, it is only necessary to satisfy a certain ratio or more. If all the conditions must be satisfied, the selection process of the other terminal selection means may become an infinite loop. Because. However, even in such a case, since it is possible to respond to requests of a certain ratio or more, an inconvenient state that cannot be handled at all can be avoided.

  According to the third aspect of the present invention, even if all the requests from the human being who communicates with the robot are not satisfied, it is possible to avoid the inconvenience that cannot be dealt with at all by the operator remotely operating the robot. It is possible to respond as much as possible. In addition, it is possible to prevent the operation terminal selection process from entering an infinite loop.

  A fourth invention is according to the third invention, wherein the central control device further comprises a terminal number determination means for determining whether or not the number of operation terminals selected by the selection means is equal to or greater than a certain number, and the other terminal The selection means continues to select other operation terminals until it is determined by the terminal number determination means that the number is a certain number or more.

  In the fourth invention, the terminal number determination means (S81) of the central control apparatus determines whether or not the number of operation terminals selected by the selection means is a certain number or more. The other terminal selection means continues to select other operation terminals until it is determined by the terminal number determination means that the number is equal to or greater than a certain number. That is, the number of operation terminals is limited. For example, if the fixed number is 3, and if three operation terminals are selected by the central control unit, the condition included in the condition list satisfies a certain ratio (for example, 80%) or more. Even if not, the selection of another operation terminal is terminated.

  According to the fourth invention, similarly to the third invention, it is possible to prevent the selection process of the operation terminal from becoming an infinite loop. In addition, since the number of operation terminals is limited, it is possible to prevent the number of operators to be remotely operated from increasing, and it is possible to prevent time and the like from being adjusted between operators. Thereby, the robot can be remotely operated smoothly.

  The fifth invention is dependent on the second invention, and when the selection means satisfies a part of the conditions in the condition list based on the terminal information of one operation terminal, the remaining conditions excluding the part of the condition are included. Including other terminal operation means for selecting one or a plurality of other operation terminals having satisfied terminal information, and the central control device determines whether or not the number of operation terminals selected by the selection means is equal to or greater than a certain number Terminal number determination means is further provided, and the other terminal selection means continues to select other operation terminals until it is determined by the terminal number determination means that the number is equal to or greater than a certain number.

  In the fifth invention, when the other terminal selection means (S69, S75, S77, S79) satisfies a partial condition in the condition list according to the terminal information of one operation terminal, the remaining condition excluding the partial condition is excluded. One or a plurality of other operation terminals having terminal information satisfying the above condition are selected. The terminal number determination means (S81) of the central control apparatus determines whether or not the number of operation terminals selected by the selection means is a certain number or more. The other terminal selection means continues to select other operation terminals until it is determined by the terminal number determination means that the number is equal to or greater than a certain number.

  According to the fifth invention, the operation terminal selection process is prevented from becoming an infinite loop, and the number of operators can be limited to smoothly operate the robot remotely.

  A sixth invention is dependent on any one of the first to fifth inventions, and the central control device controls the robot information transmitted from the robot to the plurality of operation terminals at a communication speed between the robot and the plurality of operation terminals. Transmission information control means is further provided.

  In the sixth invention, the transmission information control means (S11, S301 to S329) of the central control device controls the robot information transmitted from the robot to the plurality of operation terminals according to the communication speed between the robot and one or more operation terminals. . For example, it is assumed that there are three types of data transmitted from the robot to the operation terminal: video data, audio data, and distance data. The operator remotely controls the robot, for example, referring to data transmitted from the robot. For example, when the communication speed is high, the robot transmits video data, audio data, and distance data. When the communication speed is low, the robot transmits only distance data. Furthermore, when the communication speed is about the middle of them, the robot transmits only video data or only audio data.

  According to the sixth aspect, since the type of data transmitted by the robot is selected according to the communication speed, for example, the amount of data to be transmitted can be adjusted. Therefore, it is possible to prevent the network from becoming overloaded. That is, since the communication is not delayed unnecessarily, the robot can be remotely operated smoothly.

  According to the present invention, one or a plurality of operators capable of responding to a request from a person who communicates with the robot is selected, and the robot and the operation terminal operated by the selected operator are communicably connected. Can satisfy human demands. That is, it is possible to respond appropriately to the request of the other party who communicates with the robot and to respond smoothly.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  Referring to FIG. 1, a robot remote control system 10 of this embodiment includes robots 12a and 12b. The robots 12a and 12b are connected to the central controller 14 and the operation terminals 16a, 16b and 16c via the network 100. Hereinafter, when it is not necessary to distinguish the robots 12a and 12b, they will be referred to as robots 12. Similarly, when there is no need to distinguish each of the operation terminals 16a-16c, they will be referred to as operation terminals 16.

  The robot 12 is an interaction-oriented robot (communication robot), and mainly includes a communication action including at least one of body movement such as gesture gesture and voice with a communication target (communication target) such as the human A. It has a function to execute. The robot 12 is, for example, a reception robot or a customer service robot, and is disposed in various places and situations such as a certain event venue or a sales floor of a home appliance mass retailer. To play a role.

  However, when it becomes difficult to respond only by autonomous control, the robot 12 calls an operator when a more detailed response (communication or the like) is required by a human. When the robot 12 receives operation command information input by the calling operator, the robot 12 controls its own operation based on the received operation command. That is, the robot 12 calls an operator according to its own situation and is operated by the operator.

  For simplicity, FIG. 1 shows two robots (12a, 12b) and three operation terminals (16a-16c) for remotely operating the robot 12, but it is not necessary to be limited by these. Absent. One robot 12 or three or more robots 12 may be used. Further, the number of operation terminals is not limited as long as it is two or more.

  The hardware configuration of the robot 12 will be described with reference to FIG. FIG. 2 is a front view showing the appearance of the robot 12 of this embodiment. The robot 12 includes a carriage 18, and two wheels 20 and one slave wheel 22 for autonomously moving the robot 12 are provided on the lower surface of the carriage 18. The two wheels 22 are independently driven by a wheel motor 24 (see FIG. 3), and the carriage 18, that is, the robot 12 can be moved in any direction, front, back, left, and right. The slave wheel 22 is an auxiliary wheel that assists the wheel 20. Therefore, the robot 12 can move in the arranged space by autonomous control. However, the robot 12 may be fixedly arranged at a certain place.

  A cylindrical sensor mounting panel 26 is provided on the carriage 18, and a number of infrared distance sensors 28 are mounted on the sensor mounting panel 26. These infrared distance sensors 28 measure the distance from the sensor mounting panel 26, that is, an object (such as a human being or an obstacle) around the robot 12.

  The body 30 is provided on the sensor mounting panel 26 so as to stand upright. In addition, the above-described infrared distance sensor 28 is further provided in the upper front upper portion of the body 30 (a position corresponding to a human chest), and measures the distance mainly to a human in front of the robot 12. Further, the body 30 is provided with a support column 32 extending from substantially the center of the upper end of the side surface, and an omnidirectional camera 34 is provided on the support column 32. The omnidirectional camera 34 captures the surroundings of the robot 12 and is distinguished from an eye camera 58 described later. As this omnidirectional camera 34, for example, a camera using a solid-state imaging device such as a CCD or a CMOS can be adopted. In addition, the installation positions of the infrared distance sensor 28 and the omnidirectional camera 34 are not limited to the portions, and can be changed as appropriate.

  An upper arm 38R and an upper arm 38L are provided at upper end portions on both sides of the body 30 (a position corresponding to a human shoulder) by a shoulder joint 36R and a shoulder joint 36L, respectively. Although illustration is omitted, each of the shoulder joint 36R and the shoulder joint 36L has three orthogonal degrees of freedom. That is, the shoulder joint 36R can control the angle of the upper arm 38R around each of the three orthogonal axes. A certain axis (yaw axis) of the shoulder joint 36R is an axis parallel to the longitudinal direction (or axis) of the upper arm 38R, and the other two axes (pitch axis and roll axis) are orthogonal to the axis from different directions. It is an axis to do. Similarly, the shoulder joint 36L can control the angle of the upper arm 38L around each of three orthogonal axes. A certain axis (yaw axis) of the shoulder joint 36L is an axis parallel to the longitudinal direction (or axis) of the upper arm 38L, and the other two axes (pitch axis and roll axis) are orthogonal to the axis from different directions. It is an axis to do.

  In addition, an elbow joint 40R and an elbow joint 40L are provided at the tips of the upper arm 38R and the upper arm 38L, respectively. Although illustration is omitted, each of the elbow joint 40R and the elbow joint 40L has one degree of freedom, and the angles of the forearm 42R and the forearm 42L can be controlled around this axis (pitch axis).

  A sphere 44R and a sphere 44L corresponding to a human hand are fixedly provided at the tips of the forearm 42R and the forearm 42L, respectively. However, when a finger or palm function is required, a “hand” in the shape of a human hand can be used. Although not shown, the front surface of the carriage 18, the portion corresponding to the shoulder including the shoulder joint 36 </ b> R and the shoulder joint 36 </ b> L, the upper arm 38 </ b> R, the upper arm 38 </ b> L, the forearm 42 </ b> R, the forearm 42 </ b> L, the sphere 44 </ b> R, and the sphere 44 </ b> L, A contact sensor (shown generically in FIG. 3) 46 is provided. The contact sensor 46 on the front surface of the carriage 18 detects contact of a person or another obstacle with the carriage 18. Therefore, the robot 12 can detect that there is a contact with an obstacle during its movement and immediately stop the driving of the wheel 20 to suddenly stop the movement of the robot 12. Further, the other contact sensors 46 detect whether or not the respective parts are touched. In addition, the installation position of the contact sensor 46 is not limited to the said site | part, and may be provided in an appropriate position (position corresponding to a person's chest, abdomen, side, back, and waist).

  A neck joint 48 is provided at the upper center of the body 30 (a position corresponding to a person's neck), and a head 50 is further provided thereon. Although illustration is omitted, the neck joint 48 has a degree of freedom of three axes, and the angle can be controlled around each of the three axes. A certain axis (yaw axis) is an axis directed directly above (vertically upward) of the robot 12, and the other two axes (pitch axis and roll axis) are axes orthogonal to each other in different directions.

  The head 50 is provided with a speaker 52 at a position corresponding to a human mouth. The speaker 52 is used for the robot 12 to communicate with humans around it by voice or sound. A microphone 54R and a microphone 54L are provided at a position corresponding to a human ear. Hereinafter, the right microphone 54R and the left microphone 54L may be collectively referred to as a microphone 54. The microphone 54 captures ambient sounds, particularly a human voice that is an object for performing communication. Furthermore, an eyeball portion 56R and an eyeball portion 56L are provided at positions corresponding to human eyes. The eyeball portion 56R and the eyeball portion 56L include an eye camera 58R and an eye camera 58L, respectively. Hereinafter, the right eyeball portion 56R and the left eyeball portion 56L may be collectively referred to as the eyeball portion 56. Further, the right eye camera 58R and the left eye camera 58L may be collectively referred to as an eye camera 58.

  The eye camera 58 captures a human face approaching the robot 12, other parts or objects, and captures a corresponding video signal. The eye camera 58 can be the same camera as the omnidirectional camera 34 described above. For example, the eye camera 58 is fixed in the eyeball unit 56, and the eyeball unit 56 is attached to a predetermined position in the head 50 via an eyeball support unit (not shown). Although illustration is omitted, the eyeball support portion has two degrees of freedom, and the angle can be controlled around each of these axes. For example, one of the two axes is an axis (yaw axis) in a direction toward the top of the head 50, and the other is orthogonal to the one axis and goes straight in a direction in which the front side (face) of the head 50 faces. It is an axis (pitch axis) in the direction to be. By rotating the eyeball support portion around each of the two axes, the tip (front) side of the eyeball portion 56 or the eye camera 58 is displaced, and the camera axis, that is, the line-of-sight direction is moved. It should be noted that the installation positions of the above-described speaker 52, microphone 54, and eye camera 58 are not limited to the portions, and may be provided at appropriate positions.

  As described above, the robot 12 of this embodiment includes independent two-axis driving of the wheels 20, three degrees of freedom of the shoulder joint 36 (6 degrees of freedom on the left and right), and one degree of freedom of the elbow joint 40 (two degrees of freedom on the left and right). It has a total of 17 degrees of freedom: 3 degrees of freedom for the neck joint 48 and 2 degrees of freedom for the eyeball support (4 degrees of freedom on the left and right).

  FIG. 3 is a block diagram showing the electrical configuration of the robot 12. With reference to FIG. 3, the robot 12 includes a CPU 60. The CPU 60 is also called a microcomputer or a processor, and is connected to the memory 64, the motor control board 66, the sensor input / output board 68, and the audio input / output board 70 via the bus 62.

  The memory 64 includes a ROM, an HDD, and a RAM (not shown). In the ROM and the HDD, a control program for controlling the operation of the robot 12 is stored in advance. For example, a detection program for detecting the output (sensor information) of each sensor and a communication program for transmitting / receiving necessary data and commands to / from external computers (such as the central control device 14 and the operation terminal 16). To be recorded. The RAM is used as a work memory or a buffer memory.

  The motor control board 66 is composed of, for example, a DSP, and controls driving of motors of axes such as arms, neck joints, and eyeballs. That is, the motor control board 66 receives control data from the CPU 60, and controls two motors (collectively indicated as “right eyeball motor 72” in FIG. 3) that control the angles of the two axes of the right eyeball portion 56R. Control the rotation angle. Similarly, the motor control board 66 receives control data from the CPU 60, and shows two motors for controlling the respective angles of the two axes of the left eyeball portion 56L (in FIG. 3, collectively referred to as “left eyeball motor 74”). ) To control the rotation angle.

  The motor control board 66 receives control data from the CPU 60, and includes three motors for controlling the angles of the three orthogonal axes of the right shoulder joint 36R and one motor for controlling the angle of the right elbow joint 40R. The rotation angles of a total of four motors (collectively indicated as “right arm motor 76” in FIG. 3) are controlled. Similarly, the motor control board 66 receives control data from the CPU 60, three motors for controlling the angles of the three orthogonal axes of the left shoulder joint 36L, and one motor for controlling the angle of the left elbow joint 40L. The rotation angles of a total of four motors (collectively indicated as “left arm motor 78” in FIG. 3) are controlled.

  Further, the motor control board 66 receives control data from the CPU 60, and controls three motors that control the angles of the three orthogonal axes of the neck joint 48 (in FIG. 3, collectively indicated as "head motor 80"). Control the rotation angle. The motor control board 66 receives control data from the CPU 60 and controls the rotation angles of the two motors that drive the wheels 20 (collectively indicated as “wheel motors 24” in FIG. 3). In this embodiment, a motor other than the wheel motor 24 uses a stepping motor (that is, a pulse motor) in order to simplify the control. However, a DC motor may be used similarly to the wheel motor 24. The actuator that drives the body part of the robot 12 is not limited to a motor that uses a current as a power source, and may be changed as appropriate. For example, in another embodiment, an air actuator may be applied.

  Similarly, the sensor input / output board 68 is also constituted by a DSP, and takes in signals from each sensor and gives them to the CPU 60. That is, data relating to the reflection time from each of the infrared distance sensors 28 is input to the CPU 60 through the sensor input / output board 68. The video signal from the omnidirectional camera 34 is input to the CPU 60 after being subjected to predetermined processing by the sensor input / output board 68 as necessary. Similarly, the video signal from the eye camera 58 is also input to the CPU 60. Further, signals from the plurality of contact sensors described above (collectively indicated as “contact sensors 46” in FIG. 3) are provided to the CPU 60 via the sensor input / output board 68.

  Similarly, the voice input / output board 70 is also configured by a DSP, and voice or voice according to voice synthesis data provided from the CPU 60 is output from the speaker 52. Also, voice input from the microphone 54 is given to the CPU 60 via the voice input / output board 70.

  The CPU 60 is connected to the communication LAN board 82 via the bus 62. The communication LAN board 82 is configured by a DSP and provides transmission data given from the CPU 60 to the wireless communication device 84, and sends the transmission data from the wireless communication device 84 to the external computer (the central control device 14 and the operation terminal via the network 100. 16). The communication LAN board 82 receives data via the wireless communication device 84 and gives the received data to the CPU 60. That is, the robot 12 can perform wireless communication with the central control device 14 and the operation terminal 16 by the communication LAN board 82 and the wireless communication device 84.

  Further, the wireless tag reader 86 is connected to the CPU 60 via the bus 62. The wireless tag reader 86 receives a radio wave superimposed with identification information transmitted from the wireless tag (RFID tag) via an antenna (not shown). The wireless tag reader 86 amplifies the received radio wave signal, separates the identification signal from the radio wave signal, demodulates (decodes) the identification information, and supplies the identification information to the CPU 60. The wireless tag is attached to a person in an event venue or a sales floor of a home appliance mass retailer, and the wireless tag reader 86 detects a wireless tag within a communicable range. Note that the wireless tag may be an active type or a passive type that is driven in accordance with radio waves transmitted from the wireless tag reader 86.

  Returning to FIG. 1, the central control device 14 is a computer that controls the call of the operator in the robot remote operation system 10, and manages information indicating the states of the robot 12 and the operation terminal 16. Although not shown, the central control device 14 includes a CPU, and a memory, a database, a communication device, and the like are connected to the CPU. The central control device 14 is connected to the network 100 via a communication device in a wired or wireless manner.

  The memory stores a control program and necessary data for controlling the operation of the central controller 14. The control program includes, for example, a communication program for transmitting and receiving necessary data and commands between the robot 12 and the operation terminal 16, and an appropriate operator (operation terminal 16 when an operator call request is received from the robot 12. ) Including a selection program for selecting.

  The database stores a robot information table (see FIG. 4) indicating information on the robot 12, and an operator terminal information table (see FIG. 5) indicating information on the operation terminal 16. The database also stores human information (for example, wireless tag identification information, language used, and action history) existing at event venues and sales floors of home appliance mass retailers.

  Referring to FIG. 4, the robot information table stores information such as the robot name, state, position, and conversation partner. The name of the robot 12 (for example, R1, R2,...) Is registered in the robot name column. In the status column, the current status of the corresponding robot 12 is registered. Specifically, “BUSY” or “IDLE” is registered. BUSY means a state in which a communication action such as dialogue with a person is executed on the sales floor of a consumer electronics retailer. IDLE means a state in which a communication action such as dialogue with a person is not executed, that is, a free state.

  Returning to FIG. 4, information on the current position of the robot 12 is registered in the position column. In this embodiment, information indicating a place such as a building or a venue where the robot 12 is arranged, and coordinates of a position where the robot 12 currently exists in the place are registered. Specifically, “home appliance mass retailer”, “sales floor”, and “120, 140” are registered in the position column corresponding to the robot name “R1”. From this, it can be seen that the robot 12 with the robot name “R1” is placed on the “sales floor” of the “home appliance mass retailer” and its current position is a position represented by coordinates (120, 140).

  The memory or database of the central control unit 14 stores map (map) data such as a place where the robot 12 is arranged, and the map data is expressed in XY coordinates, thereby expressing the position of the robot 12 in coordinates. ing. However, since the robot 12 moves, the amount of movement control may be obtained from the robot 12, or the position of the robot 12 may be monitored using a separate environmental sensor (camera, infrared sensor, etc.).

  In the dialogue partner column, information on the person who is currently performing the communication action of the robot 12 is registered. Specifically, a name (for example, Taro Yamada), a language to be used (for example, Japanese), a current position (coordinates), and the like are registered in association with identification information (ID) of the conversation partner. In addition, when the robot 12 is not interacting with a human, “NULL” indicating that there is no target for communication behavior is registered.

  The central control device 14 creates and updates the robot information table by acquiring information indicating the state of the robot 12, information indicating the position, information indicating the conversation partner, and the like at regular intervals through communication with the robot 12. Although not shown in FIG. 4, for example, information such as the number of times the conversation partner has visited, the elapsed time since the visit, and the history of actions that the robot 12 has already performed on the partner (for example, greetings have been completed, etc.) If necessary, it may be acquired from the database and registered as appropriate. In the robot information table, the address (IP address) of the robot 12 may also be registered.

  Referring to FIG. 5, information such as ID, name, state, location, status value, and skill list is registered in the operator terminal information table. Identification information for identifying the operation terminal 16 or an operator who operates the operation terminal 16 is registered in the ID column. In this embodiment, the ID is represented by a numerical value. However, “id_max” is a constant corresponding to the maximum value of the operation terminal or the operator. In the name column, the name of the operator who operates the operation terminal 16 is registered. In the status column, the current status of the operation terminal 16 is registered. Specifically, “IDLE” and “BUSY-R1” are described. “IDLE” means a state of waiting for a call from the central control device 14 without operating the robot 12 (standby state). “BUSY” means a state where the robot 12 is remotely operated. For example, when “BUSY-R1” is described, the robot 12 having the robot name “R1” is operated. I understand that. The location (prefecture or region) where the operation terminal 16 is installed is registered in the location column.

  The status value information is further classified into information on experience, the number of languages, and the number of skills. This status value is information to be compared with conditions (user conditions) arbitrarily determined by the user of the robot remote control system 10 when searching for an operator, and can be set and changed by the user. . Details of the operator search process (see FIG. 13) will be described later. In the experience column, the operation history (year / month) of the robot 12 is registered. In the number of languages column, the number of languages that the operator can speak is registered. For example, when the operator can speak Japanese, English, and German, “3” is registered in the number of languages column. The number of skills, knowledge, etc. (hereinafter referred to as “skills”) possessed by the operator is registered in the skill number column. Specific skill contents are registered in the skill list. The skill list information describes the skill of each operator. This skill list is compared with conditions required by the robot 12 when selecting an operator. The details of the operator selection process (see FIGS. 8 to 11) will be described later.

  The skill list includes information on language, skill 1, skill 2,..., Skill M (M: natural number). In the language column, languages that the operator can speak are registered. In the fields of Skill 1, Skill 2,..., Skill M, a field of expertise or a field in charge (content) is registered as the skill of the operator. Note that the address (IP address) of the operation terminal 16 may also be registered in the operator terminal information table.

  For example, the central control device 14 obtains information on the position, state, location, status value, and skill list of the operation terminal 16 at regular intervals by communication with the operation terminal 16, and generates an operator terminal information table. Update. Alternatively, when there is a change in the state, location, status value, and skill list, the operator operating the operation terminal 16 notifies the administrator of the central control device 14, and the administrator of the central control device 14 The information table may be generated and updated.

  For example, referring to this operator terminal information table, it can be seen that the operation terminal 16 with ID “1” is installed in “Osaka” and the name of the operator who operates the operation terminal 16 is “Yamada”. . The operator (Yamada) can speak “Japanese”, the number of languages is “1”, and the operation history of the robot 12 is “36 months”. Further, it can be seen that the current state of the operation terminal 16 is “IDLE”. Furthermore, the number of skills of the operator (Yamada) is “1”, and it can be seen from the contents of skill 1 that he is “person in charge of personal computer”.

  Similarly, the operation terminal 16 with the ID “2” is installed in “Tokyo”, and it is understood that the name of the operator who operates the operation terminal 16 is “Suzuki”. The operator (Suzuki) can speak “Japanese” and “English”, the number of languages is “2”, and the operation history of the robot 12 is “2 months”. Further, it can be seen that the current state of the operation terminal 16 is “IDLE”. Furthermore, the number of skills of the operator (Suzuki) is “2”, and the content of skill 1 indicates that “iPod is in charge” (“iPod” is a registered trademark). You can see that there is knowledge of "Rugby".

  Further, it is understood that the operation terminal 16 having the operation terminal ID “id_max” is installed in “Fukuoka”, and the name of the operator who operates the operation terminal 16 is “Miyazaki”. The operator (Miyazaki) can speak “Japanese”, “English”, and “German”, the number of languages is “3”, and the operation history of the robot 12 is “12 months”. I understand. Further, it can be seen that the current state of the operation terminal 16 is “BUSY-R1” (a state in which the robot R1 is being operated). Furthermore, the number of skills of the operator (Miyazaki) is “M”. From the contents of skill 1, it is understood that the person is “in charge of home appliances”, and from the contents of skill 2, it is understood that there is knowledge of “soccer”. It can be seen from the contents of M that there is knowledge of “baseball”.

  Although illustration is omitted, when the operation terminal 16 is not operated by the operator (when offline), “NULL” (that is, a state in which remote operation is not possible) may be registered.

  The operation terminal 16 shown in FIG. 1 is a computer and includes a CPU (not shown). A memory, a display device, an input device, a speaker, a microphone, a communication device, and the like are connected to the CPU. The operation terminal 16 is connected to the network 100 via a communication device in a wired or wireless manner. In the memory, a control program and necessary data for controlling the operation of the operation terminal 16 are recorded. The control program includes, for example, a communication program for transmitting and receiving necessary data and commands between the robot 12 and the central control device 14, a detection program for detecting an operation command input from the input device, and an image on the display device. Includes a display program for display.

  The display device is an LCD or CRT. As will be described later, a remote operation screen 210 (see FIG. 6) including a robot camera image 212, robot information 214, an operation panel 216, and the like is displayed on the display device. The The input device is a computer mouse, a keyboard, a touch panel, or the like. For example, the operator can input a remote operation command for remotely operating the robot 12 by operating the input device while viewing the remote operation screen 210 displayed on the display device. The speaker outputs a human voice mainly detected through the microphone 54 of the robot 12 that is remotely operated. Further, the microphone mainly detects the voice of the operator, and the voice of the operator is output through the speaker 52 of the robot 12.

  Referring to FIG. 6, on remote operation screen 210, for example, robot camera image 212, robot information 214, and operation panel 216 are displayed. In the robot camera image 212, a captured image of the eye camera 58 received from the robot 12 is displayed. Thus, the operator can see an image captured by the eye camera 58 of the robot 12, for example, a human being interacting in real time.

  In the robot information 214, information on the robot 12, the situation, information on the conversation partner, and the like are displayed. Specifically, the place where the robot 12 is placed (such as an event venue or a home appliance mass retailer), the name (or identification information), a map of the place where the robot 12 is placed, and information on the person who is interacting are displayed. On the map, icons indicating the robot 12 and the conversation partner are displayed corresponding to each location. Accordingly, the operator can easily grasp where the robot 12 is located at the installation location and where the conversation partner is located with respect to the robot 12.

  The operation terminal 16 may store map information in advance or may receive the map information together with information related to the robot 12 from the central control device 14. The positions of the robot 12 and the conversation partner are transmitted from the robot 12 to the operation terminal 16 at regular intervals, for example. Since the robot 12 stores the initial position and orientation, the robot 12 always knows its current position and orientation, and estimates the approximate position of the conversation partner from the output information of the infrared distance sensor 28 and its own position. it can. In addition, an icon indicating a person other than the conversation partner existing at the place may be displayed on the map. In this case, the operator can know the situation of the robot 12 at the location in more detail.

  Further, the position information of the person at the location where the robot 12 is arranged is detected by another computer connected to the sensor of the environment using a sensor of the environment such as a ceiling camera or a wireless tag reader installed in the surroundings. Alternatively, it may be given from the other computer to the robot 12 and transmitted from the robot 12 to the operation terminal 16. Alternatively, the position information may be transmitted from the other computer described above to the operation terminal 16 via the central control device 14. In addition, since the position of the robot 12 and the like can be detected by the environmental sensor, the robot 12 may acquire information such as its own position and orientation from the above-mentioned other computer together with the human position information.

  In addition, as information of the person who is interacting, the name, affiliation (for example, counter person or visitor), the number of visits to the store, the date of the previous visit, the elapsed time since this visit, and the robot 12 has already performed for the partner. A history of communication behavior is displayed. As a result, the operator can know information on the conversation partner of the robot 12. Information on the conversation partner is transmitted from the central controller 14 to the operation terminal 16. Note that the information presented to these operators is an example, and appropriate information is appropriately displayed in the robot information 214. With such robot information 214, the operator can easily grasp the information of the robot 12 and the conversation partner and the situation where both are placed.

  Further, when the robot 12 and the conversation partner are talking, voice information is transmitted from the robot 12 and the voice is output from the speaker of the operation terminal 16. Therefore, the operator can grasp the current situation more easily by further listening to the conversation partner's words. In addition, since the operator can easily determine the contents of the question from the conversation partner while confirming the status of the robot 12 on the remote operation screen, the answer to the question is interactively communicated via the operation terminal 16 and the robot 12. You can answer the other party.

  On the operation panel 216, an operation command for controlling the operation of the robot 12 is input by the operator operating the input device. For example, as shown in FIG. 6, the operation panel 216 is provided with a button for instructing movement, a button for instructing communication behavior, a button for instructing the end of remote operation, and the like. Specifically, the “forward” button is a button for moving the robot 12 forward. When the operator selects the “forward” button, the robot 12 moves forward, for example, by a certain distance. Similarly, the “reverse” button causes the robot 12 to retract, the “right” button causes the robot 12 to turn right, and the “left” button causes the robot 12 to turn left. The “bow” button is a button for causing the robot 12 to execute a communication action of bowing. Specifically, when the operator selects the “bow” button, the robot 12 controls the neck joint 48 and turns the head 50 downward, that is, bows. The “right pointing” button is a button for causing the robot 12 to execute a communication action of pointing right. Specifically, when the operator selects the “right finger” button, the robot 12 controls the right shoulder joint 36R and the right elbow joint 40R to raise the arm forward. Similarly, when the operator selects a button displayed on the operation panel 216, the robot 12 can execute a predetermined communication action including gestures and speech such as nodding and pointing left. The “end” button is a button for ending the remote operation. When the operator selects the “end” button, the robot 12 shifts from the remote operation mode to the autonomous control mode.

  The operation terminal 16 is called based on the operator selected by the central controller 14. The question of the conversation partner is acquired by the microphone 54 of the robot 12 and is output from the speaker via the network 100. When the operator hears the question asked by the conversation partner, the operator responds by answering the question of the conversation partner with the communication behavior corresponding to the operation panel 216 or the operator's own voice.

  In the robot remote operation system 10, as described above, the robot 12 usually performs communication behavior with humans by autonomous control and provides services such as reception and guidance. However, the robot 12 calls the operation terminal 16, that is, the operator as necessary, such as when it becomes difficult to deal with by autonomous control alone. Although not shown, the robot 12 stores in advance a condition for calling an operator (operator calling condition) in the memory 64. When it is determined that the operator calling condition is satisfied, the robot 12 calls the operator. The robot 12 transmits an operator call request to the central controller 14. When the central control device 14 receives a call request from the operator, the central control device 14 selects the operator. Briefly, a condition (question or the like) required by a person with whom the robot 12 interacts (communications) is detected, and one or a plurality of operators that meet this condition are selected. Then, the robot information is transmitted to the selected operator, and the operation terminal 16 operated by each operator and the robot 12 are communicably connected.

  Specifically, the CPU of the central control device 14 executes a plurality of processes in parallel, including the processes shown in FIGS. Further, the CPU 60 of the robot 12 executes a plurality of processes in parallel including the processes shown in FIGS. Furthermore, the CPU of the operation terminal 16 executes a plurality of processes in parallel including the remote operation handling process shown in FIG.

  As shown in FIG. 7, when starting the overall processing, the CPU of the central control device 14 acquires the state of each operation terminal 16 and updates the table in step S1. Specifically, the CPU of the central control unit 14 acquires information including the position, operator information, current usage status, and the like from each operation terminal 16, and creates an operator terminal information table as shown in FIG. Update. In the next step S3, the state of each robot 12 is acquired and the table is updated. Specifically, the CPU of the central controller 14 acquires information including the position of the robot 12, the current operating state, information on the conversation partner, and the like from each robot 12, and the robot information as illustrated in FIG. 4. Update the table.

  Subsequently, in step S5, it is determined whether there is a call from the robot x, that is, whether an operator call request from a certain robot 12 is received. Here, “robot x” means one of the plurality of robots 12. If “NO” in the step S5, that is, if there is no call from the robot x, it is determined whether or not the entire process is ended in a step S15. Here, for example, it is determined whether or not an end instruction is given from the user of the robot remote control system 10 or the administrator of the central controller 14. If “YES” in the step S15, that is, if there is an end instruction, the entire process is ended as it is. On the other hand, if “NO” in the step S15, that is, if there is no end instruction, the process returns to the step S1. As described above, in the central control device 14, the update of the operator operation terminal table, the update of the robot information table, and the determination of the presence / absence of a call request are repeatedly executed at regular intervals.

If YES in step S5, that is, if there is a call from the robot x, an operator selection process (see FIGS. 8 to 11) described later is performed in step S7. In the next step S9, it is determined whether or not the operator OP _k has been selected. The “operator OP _k ” means one or more operators selected in the operator selection process. However, the operator may not be selected. That is, the variable k takes one of 0, 1,..., J (maximum value). If “NO” in the step S9, that is, if no operator OP _k is selected, an “error” is transmitted to the robot x that has requested the operator to be called in a step S13, and the process proceeds to the step S15. move on. That is, the CPU of the central controller 14 informs the robot 12 that there is no operator OP _k (operation terminal 16) that operates the robot x.

On the other hand, if “YES” in the step S9, that is, if the operator OP _k (the operation terminal 16) that operates the robot x that has requested the operator to call is selected, a sensor information selection process described later is performed in a step S11. (See FIG. 14) is executed, and the process proceeds to step S15. In step S11, the amount of data is reduced by selecting sensor information (data type) to be transmitted from the robot x to the operation terminal 16 according to the communication speed between the robot x and the operation terminal 16 operated by the operator OP _k. Adjusted. This is to prevent a delay in communication time.

FIGS. 8-11 is a flowchart which shows the operator selection process of step S7 shown in FIG. As shown in FIG. 8, when starting the operator selection process, the CPU of the central controller 14 initializes variables in step S31. Specifically, 1 is set for each of variable k and variable j, and 0 is set for each of variable jpn, variable eng, and variable deu. Here, the variable k is a variable for individually identifying the selected operator OP _k , and the variable j indicates the count value (total number) of the number of the selected operator OP _k . The variable jpn stores the calculation result obtained by calculating the Japanese character from the voice of the person interacting with the robot x, and the variable eng indicates the calculation result of calculating the English character from the voice of the person interacting with the robot x. The variable deu stores the calculation result obtained by calculating the German-likeness from the human voice with which the robot x is interacting.

Subsequently, in step S33, the condition list L and the remaining condition list LA are initialized. That is, all the conditions included in the condition list L and the remaining condition list LA are deleted. The condition list L describes one or more conditions for searching for an operator. In this embodiment, the condition list L is expressed as a condition (L ₀ + L _iL ) (iL = 1, 2,..., LN). Conditions L ₀ is a condition of the language, in this example, Japanese, one of the English and German are set. The condition L _iL is a condition other than language, and is individually identified by the variable iL.

  In the next step S35, it is determined whether or not the robot x has a voice input. That is, the CPU of the central control device 14 determines whether or not the voice of the conversation partner is detected by the robot x, a voice signal corresponding to the detected voice is transmitted from the robot x, and the voice signal is received. If “NO” in the step S35, that is, if there is no voice input to the robot x, the robot x is instructed to output a voice in a step S37, and the process returns to the step S35. In step S37, for example, a command for causing the robot 12 to output a voice that causes the conversation partner to speak is sent to the robot x, such as "What is it for?" In this way, the CPU of the central control device 14 listens to questions (requests) of the conversation partner and creates the condition list L.

  If “YES” in the step S35, that is, if there is a voice input to the robot x, a Japanese-likeness calculation is performed in a step S41. Here, the Japanese-likeness calculation is an operation for obtaining a ratio of how close the language of the conversation partner of the robot x is to Japanese. Specifically, the CPU of the central control unit 14 recognizes the received voice signal by the DP matching method or the HMM (Hidden Markov Model) method using a Japanese dictionary, and the recognition result and its score (approximation degree). Or a ratio indicating accuracy). In the next step S43, the calculation result is stored in the variable jpn. Here, among the results calculated as described above, the score is stored in the variable jpn as Japanese-likeness.

  In the following step S45, English-likeness calculation is performed. Here, the CPU of the central control device 14 performs speech recognition using the English dictionary in the same manner as in step S41, and obtains a recognition result and its score. In step S47, the calculation result (score) is stored in the variable eng as English-likeness. As shown in FIG. 9, in the next step S49, a German-likeness calculation is performed. Here, the CPU of the central controller 14 performs speech recognition using the German dictionary in the same manner as in step S41, and obtains a recognition result and a score. In the next step S51, the calculation result (score) is stored in the variable deu as German-likeness.

  In the next step S53, it is determined whether or not the variable jpn is larger than the variable eng. That is, here, the CPU of the central controller 14 determines whether the language of the conversation partner of the robot x is close to Japanese or English. If YES here, that is, if variable jpn is larger than variable eng, it is determined that the language of the conversation partner of robot x is closer to Japanese than English, and whether variable jpn is larger than variable deu in step S55. Judge whether or not. That is, here, the CPU of the central controller 14 determines whether the language of the conversation partner of the robot x is close to Japanese or German.

If YES in the step S55, that is, if the variable jpn is greater than the variable deu, dialogue partner of the language of the robot x is determined that the closer to the Japanese than German, in step S57, the condition L ₀ to Japanese Set and proceed to step S65. On the other hand, if NO in step S55, that is, if variable jpn is equal to or smaller than variable deu, it is determined that the language of the conversation partner of robot x is closer to German than Japanese, and in step S61, condition L _{0 is set} to German. Set the word and go to step S65.

  If NO in step S53 described above, that is, if the variable jpn is equal to or smaller than the variable eng, it is determined that the language of the conversation partner of the robot x is closer to English than Japanese, and the variable eng is a variable in step S59. It is determined whether it is larger than deu. Here, the CPU of the central controller 14 determines whether the language of the conversation partner of the robot x is close to English or German.

If YES in the step S59, that is, greater than the variable eng is variable deu, dialogue partner of the language of the robot x is determined that the closer to the English than German, in step S63, to set the conditions L ₀ in English Then, the process proceeds to step S65. On the other hand, if NO in step S59, that is, if the variable eng is equal to or smaller than the variable deu, it is determined that the language of the conversation partner of the robot x is closer to German than English, and the process proceeds to step S61.

Subsequently, in step S65, a condition list creation process (see FIG. 12) described later is executed, an operator search process (see FIG. 13) described later is executed in step S67, and in step S69 shown in FIG. 10, step S67 is executed. It is determined whether or not the operator OP _k (k = 1) retrieved in step 1 satisfies all the conditions in the condition list L. That is, the CPU of the central controller 14 determines whether the language and skill of the searched first operator OP _k matches or includes all the conditions in the condition list L.

If “YES” in the step S69, that is, if the first operator OP _k satisfies all the conditions in the condition list L, the process proceeds to a step S83 shown in FIG. On the other hand, if NO in step S69, that is, if the first operator OP _k satisfies only a part of the conditions in the condition list L, the remaining condition list LA is created in step S71. That is, a list (remaining condition list LA) for the remaining conditions not satisfied by the operator OP _k is created (updated). Here, the remaining condition list LA can be expressed as a remaining condition (LA ₀ + LA _iLA ) (iLA = 1, 2,..., LAN). The remaining condition LA ₀ is a language condition and matches the condition L ₀ . The remaining condition LA _iLA is a condition other than language. However, the variable LNA _indicates the maximum value (maximum number) of the remaining conditions LA _iLA other than the language. Specifically, the remaining condition list LA is created (updated) according to Equations 1 and 2.
[Equation 1]
Residual condition LA ₀ = condition L ₀
[Equation 2]
Residual condition LA _iLA = condition L _iL -GetList (operator OP _k )
However, in Equation 2, the condition (skill) satisfied by the operator OP _k (k = 0, 1,..., J) is read from the operator terminal information table by the “GetList (operator OP _k )” function and included in the condition list L. The remaining condition LA _iLA is determined by deleting from the condition L _iL . For example, when the operator OP ₁ has the skill of “personal computer department” and the condition list L includes “personal computer department” and “iPod” as conditions, if the remaining condition list LA is created according to Equation 2, the condition list From the conditions included in L, the skill “personal computer department” possessed by the operator OP ₁ is deleted, and “iPod” is set as a condition in the remaining condition list LA.

Subsequently, in step S73, the variable k and the variable j are incremented. That is, the second operator OP _k that retrieved, the total number j of the selected operator OP _k are respectively 1 added. In the next step S75, a search process (see FIG. 13) for the operator that best satisfies the remaining condition list LA is executed. Further, in step S77, it is determined whether or not the operator OP _k searched in step S75 satisfies all the conditions in the remaining condition list LA. If YES here, that is, if the operator OP _k searched in step S75 satisfies all the conditions in the remaining condition list LA, the process proceeds to step S83.

Here, if NO in step S77, that is, if the operator OP _k searched in step S75 satisfies a part of the remaining condition list LA, in step S79, among all the conditions in the condition list L, It is determined whether or not the condition of 80% or more is satisfied. For example, when the condition list L includes five conditions, it is determined whether or not four of the conditions are satisfied. If “YES” in the step S79, that is, if 80% or more of the conditions in the condition list L are satisfied, the process proceeds to a step S83. On the other hand, if NO in step S79, that is, of all the conditions of the condition list L, if not satisfied only 80% less than condition, in step S81, the number of the operator OP _k is three or more retrieved It is determined whether or not. That is, it is determined whether or not the variable k or the variable (total number) j is 3 or more. If “NO” in the step 81, if the operator OP _k is less than three, the process returns to the step S71 as it is to update the remaining condition list LA and search for the next operator OP _k .

As described above, even when the number of conditions exceeding a certain ratio among all the conditions in the condition list L is satisfied, or when the number of retrieved operators OP _k exceeds a certain number, the processing described later is performed. This is to avoid inconveniences such as an infinite loop or in most cases an “error” is transmitted to the robot 12. In addition, if the number of selected operators OP _k becomes too large, it becomes troublesome to adjust the remote operation method and order of the robot 12 between the operators OP _k . However, “80%” and “3 people” are merely examples, and the present invention is not limited to these and can be arbitrarily set. Further, the ratio satisfying the condition list L and the number of operators OP _k may be variably set according to the number of conditions included in the condition list L.

As shown in FIG. 11, in step S83, the operation terminal 16 used by the selected operator OP _k is introduced to the robot x. For example, the CPU of the central controller 14 transmits connection information (IP address) of the operation terminal 16 used by the selected operator OP _k to the robot x. Subsequently, in step S85, the on operation terminal 16 used by the operator OP _k selected, it transmits a remote control request for the robot x. For example, the connection information (IP address) of the robot x that transmitted the call request is sent to the operation terminal 16 operated by the selected operator OP _k .

Although not shown, in step S85, the in operator terminal information table, the operator OP _k is selected states of the operation terminal 16 to be used is updated to "BUSY-Rx".

In step S87, it is determined whether or not the variable j is greater than zero. That is, it is determined whether or not there are two or more selected operators OP _k . If “NO” in the step S87, that is, if the variable j is 0 or less, the process returns to the entire process illustrated in FIG. On the other hand, if “YES” in the step S87, that is, if the variable j is 1 or more, the communication between the selected operators OP _k is established in a step S89, and the process returns to the entire process.

For example, in step S89, the CPU of the central control device 14 notifies the connection information (IP address) of the operation terminal 16 used by other operators to the two or more selected operators, and chats. A command to start the software (execute voice chat processing) is transmitted. Accordingly, each operator terminal 16 of the selected operator OP _k is communicably connected to the other operator terminals 16. For example, the operators OP _k exchange information and adjust the remote operation of the robot x by voice chat. be able to. As a result, the remote operation of the robot x can be smoothly executed, and the robot x can communicate with the conversation partner of the robot x smoothly.

In this embodiment, for the sake of simplicity, the operators OP _k communicate with each other by voice chat. However, the chat may be performed by text input. Alternatively, it may be a chat using both voice and text.

FIG. 12 is a flowchart of the condition list creation process in step S65 shown in FIG. As shown in FIG. 12, when starting the condition list creation process, the CPU of the central controller 14 creates the keyword list KL from the voice detected by the robot x in step S101. Briefly, the voice signal corresponding to the voice of the conversation partner detected by the robot x is recognized. Here, a well-known morphological analysis process is performed on the result of speech recognition described above, and a word (keyword) corresponding to a noun is extracted. All extracted keywords are set in the keyword list KL. Here, the extracted keyword is set to the condition list L as a condition L _iL non condition L ₀ (language). For example, "Where is the computer? Later, also looking for iPod." Result of voice recognition in the case of, as a keyword, "PC" and "iPod" is extracted, for example, "PC as a condition L ₁ "is set," iPod "is set as a condition L _2.

  In step S103, the number of keywords in the keyword list KL is set in the variable KLN. The number of keywords set here is the total number of extracted keywords. In the next step S105, variables are initialized. That is, 1 is set in the variable LN for counting the number of conditions included in the condition list L, and 0 is set in the variable iLB for sequentially specifying the extracted keywords.

  Subsequently, in step S107, it is determined whether or not the variable KLN is greater than zero. That is, it is determined whether or not a keyword has been extracted from the voice of the conversation partner. If “NO” in the step S107, that is, if the variable KLN is 0, it is determined that no keyword is extracted from the voice of the conversation partner, and the process directly returns to the operator selection process shown in FIGS. On the other hand, if “YES” in the step S107, that is, if the variable KLN is 1 or more, it is determined that the keyword is extracted from the voice of the conversation partner, and the process proceeds to the step S109.

In step S109, the keyword KL _iLB is set in the condition L _LN . For example, when “PC” is set as the keyword KL ₀ , “PC” is set as the condition L ₁ . Subsequently, in step S111, each of the variable iLB and the variable LN is incremented. In the next step S113, it is determined whether or not the variable iLB and the variable KLN are equal. That is, it is determined whether or not all the keywords included in the keyword list KL are set in the condition list L. If “YES” in the step S113, that is, if the variable iLB and the variable KLN are equal, the process directly returns to the operator selection process. On the other hand, if “NO” in the step S113, that is, if the variable iLB and the variable KLN are not equal, the process returns to the step S109.

FIG. 13 is a flowchart showing the operator search processing in step S67 shown in FIG. 9 and step S75 shown in FIG. As shown in FIG. 13, when the CPU of the central control device 14 starts the operator search process, in step S201, the CPU determines whether or not the remaining condition list LA has been initialized. That is, the CPU of the central control device 14 determines whether an operator that satisfies the condition list L is searched for or an operator that satisfies the remaining condition list LA is searched. Therefore, if YES in step S201, that is, if the remaining condition list LA is initialized, it is determined that the operator is to be searched using the condition list L, and the condition list L is set as the search condition list LB in step S203. Set. However, if NO in step S201, that is, if the remaining condition list LA is not initialized, it is determined that the operator is searched using the remaining condition list LA, and the remaining condition list is set as the search condition list LB in step S205. Set LA. Here, “search condition LB _iLC ” indicates one of the conditions included in the search condition list LB, and “iLC” is a variable for individually specifying each condition included in the set of search condition lists LB. is there. In addition, since the condition list L or the remaining condition list LA is set in the search condition list LB, the total number of conditions included in the search condition list LB is the variable LN corresponding to the total number of conditions included in each list or Variable LAN.

  In step S207, variables are initialized. Specifically, 1 is set to the variable iLC, the variable ka, and the variable id. Here, the variable iLC is a variable for individually specifying the conditions included in the search condition list LB. The variable ka is a variable for individually specifying skills included in the skill list. The variable id is a variable for individually specifying an operator (operation terminal 16).

  In the next step S209, the array match [1-id_max] is initialized. That is, the CPU of the central control device 14 sets 0 in the array match [1] to match [id_max]. Here, the array element number of the array match [1 to id_max] corresponds to the ID of the operator (operation terminal 16) in the operator terminal information table shown in FIG. In the array match [1 to id_max] designated by the operator ID, the number (match number) of the skill that the operator has and the condition included in the condition list L are set (stored).

  In a succeeding step S211, it is determined whether or not the variable id is larger than id_max. That is, it is determined whether or not the variable id exceeds the total number of operators registered in the operator terminal information table. If “NO” in the step S211, that is, if the variable id is equal to or less than id_max, it is determined whether or not the id_state (id) is “IDLE” in a step S213. Here, the function id_state (id) is a function that returns a state described corresponding to the ID specified by the variable id. That is, the CPU of the central control device 14 refers to the operator terminal information table and determines whether or not the operator (operation terminal 16) having the ID specified by the variable id is in a “standby state”.

  If NO in step S213, that is, if id_state (id) is not "IDLE", the variable id is incremented in step S227, the variable ka is initialized, and the process returns to step S211. On the other hand, if “YES” in the step S213, that is, if id_state (id) is “IDLE”, it is determined whether or not the variable ka is larger than id_sn (id) in a step S215.

Here, the function id_sn (id) is a function that returns the number of skills possessed by the operator of the ID specified by the variable id. Therefore, in step S215, it is determined whether or not the number of matches / mismatch between all the skills of the operator with the ID specified by the variable id and all the conditions included in the condition list L has been examined. If “YES” in the step S215, that is, if the variable ka is larger than id_sn (id), the process proceeds to a step S227 as it is. On the other hand, if “NO” in the step S215, that is, if the variable ka is equal to or less than id_sn (id), it is determined whether or not the variable iLC is larger than the variable LBN in a step S217. That is, it is determined whether or not the search condition LB _iLC is larger than the total number (LBN) of the search condition list LB. If “YES” in the step S217, that is, if the variable iLC is larger than the variable LBN, the variable ka is incremented in a step S225, the variable iLC is initialized, and the process proceeds to the step S215. On the other hand, if “NO” in the step S217, that is, if the variable iLC is equal to or less than the variable LBN, it is determined whether or not the search condition LB _iLC matches id_sk (id, ka) in a step S219.

Here, the function id_sk (id, ka) is a function that returns the skill ka possessed by the operator of the ID specified by the variable id. For example, as can be seen with reference to the operator information terminal table shown in FIG. 5, the function id_sk (2, 1) returns “iPod”. If YES in step S219, that is, if the search condition LB _iLC matches id_sk (id, ka), the array match [id] is incremented in step S221, and the process proceeds to step S223. That is, in step S221, for the operator with the ID indicated by the variable id, the number of matches between the skill of the operator and the condition in the condition list L or the remaining condition list LA is counted. On the other hand, if “NO” in the step S219, that is, if the search condition LB _iLC does not match the id_sk (id, ka), the process proceeds to a step S223 as it is. In step S223, the variable iLC is incremented. Accordingly, a process for determining whether or not the next search condition LB _iLC matches the skill returned by the function id_sk (id, ka) is executed.

Also, if YES in step S211, that is, if id is greater than Id_max, set in step S229, in the sequence match [1~id_max], the operator of the maximum value and the user satisfies ID to the operator OP _k Then, the process returns to the operator selection process.

Here, as described above, the user condition is a condition that can be arbitrarily determined by the user of the robot remote control system 10 when searching for an operator. For example, when the user condition is “want to call an operator with abundant experience”, an array in which the largest value among the values included in each of the arrays match [0 to id_max] is set. When there are two or more operators, the operator having the largest value included in the “experience” status value of the operator having an ID corresponding to the number of arrays is the operator searched by the operator OP search process. For example, when the number of matches indicated by the array match [2] and the array match [id_max] is the maximum, the operator whose ID is “2” is understood with reference to the operator terminal information table shown in FIG. From the experience (February) and the experience (December) of the operator whose ID is “id_max”, the operator whose ID is “id_max” satisfies the user condition. This operator (Miyazaki) is set as the operator OP _k .

In another embodiment, when the user condition is “I want to preserve as many operators as possible corresponding to many keywords”, the number of skills among the plurality of operators having the maximum number of matches is The fewest operators are set as operator OP _k .

Furthermore, in another embodiment, when the user condition is “I want to preserve as many operators as possible who can speak multiple languages”, the number of languages among the multiple operators with the largest number of matches. The operator having the smallest number is set as the operator OP _k .

FIG. 14 is a flowchart of the sensor information selection process in step S11 shown in FIG. Referring to FIG. 14, in step S301, variable k is initialized (k = 1). Subsequently, in step S303, it is determined whether or not the variable k is equal to or less than the variable j. That, k-th operator OP _k determines whether at least the total number j of the operator OP _k. If “NO” in the step S303, that is, if the variable k is larger than the variable j, the process proceeds to a step S311. On the other hand, if “YES” in the step S303, that is, if the variable k is equal to or less than the variable j, whether or not the communication speed S between the robot x and the operation terminal 16 operated by the operator OP _k is equal to or higher than the threshold value Sa in a step S305. Judging. That is, if the communication speed S is larger than the threshold value Sa, the audio information is transmitted to the operation terminal 16 operated by the operator OP _k .

If “NO” in the step S305, that is, if the communication speed between the robot x and the operation terminal 16 is less than the threshold value Sa, the process proceeds to a step S309 as it is. On the other hand, if “YES” in the step S305, the voice information is transmitted to the operator OP _k in a step S307, and the process proceeds to the step S309. In this step S307, the CPU of the central control device 14 sends a command for that purpose to the robot x in order to transmit the voice signal corresponding to the voice of the conversation partner of the robot x as voice information to the operation terminal 16 operated by the operator OP _k. Send to. The operator OP _k can easily know the utterance content of the conversation partner by listening to the voice of the voice signal, and thus can talk with the conversation partner. For this reason, it can respond appropriately. In step S309, the variable k is incremented, and the process returns to step S303.

In step S311, the variable k is initialized. Subsequently, in step S313, it is determined whether or not the variable k is equal to or less than the variable j. If “YES” in the step S313, it is determined whether or not the communication speed S between the robot x and the operation terminal 16 operated by the operator OP _k is equal to or higher than the threshold value Sv in a step S315. For example, if the communication speed S is higher than the threshold value Sv, the camera information is transmitted to the operation terminal 16 operated by the operator OP _k .

If “NO” in the step S315, that is, if the communication speed between the robot x and the operation terminal 16 is less than the threshold value Sv, the variable k is incremented in a step S319, and the process returns to the step S313. On the other hand, if “YES” in the step S315, that is, if the communication speed between the robot x and the operation terminal 16 is equal to or higher than the threshold value Sv, the camera information is transmitted to the operator OP _k in a step S317, and the process proceeds to the step S319. In this step S317, the CPU of the central control device 14 transmits, as video information, a video signal corresponding to the video taken by the omnidirectional camera 34 or the eye camera 58 of the robot x to the operation terminal 16 operated by the operator OP _k. In order to do so, a command for that purpose is transmitted to the robot x. The operator OP _k can know the situation around the robot x, the facial expression of the conversation partner, gesture gestures, and the like by viewing the video. Therefore, for example, it is possible to respond by looking at the facial expression of the other party, so it is possible to respond more appropriately.

In step S321, the variable k is initialized. In step S323, it is determined whether the variable k is equal to or less than the variable j. If “NO” in the step S323, the process returns to the entire process illustrated in FIG. On the other hand, if “YES” in the step S323, it is determined whether or not the communication speed S between the robot x and the operation terminal 16 operated by the operator OP _k is equal to or higher than the threshold value Ss in a step S325. For example, if the communication speed S is larger than the threshold value Ss, the distance information is transmitted to the operation terminal 16 operated by the operator OP _k .

If “NO” in the step S325, that is, if the communication speed S between the robot x and the operation terminal 16 is less than the threshold Ss, the process proceeds to a step S329 as it is, the variable k is incremented, and the process returns to the step S323. On the other hand, if “YES” in the step S325, that is, if the communication speed S between the robot x and the operation terminal 16 is equal to or higher than the threshold value Ss, the distance information is transmitted to the operator OP _k in a step S327, and the process proceeds to the step S329. In this step S327, the CPU of the central controller 14 determines the distance data on the distance between the robot x and the conversation partner detected by the infrared distance sensor 28 of the robot x and the conversation partner and the robot 12 detected by the contact sensor 46. Is transmitted to the operation terminal 16 operated by the operator OP _k as distance information, a command for that is transmitted to the robot x. By viewing this distance information, the operator OP _k can recognize the state (distance to the conversation partner) between the conversation partner and the robot 12. Thereby, for example, skinship can be achieved through the robot x.

Note that the values of the threshold value Sa, the threshold value Sv, and the threshold value Ss can be arbitrarily changed by the user. For example, since the communication speed of the analog telephone is about 54 kbps, the threshold value Sa may be set to 54 kbps. Alternatively, by setting the threshold value Sa to 0, the robot 12 may always transmit voice information to the operation terminal 16 used by the operator OP _k .

  However, the communication speed S between the robot 12 and the operation terminal 16 is based on the maximum communication speed (best effort) of each communication line between the robot 12 and the operation terminal 16 that has transmitted the call request stored in the central controller 14. Can be decided. Further, the communication speed S may be determined from the time taken to transmit the reference data by transmitting the reference data from the central controller 14 to each of the robot 12 and the operation terminal 16.

  In this embodiment, information to be transmitted from the robot 12 to the operation terminal 16 is determined depending on whether or not the communication speed S exceeds a certain threshold. However, when the communication environment is not considered, Any one or two or more of audio information, camera information, and distance information may be transmitted.

  Furthermore, in this embodiment, the information to be transmitted is determined according to the communication speed S, but the information to be transmitted may be determined according to the environment where the robot 12 is installed.

  Here, FIG. 15 shows a display example of the remote operation screen 210 displayed on the display device of the operation terminal 16 when the camera information and the sensor information are restricted. As can be seen from FIG. 15, the display content of the operation panel 216 is the same as that of the remote operation screen 210 shown in FIG. Since the display unit of the robot camera image 212 is limited in camera information, a character string (text) “camera information is being restricted” is displayed. Similar to the remote operation screen 210 shown in FIG. 6, the display unit of the robot information 214 displays the position where the robot is installed, the name of the robot, the map of the installed location, and the information of the person who is interacting with the robot. Is done. However, as shown in the lower part of the robot information 214, the voice information is displayed as “communication” as in the remote operation screen 210 shown in FIG. 6, but the distance information is the text “restricted”. Is displayed.

  FIG. 16 is a flowchart showing the overall processing of the CPU 60 of the robot 12 (robot x). As shown in FIG. 16, when starting the entire process, the CPU 60 of the robot 12 (robot x) executes an operator call condition determination process (see FIG. 17) described later in step S401, and in step S403, calls an operator call flag. It is determined whether or not is on. Although not shown in the figure, the operator call flag is composed of a 1-bit register and is provided in the memory 64 (RAM). This call flag is turned on / off by executing an operator call condition determination process. Specifically, if the operator call condition is satisfied, the call flag is turned on. Conversely, if the operator call condition is not satisfied, the call flag is turned off. When the call flag is turned on, the data value “1” is set in the register. Conversely, when the call flag is turned off, the data value “0” is set in the register.

  If “NO” in the step S403, that is, if the calling flag is turned off, an independent action process is executed in a step S409. In the autonomous behavior process, communication behavior is executed with human beings as needed by independent control. For example, when a person is faced, a greeting is given to the person, and when a person is asked about the way, directions are given. On the other hand, if “YES” in the step S403, that is, if the calling flag is turned on, a remote operation handling process (see FIG. 18) described later is executed in a step S405.

  In step S407, it is determined whether there is a stop command. For example, a stop command (stop command) is transmitted from the operation terminal 16 to the robot 12 via the network 100 according to the operation of the operator. However, a stop button may be provided in the environment where the robot 12 or the robot 12 is arranged, and a stop command may be input to the robot 12 by operating the button. If “YES” in the step S407, that is, if there is a stop command, the entire processing is ended as it is. On the other hand, if NO in step S407, that is, if there is no stop command, the process returns to step S401.

  FIG. 17 is a flowchart of the operator call condition determination process in step S401 shown in FIG. As shown in FIG. 17, when starting the operator calling condition determination process, the CPU 60 of the robot 12 turns off the calling condition flag in step S501, and detects surrounding information in step S503. In other words, the outputs of sensors (infrared distance sensor 28, omnidirectional camera 34, microphone 54, eye camera 58, contact sensor 46, and wireless tag reader 86) necessary for determining the operator calling condition are detected, and the surroundings and their own conditions Recognize (state).

  In step S505, it is determined whether a specific sentence or word has been detected. If YES in step S505, that is, if a specific sentence or word is detected, it is determined that the operator calling condition is satisfied, the calling condition flag is turned on in step S519, and the entire robot 12 shown in FIG. Return to processing. For example, as a specific sentence or word, a sentence or word that means that the robot 12 cannot understand the intention of the other party (the conversation partner), for example, “No”, “Call the person in charge”, “That ’s not right But "and" I don't know "can be considered.

  On the other hand, if “NO” in the step S505, that is, if a specific sentence or word is not detected, it is determined whether or not a specific facial expression is acquired in a step S507. For example, the specific facial expression may be that anger has been detected by recognizing a human facial expression, or that a specific person has been detected by the face specifying function. If “YES” in the step S507, that is, if a specific facial expression is acquired, it is determined that the operator calling condition is satisfied, and the process proceeds to a step S519.

  If “NO” in the step S507, that is, if a specific facial expression is not acquired, it is determined whether or not a specific human is detected in a step S509. For example, as a method for detecting a specific person, a wireless tag having a specific ID may be detected (for example, a person registered as a VIP appears). If “YES” in the step S509, that is, if a specific person is detected, it is determined that the operator calling condition is satisfied, and the process proceeds to a step S519. On the other hand, if NO in step S509, that is, if a specific person has not been detected, it is determined in step S511 whether or not a long-time conversation with the person is present. For example, it may be determined as YES when the distance acquired by the infrared distance sensor 28 is within 1 m for 5 minutes or longer.

  If “YES” in the step S511, that is, if it is a situation in which a conversation with a human is performed for a long time, it is determined that the operator calling condition is satisfied, and the process proceeds to a step S519. On the other hand, if “NO” in the step S511, that is, if it is not a situation where the person is talking for a long time, in a step S513, it is determined whether or not the going hand is blocked. If “YES” in the step S513, that is, if the hand is closed, it is determined that the operator calling condition is satisfied, and the process proceeds to the step S519. On the other hand, if “NO” in the step S513, that is, if the moving hand is not blocked, it is determined whether or not there are many people around in a step S515. Here, the state where there are many people in the surroundings may be detected based on the output of the robot 12 or other sensors provided in the surrounding environment.

  If “YES” in the step S515, that is, if there are many people around, it is determined that the operator calling condition is satisfied, and the process proceeds to the step S519. On the other hand, if “NO” in the step S515, that is, if there are not many people around, it is determined whether or not the operator call button has been pressed in a step S517. The operator call button may be provided in the robot 12 or the surrounding environment. If “YES” in the step S517, that is, it is determined that the operator calling condition determination is satisfied, and the process proceeds to the step S519. On the other hand, if NO in step S517, that is, if the operator call button has not been pressed, the process returns to the entire process of the robot 12 shown in FIG.

  FIG. 18 is a flowchart of the remote operation handling process of step S405 shown in FIG. As shown in FIG. 18, when starting the remote operation handling process, the CPU 60 of the robot 12 transmits an operator call request to the central control device 14 and waits for a response to the request in step S601. Subsequently, in step S603, it is determined whether or not the operation terminal 16 has been introduced from the central controller 14. For example, it is determined whether a call response including an address (IP address) of the operation terminal 16 has been received or an error has been received. If “NO” in the step S603, that is, if an error is received from the central controller 14, an error process is executed in a step S613, and the process returns to the entire process of the robot 12 illustrated in FIG.

  For example, in the error process, there is no operation terminal 16 (operator) that can be used, and control by remote operation by the operator cannot be expected, so the robot 12 performs the error handling process autonomously. For example, when a specific sentence or word is detected and the robot 12 cannot handle it, “Please wait for a while because there is no available operator.” "There is no operator who can respond." As a result, the robot 12 can inform the conversation partner that the robot 12 is in an incapable state, and can draw a different response from the conversation partner and change to a state that can be handled by independent control. Be expected. Also, in situations where VIPs come or are surrounded by many people, independent control may be continued without specifically presenting utterances or gestures for handling errors. However, the call request may be retransmitted after a predetermined time has elapsed since the error was received.

  When it is determined in step S613 that the waiting time for receiving a response to the call request has expired, for example, the communication partner is notified that communication is difficult or the call is retried, and then the entire process is performed. Return.

  If YES in step S603, that is, if the operation terminal 16 is introduced from the central controller 14, the process proceeds to step S605. As described above, the introduction from the central control device 14 is performed when the central control device 14 transmits the identification information (IP address) of the operation terminal 16. Here, when the operation terminal 16 is introduced, the process shifts to the remote operation support mode, and the processes of subsequent steps S605 to S611 are repeatedly executed after a predetermined time. In this remote operation compatible mode, the robot 12 performs an operation according to the operation command when there is a remote operation, but autonomously controls when there is no remote operation.

  In step S605, sensor information (voice information, camera information, sensor information, etc.) is transmitted to the introduced operation terminal 16. That is, the voice information acquired by the microphone 54 (speech voice of the conversation partner), the camera information captured by the eye camera 58, the distance to the obstacle (dialog partner) by the infrared distance sensor 28, and the like are transmitted to the operation terminal 16. The

  Subsequently, in step S607, it is determined whether or not an operation command (including transmission of audio information) for controlling its own operation has been received from the operation terminal 16. If “NO” in the step S607, that is, if an operation command is not received, the process returns to the step S605.

  On the other hand, if “YES” in the step S607, that is, if an operation command is received from the operation terminal 16, it is determined whether or not a remote operation end request is made in a step S609. That is, it is determined whether or not the operation command received in step S607 is a remote operation end command.

  If “NO” in the step S609, that is, if an operation command instructing a communication action is received, an operation corresponding to the operation command is executed in a step S611, and the process returns to the step S605. By the processing in step S611, as described above, the operation of the robot 12 is controlled based on the remote operation. For example, a conversation between an operator who uses the operation terminal using the speaker 52 and the microphone 54 and a conversation partner, body movements such as bowing and pointing, and movements such as advancing and turning are executed.

  When a communication action is being executed by autonomous control when an operation command is received, the operation corresponding to the operation command may be executed after waiting for the action being executed to end. Alternatively, the action corresponding to the operation command may be executed after the action being executed is stopped.

  On the other hand, if “YES” in the step S609, the process returns to the entire process of the robot 12 illustrated in FIG. That is, if YES is determined in step S609 or if the process in step S613 is completed, the operation mode of the robot 12 returns from the remote operation support mode to the autonomous control mode.

  FIG. 19 is a flowchart of the remote operation processing of the CPU of the operation terminal 16. As shown in FIG. 19, when starting the remote operation process, the CPU of the operation terminal 16 determines whether or not there is a remote operation request from the central control device 14 in step S701. Specifically, it is determined whether a remote operation request including the IP address of the robot 12 transmitted from the central control device 14 has been received.

  If “NO” in the step S701, that is, if there is no remote operation request from the central control device 14, the process returns to the same step S701 as it is. However, the process of step S701 is executed every certain time (for example, 30 seconds). On the other hand, if “YES” in the step S701, that is, if there is a remote operation request from the central control device 14, the remote operation screen 210 as shown in FIG. 6 or FIG. 15 is displayed on the display device in a step S703. The

  Note that at the beginning of displaying the remote operation screen 210, the camera image of the robot 12 that called the operation terminal 16 and the information related to the robot 12 are not acquired, and therefore the content of the robot camera image 212 is not displayed. Subsequently, in step S705, it is determined whether or not there is an instruction to perform voice chat processing. If “YES” in the step S705, that is, if the execution instruction of the voice chat process is instructed from the central control device 14, the voice chat process is executed in a step S707. That is, the voice chat process is executed in accordance with the instruction from the central controller 14. If the process of step S707 ends, the process proceeds to step S709. Similarly, even if NO in step S705, the process proceeds to step S709. Further, when the process of step S705 or step S707 is completed, the subsequent processes of steps S709 to S717 are repeatedly executed at regular intervals.

  In step S709, the robot information is received and updated. That is, the robot information is received from the central controller 14 or the robot 12 and the display of the robot information 214 on the remote operation screen 210 is updated. When the robot information 214 is displayed for the first time, the position, name, dialogue partner information, etc. of the robot 12 are acquired from the central controller 14. Further, the position information of the robot 12 and the opponent for map display is acquired from the robot 12.

  Subsequently, in step S711, reception and update of sensor information (voice information, camera information, distance information, etc.) is performed. That is, the audio information is received from the robot 12 and the audio information is output from the speaker of the operation terminal 16. Also, the camera information is received from the robot 12 and the display of the robot camera image 212 on the remote operation screen is updated. Furthermore, distance information is received from the robot 12, and the distance to the obstacle (the conversation partner) is displayed.

  Depending on the communication speed S between the robot 12 and the operation terminal 16, there is sensor information that is not transmitted from the robot 12, and in this case, display of some or all sensor information is limited.

  In step S713, it is determined whether an operation command has been input. For example, based on input data from the input device and button arrangement data on the operation panel 216, it is determined whether or not an operation command button has been selected by the operator. If “NO” in the step S713, that is, if there is no input of an operation command, the process returns to the step S709 as it is.

  On the other hand, if “YES” in the step S713, that is, if an operation command is input, the selected operation command is specified in a step S715, and the operation command is transmitted to the robot 12.

  In step S717, it is determined whether or not there has been a remote operation termination request. That is, it is determined whether or not the operation command input in step S713 is a command (end command) for instructing the end of the remote operation. If “NO” in the step S713, that is, if an end command is not input, the process returns to the step S709 as it is. On the other hand, if YES in step S717, that is, if an end command is input by the operator, the remote operation processing is ended.

  According to this embodiment, when there is a call request from the robot, one or more operators having skills that satisfy the request of the robot's conversation partner are selected, and the selected one or more operators remotely operate the robot. In addition, it is possible to communicate with the conversation partner through the robot. Therefore, it is possible to appropriately respond to the request of the conversation partner.

  In this embodiment, the infrared distance sensor 28 is used. However, instead of the infrared distance sensor 28, the distance between the robot 12 and the conversation partner may be measured using an ultrasonic distance sensor, a millimeter wave radar, or the like. Good. In the operator selection process, “likeness calculation” of three languages is performed, but the number of languages may be increased as necessary.

  In addition, the call flag of the operator of the robot 12 is detected by a sensor (for example, a skin sensor) that detects contact on the body surface, and detects that the robot 12 itself has been beaten. Good. Further, when transmitting a call request, the robot 12 may further temporarily store robot sensor information (such as voice information, camera information, and distance information).

FIG. 1 is an illustrative view showing an outline of a robot remote control system according to an embodiment of the present invention. FIG. 2 is an illustrative view showing the appearance of the robot shown in FIG. 1 from the front. FIG. 3 is a block diagram showing an electrical configuration of the robot shown in FIG. FIG. 4 is an illustrative view showing one example of a robot information table stored in the database of the central controller shown in FIG. FIG. 5 is an illustrative view showing one example of an operator terminal information table stored in the database of the central controller shown in FIG. 6 is an illustrative view showing one example of a remote operation screen displayed on the display device of the operation terminal shown in FIG. FIG. 7 is a flowchart showing the entire processing of the CPU of the central control measure shown in FIG. FIG. 8 is a flowchart showing a part of the CPU operator selection process of the central control measure shown in FIG. FIG. 9 is another part of the CPU operator selection process of the central control measure shown in FIG. 1, and is a flowchart subsequent to FIG. FIG. 10 is another part of the CPU operator selection process of the central control measure shown in FIG. 1, and is a flowchart subsequent to FIG. 11 is still another part of the CPU operator selection process of the central control measure shown in FIG. 1, and is a flowchart subsequent to FIG. FIG. 12 is a flowchart showing a condition list creation process of the CPU of the central control measure shown in FIG. FIG. 13 is a flowchart showing the search process of the CPU operator of the central control measure shown in FIG. FIG. 14 is a flowchart showing sensor information selection processing of the CPU of the central control measure shown in FIG. FIG. 15 is an illustrative view showing another example of a remote operation screen displayed on the display device of the operation terminal shown in FIG. FIG. 16 is a flowchart showing the overall processing of the CPU of the robot shown in FIG. FIG. 17 is a flowchart showing operator call condition determination processing of the CPU of the robot shown in FIG. FIG. 18 is a flowchart showing a remote operation handling process of the CPU of the robot shown in FIG. FIG. 19 is a flowchart showing remote operation processing of the CPU of the operation terminal shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Robot remote control system 12 ... Communication robot 14 ... Central controller 16 ... Operation terminal 28 ... Ultrasonic distance sensor 34 ... Omnidirectional camera 46 ... Contact sensor 52 ... Speaker 54 ... Microphone 58 ... Eye camera 60 ... CPU
62 ... Bus 64 ... Memory 66 ... Motor control board 68 ... Sensor input / output board 70 ... Audio input / output board 82 ... Communication LAN board 84 ... Wireless communication device 86 ... Wireless tag reader 100 ... Network

Claims (6)

A robot that performs communication with humans through at least one of body movement and voice by remote control or autonomous control, a plurality of operation terminals that perform remote control of the robot via a network, and the network via the network In a robot remote control system comprising a central controller that mediates between a robot and the operation terminal,
The robot is
A determination means for determining whether or not it is difficult to execute a communication action with the human by autonomous control; and when the determination result of the determination means is affirmative, the central control apparatus is notified of a request for remote operation A notification means,
The central controller is
A request notification receiving means for receiving a notification of a remote operation request from the notification means;
Condition list creating means for creating a condition list for conditions to be satisfied by an operator who performs remote operation of the robot in response to the request notification receiving means receiving a notification of a remote operation request;
Terminal information storage means for storing terminal information including operator attribute information for each of the plurality of operation terminals;
Comparison means for comparing the condition list created by the condition creation means with the terminal information stored in the terminal information storage means;
Selection means for selecting one or a plurality of operation terminals having terminal information satisfying all or a part of the conditions of the condition list as operation terminals for performing remote operation of the robot, based on a comparison result of the comparison means; and A robot remote control system comprising connection control means for connecting a robot and one or a plurality of operation terminals selected by the selection means via the network so as to communicate with each other. The robot is
Voice detection means for detecting a voice signal corresponding to the human voice; and transmission means for transmitting the voice signal detected by the voice detection means to at least the central control unit,
The central controller is
Audio receiving means for receiving an audio signal from the transmitting means;
Voice recognition means for recognizing the human voice based on a voice signal received by the voice reception means; and language specification means for specifying a language spoken by the human based on a recognition result of the voice recognition means. ,
The robot remote control system according to claim 1, wherein the condition list includes language information as a condition. When the selection means satisfies a part of the conditions in the condition list by the terminal information of one of the operation terminals, the selection unit includes one or a plurality of other information including terminal information that satisfies the remaining conditions excluding the part of the condition Including other terminal selection means for selecting the operation terminal,
The robot remote operation system according to claim 2, wherein the other terminal selection unit continues to select the other operation terminal until a condition exceeding a certain ratio with respect to all the conditions in the condition list is satisfied. The central controller is
A terminal number determination means for determining whether or not the number of operation terminals selected by the selection means is a certain number or more;
The robot remote operation system according to claim 3, wherein the other terminal selection unit continues to select the other operation terminal until the terminal number determination unit determines that the number is equal to or greater than a certain number. When the selection means satisfies a part of the condition list by the terminal information of one operation terminal, the selection unit includes one or a plurality of other terminals having terminal information that satisfies the remaining conditions excluding the part of the condition Other terminal operation means for selecting the operation terminal of
The central controller is
A terminal number determination means for determining whether or not the number of operation terminals selected by the selection means is a certain number or more;
The robot remote operation system according to claim 2, wherein the other terminal selection unit continues to select the other operation terminal until the terminal number determination unit determines that the number is equal to or greater than a certain number. The central controller is
The transmission information control means for controlling the information of the robot transmitted from the robot to the one or the plurality of operation terminals according to the communication speed between the robot and the one or the plurality of operation terminals. The robot remote control system described in Crab.

Images