WO2024190308A1 - 情報処理装置、遠隔操作制御方法、遠隔操作制御システム、及び、自律移動体 - Google Patents

情報処理装置、遠隔操作制御方法、遠隔操作制御システム、及び、自律移動体 Download PDF

Info

Publication number
WO2024190308A1
WO2024190308A1 PCT/JP2024/005917 JP2024005917W WO2024190308A1 WO 2024190308 A1 WO2024190308 A1 WO 2024190308A1 JP 2024005917 W JP2024005917 W JP 2024005917W WO 2024190308 A1 WO2024190308 A1 WO 2024190308A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
autonomous
remote
unit
operator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2024/005917
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
慶直 袖山
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to EP24770421.6A priority Critical patent/EP4679858A4/en
Priority to JP2025506628A priority patent/JPWO2024190308A1/ja
Priority to CN202480016157.5A priority patent/CN120958841A/zh
Publication of WO2024190308A1 publication Critical patent/WO2024190308A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/20—Control system inputs
    • G05D1/22—Command input arrangements
    • G05D1/221—Remote-control arrangements
    • G05D1/227—Handing over between remote control and on-board control; Handing over between remote control arrangements
    • G05D1/2279—Handing over between remote control and on-board control; Handing over between remote control arrangements involving allocation of control between two or more remote operators, e.g. tele-assistance
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/0084—Program-controlled manipulators comprising a plurality of manipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1679—Program controls characterised by the tasks executed
    • B25J9/1689—Teleoperation
    • G—PHYSICS
    • G08—SIGNALLING
    • G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
    • G08C17/00—Arrangements for transmitting signals characterised by the use of a wireless electrical link
    • G08C17/02—Arrangements for transmitting signals characterised by the use of a wireless electrical link using a radio link
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40264—Human like, type robot arm
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2105/00—Specific applications of the controlled vehicles
    • G05D2105/30—Specific applications of the controlled vehicles for social or care-giving applications
    • G05D2105/31—Specific applications of the controlled vehicles for social or care-giving applications for attending to humans or animals, e.g. in health care environments
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2107/00—Specific environments of the controlled vehicles
    • G05D2107/60—Open buildings, e.g. offices, hospitals, shopping areas or universities
    • G05D2107/65—Hospitals
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D2109/00—Types of controlled vehicles
    • G05D2109/10—Land vehicles

Definitions

  • This technology relates to an information processing device, a remote operation control method, a remote operation control system, and an autonomous moving body, and in particular to an information processing device, a remote operation control method, a remote operation control system, and an autonomous moving body that are suitable for use when remotely controlling an autonomous moving body.
  • Patent Document 1 In recent years, research and development has been conducted into systems for remotely controlling robots (see, for example, Patent Document 1).
  • Patent Document 1 requires one remote operator for each robot, which is expected to increase the burden on the remote operator and reduce work efficiency.
  • This technology was developed in light of these circumstances, and makes it possible to efficiently remotely control autonomous moving objects such as robots.
  • the information processing device includes a matching unit that matches non-autonomous tasks, which are tasks that are not performed autonomously by the autonomous mobile body, among tasks obtained by breaking down the behavior of the autonomous mobile body, with a remote operator that remotely operates the non-autonomous tasks, and a remote control unit that controls the remote operation of the non-autonomous tasks by the remote operator.
  • non-autonomous tasks which are tasks that are not performed autonomously by the autonomous mobile body, among tasks obtained by breaking down the behavior of the autonomous mobile body, with a remote operator that remotely operates the non-autonomous tasks, and a remote control unit that controls the remote operation of the non-autonomous tasks by the remote operator.
  • an information processing device matches non-autonomous tasks, which are tasks that are decomposed from the behavior of an autonomous mobile body and are not performed autonomously by the autonomous mobile body, with a remote operator who remotely operates the non-autonomous tasks, and controls the remote operation of the non-autonomous tasks by the remote operator.
  • the autonomous mobile body includes an action planning unit that breaks down an action to be performed into a plurality of tasks, and an action control unit that controls the autonomous execution of the autonomous tasks that are executed autonomously among the tasks, and the remote execution of the non-autonomous tasks that are not executed autonomously among the tasks.
  • non-autonomous tasks which are tasks that the autonomous mobile body does not perform autonomously, are matched with a remote operator who remotely operates the non-autonomous tasks, and the remote operation of the non-autonomous tasks by the remote operator is controlled.
  • non-autonomous tasks which are tasks that the autonomous mobile body does not perform autonomously, are matched with a remote operator who remotely operates the non-autonomous tasks, and the remote operation of the non-autonomous tasks by the remote operator is controlled.
  • the behavior to be performed is decomposed into multiple tasks, and the autonomous execution of the autonomous tasks that are executed autonomously among the tasks, and the remote execution of the non-autonomous tasks that are not executed autonomously among the tasks are controlled.
  • FIG. 1 is a diagram for explaining an overview of the present technology.
  • 1 is a block diagram showing a configuration example of a remote operation control system to which the present technology is applied.
  • FIG. 2 is a block diagram showing an example of a functional configuration of a robot.
  • FIG. 2 is a block diagram showing an example of the external configuration of a robot.
  • FIG. 2 is a block diagram showing an example of a functional configuration of a server.
  • 13 is a flowchart for explaining a process of a robot.
  • 11 is a flowchart illustrating details of a task execution control process.
  • 13 is a flowchart illustrating a process of a server.
  • 11 is a diagram for explaining an example in which a robot performs an action of measuring a vital value of a care recipient.
  • FIG. 1 is a diagram showing a configuration example of a remote operation control system to which the present technology is applied.
  • FIG. 2 is a block diagram showing an example of a functional configuration of a robot.
  • FIG. 11 is a diagram for explaining an example in which a robot performs an action of measuring a vital value of a care recipient.
  • FIG. 11 is a diagram for explaining an example in which a robot performs an action of measuring a vital value of a care recipient.
  • FIG. 11 is a diagram for explaining an example in which a robot performs an action of measuring a vital value of a care recipient.
  • FIG. 13 is a diagram showing a calculation formula for the operational effect of the remote operation control system.
  • FIG. 13 is a diagram illustrating an example of calculation of the operational effect of the remote operation control system.
  • FIG. 1 is a block diagram illustrating an example of the configuration of a computer.
  • this technology breaks down the robot's actions into multiple micro-level tasks (hereafter referred to as microtasks), and the robot executes the tasks autonomously to the extent possible, while tasks that are difficult to execute autonomously are executed remotely.
  • microtasks micro-level tasks
  • a task that the robot executes autonomously will be referred to as an autonomous task, and a task that the robot does not execute autonomously will be referred to as a non-autonomous task.
  • the robot also has a module (hereinafter referred to as a task execution module) that is specialized in one skill (corresponding to one skill) for each type of skill, and each task execution module independently executes tasks related to each type of skill. This allows a remote operator to have the robot execute tasks related to the skills that he or she can handle by remotely operating the task execution module specialized in that skill.
  • a task execution module that is specialized in one skill (corresponding to one skill) for each type of skill, and each task execution module independently executes tasks related to each type of skill.
  • the robot has five types of modules: a behavior module, a dialogue module, an observation module, a work module, and a movement module.
  • a behavior module a dialogue module
  • an observation module a work module
  • a movement module a movement module
  • the dialogue module, observation module, work module, and movement module are skill-specific task execution modules (hereinafter simply referred to as task execution modules).
  • the behavior module is a module that controls the behavior of the entire robot. For example, the behavior module creates a plan for the behavior of the entire robot, breaks down each behavior into tasks, and has the robot execute the planned behavior by having the corresponding task execution module execute each task.
  • the dialogue module is a module that executes dialogue processing for the robot, either autonomously or by remote control.
  • the dialogue module executes processing to converse with people and transmit information, achieve mutual understanding, etc.
  • the observation module is a module that performs observation processing for the robot, either autonomously or by remote control. For example, the observation module performs processing to observe the robot's surroundings and recognize the surrounding conditions, such as surrounding objects (including living things such as people) and the environment.
  • the work module is a module that executes the work processing of the robot autonomously or by remote control.
  • the work module executes work using parts of the robot such as the hands.
  • the work module executes actions such as the manipulation, movement, creation, and processing of various objects, as well as robot gestures, using parts of the robot.
  • the movement module is a module that executes the movement process of the robot autonomously or by remote control.
  • the movement module executes the process of moving the robot using the robot's moving means such as legs or wheels.
  • a dialogue specialist operator who specializes in remote control of "dialogue” an observation specialist operator who specializes in remote control of "observation”
  • a task specialist operator who specializes in remote control of "task” a movement specialist operator who specializes in remote control of "movement” will remotely control the robots using the task execution modules for the skills corresponding to each robot.
  • the dialogue specialist operator, observation specialist operator, work specialist operator, and movement specialist operator will also be referred to as the dialogue operator, observation operator, work operator, and movement operator, respectively.
  • an operator who has multiple types of skills can remotely control multiple skills.
  • a behavior operator capable of remotely controlling the entire robot uses a behavior module to remotely control the behavior of the entire robot, or to remotely control tasks that are inseparable from the context or tasks that involve multiple skills.
  • the behavior operator designs behavior rules for the autonomous behavior control of the robot, and when the behavior module is no longer able to autonomously proceed with the behavioral tasks of the entire robot, the behavior operator uses the behavior module to remotely control the behavior of the entire robot.
  • a behavior operator is an entity that can operate the highest level robot behavior, and must have special authority, for example.
  • behavior operators could be employees of a business that operates a robot service, or employees of a commercial facility that has introduced and operates a robot.
  • behavior operators could be staff at a care facility that has introduced and operates the care robot, staff of a home care service provider that has introduced and operates the care robot, or a family member of a user who receives home care provided by a care robot.
  • the behavior operator can act as an interaction operator, observation operator, work operator, or movement operator, and can remotely control specific skills of the robot using the task execution module.
  • each operator will be able to remotely operate only some of the tasks of each robot.
  • each operator will be able to remotely operate multiple robots in parallel according to the skills they possess.
  • multiple operators will be able to remotely operate one robot together.
  • each robot will be able to receive assistance from an operator only for tasks that are difficult to perform autonomously.
  • each robot will be able to stably execute multitasking that requires hospitality.
  • FIG. 2 shows an example of the configuration of a remote control system 1 to which the present technology is applied.
  • the remote operation control system 1 is a system that controls the remote operation of robots 11-1 to 11-m.
  • the remote operation control system 1 includes robots 11-1 to 11-m, operation terminals 12-1 to 12-n, and a server 13.
  • the robots 11-1 to 11-m, operation terminals 12-1 to 12-n, and server 13 are interconnected via a network 21 and are capable of communicating with each other.
  • Each robot 11 corresponds to the robot described above with reference to FIG. 1.
  • the robot 11 is capable of autonomous behavior and performs tasks that are difficult to perform autonomously according to remote control using the operation terminal 12.
  • the form of the robot 11 is not particularly limited.
  • the robot 11 may be an autonomously mobile robot, or a manipulator that performs work by moving a part such as a hand in a fixed location.
  • the robot may be a human or other biological robot, or a robot that performs a specific task.
  • the robot 11 may be in a form that presents characters created using 3D CG (Computer Graphics) or the like on a display equipped with sensors and speakers.
  • 3D CG Computer Graphics
  • the operation terminal 12 is a terminal (information processing device) used by each remote operator to remotely operate each robot 11.
  • the form of the operation terminal 12 is not particularly limited.
  • a smartphone, a PC (personal computer), a tablet terminal, a game terminal, a dedicated operation terminal, etc. may be used as the operation terminal 12.
  • the operation terminal 12 may not only be a terminal operated by hand, but also an operation terminal operated by a part of the body other than the hand, or a terminal operated non-contact using voice recognition, gaze recognition, etc.
  • the server 13 executes control of the remote operation of each robot by each remote operator. For example, the server 13 matches the non-autonomous tasks of each robot 11 with each remote operator. For example, the server 13 links the matched robot 11 with the remote operator's operation terminal 12 via the network 21, and mediates between the two for processing and data required for remote operation.
  • FIG. 3 shows an example of the functional configuration of the robot 11.
  • the robot 11 includes an information processing unit 101, an input unit 102, a sensing unit 103, and a communication unit 104.
  • the information processing unit 101 includes a behavior module 111 and a group of task execution modules 112.
  • the behavior module 111 is a module that controls the behavior of the entire robot 11.
  • the behavior module 111 includes a behavior planning unit 121, a behavior control unit 122, and a learning unit 123.
  • the behavior planning unit 121 creates a plan for the behavior of the entire robot 11 based on a preset schedule, external commands, and the state of the robot 11's surroundings recognized by the state recognition unit 131 of the behavior module 111.
  • the behavior planning unit 121 also breaks down the behavior of the robot 11 into micro-level tasks (microtasks).
  • the tasks are classified into tasks related to the dialogue processing of the robot 11 (hereafter referred to as dialogue tasks), tasks related to the observation processing of the robot 11 (hereafter referred to as observation tasks), tasks related to the work processing of the robot 11 (hereafter referred to as work tasks), and tasks related to the movement processing of the robot 11 (hereafter referred to as movement tasks).
  • dialogue tasks tasks related to the dialogue processing of the robot 11
  • observation tasks tasks related to the observation processing of the robot 11
  • work tasks tasks related to the work processing of the robot 11
  • movement tasks tasks related to the movement processing of the robot 11
  • a dialogue task is, for example, a task related to processing in which the robot 11 dialogues with a person to transmit information, achieve mutual understanding, etc.
  • the observation task is, for example, a task related to the process of observing the surroundings of the robot 11 and recognizing the surrounding conditions such as surrounding objects (including living things such as people) and the environment.
  • Work tasks are, for example, tasks related to processing using each part of the robot 11 to manipulate, move, create, and process various objects, as well as to execute gestures and other actions of the robot 11.
  • the movement task is, for example, a task related to the process of moving the robot 11 using the legs, wheels, or other moving means of the robot 11.
  • the behavior control unit 122 controls the behavior of the entire robot 11.
  • the behavior control unit 122 includes a state recognition unit 131, an autonomous behavior control unit 132, and a remote behavior control unit 133.
  • the state recognition unit 131 recognizes the state around the robot 11 based on the input data from the input unit 102, the sensing data output from the sensing unit 103, and the observation results by the observation module 141b. For example, the state recognition unit 131 recognizes the inner state of people around the robot 11, such as their emotions and intentions.
  • the autonomous behavior control unit 132 controls the execution of autonomous behavior of the robot 11.
  • the autonomous behavior control unit 132 controls the execution of autonomous tasks of the robot 11 by controlling the task execution module group 112.
  • the remote behavior control unit 133 controls the execution of actions by remote operation of the robot 11.
  • the remote behavior control unit 133 controls the execution of non-autonomous tasks by remote operation of the robot 11 by controlling the task execution module group 112.
  • the remote behavior control unit 133 controls the behavior of the entire robot 11 by remote operation by the above-mentioned behavior operator based on remote operation content information indicating the content of the remote operation transmitted from the server 13.
  • the learning unit 123 learns how to autonomously execute an action based on the execution content of the action of the robot 11 by remote control (e.g., the operation content, the execution result, etc.). Also, the learning unit 123 learns how to autonomously execute an action according to the user's intention based on, for example, the user's intention and reaction recognized by the state recognition unit 131.
  • the behavior module 111 (autonomous behavior control unit 132) can grow through learning based on the user's intentions and reactions, and ultimately the user can become the behavior operator himself. This allows users who have difficulty living an autonomous life, such as people with disabilities or dementia, to become behavior operators and live an autonomous life in symbiosis with the robot 11.
  • the dialogue module 141a is a module that executes dialogue tasks autonomously or by remote control.
  • the dialogue module 141a includes an execution unit 151a and a learning unit 152a.
  • the execution unit 151a includes an autonomous dialogue unit 161a and a remote dialogue unit 162a.
  • the autonomous dialogue unit 161a executes dialogue tasks autonomously.
  • the remote dialogue unit 162a executes dialogue tasks by remote control based on remote control content information that indicates the content of the remote control and is sent from the server 13.
  • the learning unit 152a learns how to autonomously execute a dialogue task based on the execution content of the dialogue task by remote operation (e.g., the operation content, the execution result, etc.).
  • the observation module 141b is a module that executes an observation task autonomously or by remote control.
  • the observation module 141b includes an execution unit 151b and a learning unit 152b.
  • the execution unit 151b includes an autonomous observation unit 161b and a remote observation unit 162b.
  • the autonomous observation unit 161b executes the observation task autonomously.
  • the remote observation unit 162b executes the observation task by remote operation based on the remote operation content information sent from the server 13.
  • the learning unit 152b learns how to autonomously execute an observation task based on the execution content of the observation task by remote operation (e.g., the operation content, the execution result, etc.).
  • the work module 141c is a module that executes work tasks autonomously or by remote control.
  • the work module 141c includes an execution unit 151c and a learning unit 152c.
  • the execution unit 151c includes an autonomous work unit 161c and a remote work unit 162c.
  • the autonomous work unit 161c autonomously executes work tasks.
  • the remote operation unit 162c executes the work task by remote operation based on the remote operation content information sent from the server 13.
  • the learning unit 152c learns how to autonomously execute a work task based on the execution content of the work task by remote operation (e.g., the operation content, the execution result, etc.).
  • the movement module 141d is a module that executes movement tasks autonomously or by remote control.
  • the movement module 141d includes an execution unit 151d and a learning unit 152d.
  • the execution unit 151d includes an autonomous movement unit 161d and a remote movement unit 162d.
  • the autonomous movement unit 161d autonomously executes movement tasks.
  • the remote movement unit 162d executes the movement task by remote operation based on the remote operation content information sent from the server 13.
  • the learning unit 152d learns how to autonomously execute a movement task by remote control based on the execution content of the movement task (e.g., operation content, execution result, etc.).
  • the input unit 102 includes an input device used to input various input data such as commands and data.
  • the input unit 102 supplies the input data to the information processing unit 101.
  • the sensing unit 103 includes various sensors, such as a camera, LiDAR (Light Detection And Ranging), radar, and microphones, that sense the surroundings of the robot 11, and various sensors that sense the state of the robot 11.
  • the sensing unit 103 supplies the sensor data output from each sensor to the information processing unit 101.
  • the communication unit 104 communicates with the operation terminal 12 and the server 13 via the network 21.
  • the communication unit 104 supplies received data to the information processing unit 101 and obtains data to be transmitted from the information processing unit 101.
  • the task execution module 141 when it is not necessary to distinguish between the dialogue module 141a to the movement module 141d, they will be referred to as the task execution module 141.
  • the execution unit 151a of the dialogue module 141a to the execution unit 151d of the movement module 141d they will be simply referred to as the execution unit 151.
  • the learning unit 152a of the dialogue module 141a to the learning unit 152d of the movement module 141d they will be simply referred to as the learning unit 152.
  • the autonomous task execution unit 161 when it is not necessary to distinguish between the autonomous dialogue unit 161a, the autonomous observation unit 161b, the autonomous work unit 161c, and the autonomous movement unit 161d, they will be referred to as the autonomous task execution unit 161.
  • the remote dialogue unit 162a, the remote observation unit 162b, the remote work unit 162c, and the remote movement unit 162d when it is not necessary to distinguish between the remote dialogue unit 162a, the remote observation unit 162b, the remote work unit 162c, and the remote movement unit 162d, they will be referred to as the remote task execution unit 162.
  • Each task execution module 141 is equipped with the necessary hardware and software and is capable of executing tasks independently. Furthermore, the hardware and software equipped in each task execution module 141 can be shared with other task execution modules 141.
  • Fig. 4 shows an example of the external configuration of the robot 11.
  • Fig. 4A is a front view of the robot 11.
  • Fig. 4B is a left side view of the robot 11.
  • Robot 11 is a humanoid mobile manipulator robot that can perform various applications such as various types of care, condition observation, communication, and peripheral work tasks with a quality that is highly acceptable to the person receiving care.
  • the robot 11 has a head 201, a chest 202, and a base 203 that supports the chest 202.
  • the base 203 has, for example, a cart 208 at its bottom that can move in all directions. This allows the robot 11 to move in all directions.
  • the robot 11 has an arm 207L attached to the left and upper part of the chest 202, and an arm 207R attached to the right and upper part of the chest 202.
  • the robot 11 has a movable neck 204 provided between the head 201 and the chest 202 and equipped with a neck joint axis 204C.
  • the robot 11 has a movable shoulder 205L provided between the chest 202 and the arm 207L and equipped with a shoulder joint axis 205LC, and a movable shoulder 205R provided between the chest 202 and the arm 207R and equipped with a shoulder joint axis 205RC.
  • the robot 11 also has a movable waist 206 provided at the bottom of the chest 202 and equipped with a waist joint axis 206C.
  • the head 201 is equipped with an eyeball 221L and an eyeball 221R.
  • the robot 11 gazes at the subject by controlling, for example, the position of the black eye of the eyeball 221L, the position of the black eye of the eyeball 221R, and the axes (roll, pitch, yaw) of the neck 204.
  • eyeball portion 221 when there is no need to distinguish between eyeball portion 221L and eyeball portion 221R, they will simply be referred to as eyeball portion 221.
  • the arm 207L includes an elbow 231L, a wrist 232L, and a hand 233L.
  • the elbow 231L includes a pitch axis.
  • the wrist 232L includes a yaw axis.
  • Arm 207R is configured similarly to arm 207L and includes elbow 231R, wrist 232R, and hand 233R.
  • arm 207 when there is no need to distinguish between arm 207L and arm 207R, they will simply be referred to as arm 207.
  • elbow 231L and elbow 231R they will simply be referred to as elbow 231.
  • wrist 232L and wrist 232R when there is no need to distinguish between wrist 232L and wrist 232R, they will simply be referred to as wrist 232.
  • hand 233L and hand 233R when there is no need to distinguish between hand 233L and hand 233R, they will simply be referred to as hand 233.
  • a head sensor 241 is provided on the upper front part of the head 201.
  • the head sensor 241 includes, for example, a distance image sensor, a microphone, LiDAR, etc.
  • the head sensor 241 is configured so that the sensing direction is substantially the same as the line of sight of the robot 11. For example, the robot 11 can recognize people and faces using the head sensor 241.
  • a chest sensor 242 is provided on the upper front part of the chest 202.
  • the chest sensor 242 includes, for example, a non-contact vital sensor.
  • non-contact vital sensors include a body temperature sensor, a heart rate sensor, and a respiration sensor.
  • the hand 233L is provided with a hand sensor 243L.
  • the hand sensor 243L includes, for example, a contact-type vital sensor.
  • Examples of contact-type vital sensors include a heart rate sensor, a blood pressure sensor, and an oxygen saturation measurement sensor.
  • the hand sensor 243L enables vital sensing by having the subject place or grasp the hand, rather than the robot 11 itself having to touch the hand sensor 243L to the subject. This is an interface that is familiar to people with dementia and is highly acceptable to the subject.
  • Hand 233R is also provided with hand sensor 243R similar to hand sensor 243L.
  • hand sensor 243 when there is no need to distinguish between hand sensor 243L and hand sensor 243R, they will simply be referred to as hand sensor 243.
  • FIG. 5 shows an example of the functional configuration of the server 13.
  • the communication unit 301 communicates with each robot 11 and each operation terminal 12 via the network 21.
  • the communication unit 301 supplies the received data to the information processing unit 302 and obtains data to be transmitted from the information processing unit 302.
  • the remote control unit 311 controls the remote operation of the robot 11 by each remote operator using the operation terminal 12.
  • the remote control unit 311 receives task request information including information about a task for which remote operation is requested from the robot 11 via the network 21 and the communication unit 301.
  • the remote control unit 311 transmits the task request information to the operation terminal 12 of the remote operator selected by the matching unit 312 via the communication unit 301 and the network 21.
  • the remote control unit 311 receives remote operation information including information necessary for remote operation from the robot 11 via the network 21 and the communication unit 301.
  • the remote control unit 311 generates remote operation UI information for providing a user interface for remote operation (hereinafter referred to as a remote operation UI) based on the remote operation information.
  • the remote control unit 311 transmits the remote operation UI information to the operation terminal 12 of the remote operator in charge of remote operation via the communication unit 301 and the network 21.
  • the remote control unit 311 receives remote operation content information from the operation terminal 12 via the network 21 and the communication unit 301, and converts the remote operation content information for the robot 11 to be remotely operated as necessary.
  • the remote control unit 311 transmits the remote operation content information to the robot 11 to be remotely operated via the communication unit 301 and the network 21.
  • the remote control unit 311 receives a task execution result notification signal from the robot 11, notifying the result of the task execution by remote operation, via the network 21 and the communication unit 301, and transmits the signal to the operation terminal 12 of the remote operator in charge of the remote operation.
  • the task execution result notification signal is divided into, for example, a task completion notification signal for notifying the completion of the task, and a task failure notification signal for notifying the failure of the task.
  • the matching unit 312 executes a matching process between the non-autonomous tasks of multiple robots 11 and multiple remote operators. For example, the matching unit 312 selects a remote operator to be requested to remotely operate a task based on the task request information received from the robot 11 and the remote operator database and task database stored in the storage unit 303. The matching unit 312 updates the remote operator database and task database stored in the storage unit 303 as necessary based on the remote operation result notification signal.
  • the storage unit 303 stores various data necessary for processing by the server 13.
  • the storage unit 303 stores a remote operator database and a task database.
  • the remote operator database includes data regarding, for example, each remote operator's attributes, characteristics, schedule, environment, and remote operation history.
  • the attributes of the remote operator include the name, age, gender, nationality, occupation, etc. of the remote operator.
  • Characteristics of a remote operator include, for example, the skills possessed by the remote operator, the skill level, the language used, etc.
  • the remote operator's schedule includes, for example, information regarding the dates when the remote operator is available to remotely operate the robot 11.
  • the remote operator's environment includes, for example, information about the environment in which the remote operator performs remote operation.
  • the remote operator's environment includes the form and performance of the operation terminal 12 used by the remote operator, position coordinates on the Earth, and time, etc.
  • the remote operation history includes, for example, the date and time when the robot was remotely operated in the past, the contents of the task, the execution results, etc.
  • the task database includes data on, for example, the content of each task, its difficulty, and the environment required for remote operation.
  • This process is started, for example, when a trigger occurs that causes the robot 11 to start some kind of action.
  • the robot 11 starts this process when the time arrives to start a specific action according to a preset schedule, or when a command to perform some kind of action is given from outside.
  • step S1 the action planning unit 121 creates an action plan.
  • the action planning unit 121 creates a plan for a series of actions required to achieve a given objective.
  • the method of creating the action plan is not particularly limited.
  • the action planning unit 121 may create the action plan autonomously, based on information provided from the outside, or based on a dialogue with the user.
  • the action planning unit 121 may apply an action plan provided from the outside, or, if multiple action plans are provided, select an action plan from among them.
  • action plans provided from the outside include an action plan designed in advance by the operator of the remote operation control system 1, an action plan created by the user of the robot 11, and an action plan created by the action operator.
  • the behavior planning unit 121 breaks down each behavior into multiple tasks. At this time, each behavior is broken down into multiple tasks as much as possible until each task reaches a micro-level that is independent of the context in which the behavior is executed.
  • step S2 the robot 11 executes task execution control processing.
  • step S51 the autonomous behavior control unit 132 determines whether the next task to be executed can be executed autonomously. If it is determined that the next task to be executed can be executed autonomously, the process proceeds to step S52.
  • step S52 the robot 11 executes the task autonomously.
  • the autonomous behavior control unit 132 instructs the dialogue module 141a to execute the task autonomously.
  • the autonomous dialogue unit 161a of the dialogue module 141a autonomously executes the instructed task.
  • the autonomous behavior control unit 132 instructs the observation module 141b to execute the task autonomously.
  • the autonomous observation unit 161b of the observation module 141b autonomously executes the instructed task.
  • the autonomous behavior control unit 132 instructs the work module 141c to execute the task autonomously.
  • the autonomous work unit 161c of the work module 141c autonomously executes the instructed task.
  • the autonomous behavior control unit 132 instructs the movement module 141d to execute the task autonomously.
  • the autonomous movement unit 161d of the movement module 141d autonomously executes the instructed task.
  • step S53 the autonomous behavior control unit 132 determines whether the task has been completed based on information from the task execution module 141 that instructed the task to be executed. If it is determined that the task has not been completed, the process proceeds to step S54.
  • step S54 the autonomous behavior control unit 132 determines whether the task has failed based on information from the task execution module 141 that instructed the execution of the task. If it is determined that the task has not failed, the process returns to step S52.
  • steps S52 to S54 are repeatedly executed until it is determined in step S53 that the task has been completed, or until it is determined in step S54 that the task has failed.
  • step S54 if it is determined in step S54 that the task has failed, processing proceeds to step S55.
  • the task is deemed to have failed if it is not completed within the target time.
  • the target time is set to a longer time.
  • the target time is set based on the actual time required for similar tasks in the past.
  • the task when performing a task to recognize a subject or the surrounding conditions, if the confidence level (recognition accuracy) is low, the task is determined to have failed.
  • the acceptable range (threshold) of confidence is set, for example, based on the degree to which misrecognition is tolerated. For example, in the case of a task such as face recognition where misrecognition is not tolerated, the threshold is set to a high value (e.g., 99% or more). On the other hand, in the case of a task such as recognizing the content of a conversation where misrecognition is tolerated to a certain extent, the threshold is set to a relatively low value (e.g., 70% or more).
  • the confidence threshold is learned and optimized each time a similar task is performed.
  • Unexpected collisions or contact with the surrounding environment are detected, for example, by the detection values of sensors such as tactile sensors and force sensors, the current values of actuators, sound recognition of contact sounds, the input values of switches, etc.
  • the task when performing a work task, if an error occurs during the work, the task is determined to have failed. For example, if an abnormality occurs in the object that is the subject of the work, the task is determined to have failed.
  • Abnormalities in the object include, for example, the object falling, tipping over, or colliding. Abnormalities in the object are detected, for example, by image recognition, sound recognition of contact sounds, or detection values from sensors such as tactile sensors or force sensors.
  • step S51 if it is determined in step S51 that the next task to be executed cannot be executed autonomously, steps S52 to S54 are skipped and processing proceeds to step S55.
  • step S55 the remote action control unit 133 requests remote operation of the task. Specifically, the remote action control unit 133 generates task request information including information about the task for which remote operation is requested, and transmits the information to the server 13 via the communication unit 104 and the network 21.
  • the server 13 matches the requested task with a remote operator, as described below.
  • step S56 the robot 11 executes the task by remote control.
  • the remote action control unit 113 instructs the interactive module 141a to execute the task by remote control.
  • the remote interactive unit 162a of the interactive module 141a executes the instructed task according to the remote control.
  • the remote action control unit 113 instructs the observation module 141b to execute the task by remote control.
  • the remote observation unit 162b of the observation module 141b executes the instructed task according to the remote control.
  • the remote action control unit 113 instructs the work module 141c to execute the task by remote control.
  • the remote work unit 162c of the work module 141c executes the instructed task according to the remote control.
  • the remote action control unit 113 instructs the movement module 141d to execute the task by remote control.
  • the remote movement unit 162d of the movement module 141d executes the instructed task according to the remote control.
  • the remote task execution unit 162 that executes the task by remote operation (hereinafter referred to as the target remote task execution unit 162) transmits information required for remote operation (information provided for remote operation) to the server 13 as appropriate via the communication unit 104 and the network 21.
  • the target remote task execution unit 162 receives remote operation content information indicating the content of the remote operation from the server 13 via the network 21 and the communication unit 104.
  • the target remote task execution unit 162 executes the task based on the remote operation content information.
  • step S57 similar to the process in step S53, it is determined whether the task is completed. If it is determined that the task is not completed, the process proceeds to step S58.
  • step S58 similar to the process in step S54, it is determined whether the task has failed. If it is determined that the task has not failed, the process returns to step S56.
  • steps S56 to S58 are repeated until it is determined in step S57 that the task has been completed, or until it is determined in step S58 that the task has failed.
  • step S58 determines whether the task has failed. If it is determined in step S58 that the task has failed, processing proceeds to step S59.
  • step S59 the robot 11 notifies the server 13 of the task failure.
  • the remote action control unit 133 generates a task failure notification signal to notify the server 13 of the task failure, and transmits the signal to the server 13 via the communication unit 104 and the network 21.
  • step S60 the remote action control unit 133 determines whether or not to execute the task again. If it is determined that the task is to be executed again, the process returns to step S55.
  • steps S55 to S60 are repeatedly executed until it is determined in step S57 that the task has been completed, or until it is determined in step S60 that the task will not be executed again. In other words, the process of remotely executing the failed task is repeatedly executed.
  • step S60 if it is determined in step S60 that the task will not be executed again, the task execution control process ends. This may occur, for example, if the task fails a predetermined number of times or is not completed within a predetermined time.
  • step S57 if it is determined in step S57 that the task is completed, processing proceeds to step S61.
  • step S61 the robot 11 notifies the server 13 of the completion of the task.
  • the remote action control unit 133 generates a task success notification signal to notify the completion of the task, and transmits it to the server 13 via the communication unit 104 and the network 21.
  • step S62 the robot 11 executes a learning process based on the content of the remote control.
  • the learning unit 152 of the task execution module 141 that executed a task by remote control learns how to autonomously execute the task based on the content of the remote control.
  • step S3 the action plan unit 121 determines whether or not a change to the action plan is necessary. If it is determined that a change to the action plan is not necessary, the process proceeds to step S4.
  • step S4 the action planning unit 121 determines whether the action being performed has been completed based on the result of the processing in step S2. If it is determined that the action being performed has not been completed, the processing returns to step S2.
  • steps S2 to S4 are repeated until it is determined in step S2 that a change to the action plan is necessary, or until it is determined in step S4 that the action being performed has been completed.
  • step S4 determines whether the action being performed has been completed. If it is determined in step S4 that the action being performed has been completed, processing proceeds to step S6.
  • step S3 if it is determined in step S3 that a change to the action plan is necessary, the process proceeds to step S5. This may occur, for example, when the action cannot be carried out as planned due to a task failure, an unexpected event or disturbance, etc.
  • step S5 the action planning unit 121 modifies the action plan. For example, the action planning unit 121 reviews the action plan and modifies the plan of a series of actions required to achieve a given objective. The action planning unit 121 also breaks down each modified action into multiple tasks.
  • the action planning unit 121 may decide to cancel the action.
  • step S6 the action planning unit 121 determines whether there is a next action to be executed. If it is determined that there is a next action to be executed, the process returns to step S2.
  • step S6 the processes in steps S2 to S6 are repeatedly executed until it is determined that there is no next action to be executed. In this way, actions are executed according to the action plan.
  • step S6 if it is determined in step S6 that there is no next action to be performed, the processing of the robot 11 ends.
  • step S101 the remote control unit 311 determines whether or not remote operation of the task has been requested.
  • the determination process in step S101 is repeatedly executed until it is determined that remote operation of the task has been requested.
  • the remote control unit 311 receives task request information transmitted from the robot 11 in the process of step S55 in FIG. 7 via the network 21 and the communication unit 301, it determines that remote operation of the task has been requested, and the process proceeds to step S102.
  • step S102 the matching unit 312 performs matching of remote operators. That is, the matching unit 312 performs matching processing to determine the remote operator who will be in charge of the requested task.
  • the method of matching is not particularly limited.
  • the matching unit 312 selects one remote operator who is currently capable of performing remote operations from among remote operators who have the skills to execute the requested task based on the remote operator database stored in the storage unit 303.
  • the matching unit 312 recruits remote operators who have the skills to perform the requested task, and selects one from the remote operators who apply.
  • the matching unit 312 selects that many remote operators.
  • the matching unit 312 transmits the task request information to the operation terminal 12 of the selected remote operator via the communication unit 301 and the network 21.
  • the server 13 provides a user interface for remote operation.
  • the remote control unit 311 receives remote operation information from the robot 11 via the network 21 and the communication unit 301.
  • the remote control unit 311 generates remote operation UI information for providing a remote operation UI based on the remote operation information.
  • the remote control unit 311 transmits the remote operation information to the operation terminal 12 of the remote operator via the communication unit 301 and the network 21.
  • the operation terminal 12 presents the remote operation UI to the remote operator based on the remote operation UI information.
  • the remote control UI is desirable for the remote control UI to be one that is highly motivating and allows operators with different skills around the world to perform remote control simply and efficiently.
  • the remote control UI may include footage of the environment in which the robot 11 actually performs the task.
  • the remote control UI may include an image of a virtual space that uses CG (Computer Graphics) or the like to simulate the environment in which the robot 11 performs a task.
  • CG Computer Graphics
  • the remote operator may exist as an avatar or character and the remote control UI may be provided within a virtual space (metaverse space) in which the remote operator is active.
  • the remote control UI may include images dedicated to performing a task, such as game images for performing a task, regardless of the environment in which the robot 11 performs the task.
  • a remote control UI may be provided in an abstract game space or the above-mentioned virtual space.
  • the remote operator who is the game player may be given added value such as level improvement, acquisition of rewards/items, acquisition of points, and progress in the story, depending on the skill level, operation success rate, number of operations processed, operation processing speed, etc.
  • acquired points may be exchangeable for money in the virtual world or the real world.
  • step S104 the server 13 relays the remote operation content information.
  • the remote operator performs remote operation of the requested task according to the remote operation UI presented by the operation terminal 12.
  • the operation terminal 12 generates remote operation content information indicating the content of the remote operation, and transmits it to the server 13 via the network 21.
  • the remote control unit 311 of the server 13 receives the remote operation content information via the network 21 and the communication unit 301.
  • the remote control unit 311 converts the remote operation content information for use by the robot 11 as necessary. For example, depending on the contents of the remote operation UI, it may be necessary to convert the contents of the operation performed by the remote operator into information (e.g., an operation signal) for actually remotely operating the robot 11. In response to this, the remote control unit 311 converts the remote operation content information for use by the robot 11.
  • information e.g., an operation signal
  • steps S103 to S106 are repeatedly executed until it is determined in step S105 that the task has been completed, or until it is determined in step S106 that the task has failed.
  • step S106 if the remote control unit 311 receives a task failure notification signal transmitted from the robot 11 in the processing of step S59 in FIG. 7 via the network 21 and the communication unit 301, it determines that the task has failed, and the processing proceeds to step S107.
  • the remote control unit 311 updates information about the remote operator who failed the remote operation (e.g., skill level, remote operation history, etc.) in the remote operator database stored in the memory unit 303 as necessary. Also, the remote control unit 311 updates information about the task whose remote operation failed (e.g., content, difficulty, etc.) in the task database stored in the memory unit 303 as necessary.
  • information about the remote operator who failed the remote operation e.g., skill level, remote operation history, etc.
  • the remote control unit 311 updates information about the task whose remote operation failed (e.g., content, difficulty, etc.) in the task database stored in the memory unit 303 as necessary.
  • step S101 the process returns to step S101, and the processes from step S101 onwards are executed.
  • step S106 if the remote control unit 311 receives a task failure notification signal transmitted from the robot 11 in the processing of step S59 in FIG. 7 via the network 21 and the communication unit 301, it determines that the task has failed, and the processing proceeds to step S107.
  • step S105 if the remote control unit 311 receives a task completion notification signal transmitted from the robot 11 in the processing of step S61 in FIG. 7 via the network 21 and the communication unit 301, it determines that the task has been completed, and the processing proceeds to step S108.
  • step S108 the remote control unit 311 notifies the remote operator of the completion of the task. Specifically, the remote control unit 311 transmits a task completion notification signal to the operation terminal 12 of the remote operator via the communication unit 301 and the network 21.
  • the remote control unit 311 updates information about the remote operator who succeeded in remote operation (e.g., skill level, remote operation history, etc.) in the remote operator database stored in the memory unit 303 as necessary. Also, the remote control unit 311 updates information about the task that was successfully remotely operated (e.g., content, difficulty, etc.) in the task database stored in the memory unit 303 as necessary.
  • information about the remote operator who succeeded in remote operation e.g., skill level, remote operation history, etc.
  • the remote control unit 311 updates information about the task that was successfully remotely operated (e.g., content, difficulty, etc.) in the task database stored in the memory unit 303 as necessary.
  • step S101 the process returns to step S101, and the processes from step S101 onwards are executed.
  • the state recognition unit 131 detects speech from a user near the robot 11.
  • the autonomous behavior control unit 132 instructs the dialogue module 141a on the task of executing a dialogue with the user.
  • the autonomous dialogue unit 161a executes a dialogue with the user and recognizes the words "I want to drink something" in the user's speech.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the selected mobile operator to remotely operate the task by executing the matching process.
  • the remote dialogue unit 162a executes a dialogue with the user through remote operation by the dialogue operator. As a result, the remote dialogue unit 162a recognizes with a high degree of certainty that the user "wants to drink tea.”
  • the action planning unit 121 plans the action of "delivering a cup of tea to the user.”
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving to the front of the shelf on which the cup is placed.
  • the autonomous movement unit 161d moves the robot 11 to a position 1 m in front of the shelf.
  • the autonomous behavior control unit 132 instructs the observation module 141b to perform the task of searching for a cup on the shelf.
  • the shelf has no door, and the tableware inside is visible.
  • the autonomous observation unit 161b searches for a cup inside the shelf. However, there are more items stored in the shelf than usual, and the autonomous observation unit 161b has a low degree of confidence in recognizing the cup.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the observation operator selected by executing the matching process to remotely operate the task.
  • the remote observation unit 162b discovers the cup through remote operation by the observation operator.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving the robot 11 to a position 50 cm in front of the cup.
  • the autonomous movement unit 161d moves the robot 11 to a position 50 cm in front of the cup.
  • the autonomous behavior control unit 132 instructs the task of grasping the cup to the work module 141c.
  • the autonomous operator 161c attempts to grasp the cup, but there are many other objects around the cup, making it difficult to grasp the cup. As a result, the autonomous operator 161c exceeds the instructed working time while searching for the optimal approach to the cup.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the selected operator to remotely operate the task by executing the matching process.
  • the remote operation unit 162c causes the robot 11 to grasp the cup through remote control by the remote operator.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving in front of the tea dispenser.
  • the autonomous movement unit 161d moves the robot 11 to the front of the tea dispenser.
  • the autonomous behavior control unit 132 instructs the task of placing a cup in the tea dispenser to the work module 141c.
  • the autonomous operating unit 161c places a cup in the tea dispenser.
  • the autonomous behavior control unit 132 instructs the task of operating the tea dispensing switch of the tea dispenser to the work module 141c.
  • the autonomous operating unit 161c operates the tea dispensing switch of the tea dispenser. This causes the tea dispenser to pour tea into a cup.
  • the autonomous behavior control unit 132 instructs the task of grasping the cup containing tea to the task module 141c.
  • the autonomous operator 161c begins to grasp the cup.
  • the autonomous dialogue unit 161a detects an utterance made by a third party, a staff member in the kitchen, to the robot 11.
  • the autonomous dialogue unit 161a recognizes through voice recognition that the staff member's utterance is "Move aside for a moment.”
  • the behavior planning unit 121 determines that moving to a location that will not get in the way of staff members has a higher priority than grasping the cup.
  • the autonomous behavior control unit 132 instructs the work module 141c to stop the task of grasping the cup.
  • the autonomous operator 161c stops grasping the cup and returns the arm of the robot 11, which was about to grasp the cup, to its original standard position.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving from in front of the tea dispenser to a location 1 m away in the opposite direction to the staff.
  • the autonomous moving unit 161d attempts to move from in front of the tea dispenser. However, because the passage is narrow and there are many obstacles, including people, around the robot 11, the autonomous moving unit 161d fails to plan the movement path, and the robot 11 becomes unable to move.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the selected mobile operator to remotely operate the task by executing the matching process.
  • the remote movement unit 162d moves the robot 11 from in front of the tea dispenser by remote control from the movement operator.
  • the autonomous behavior control unit 132 instructs the observation module 141b to carry out the task of checking the status of obstacles, including people, in front of the tea dispenser to determine whether it is OK to return to the original task (holding the cup).
  • the autonomous observation unit 161b confirms that there are no obstacles, including people, in front of the tea dispenser. However, because there are many objects around the tea dispenser, the autonomous observation unit 161b has a low degree of confidence.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the observation operator selected by executing the matching process to remotely operate the task.
  • the remote observation unit 162b verifies that there are no obstacles in front of the tea dispenser through remote operation by the observation operator.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving in front of the tea dispenser.
  • the autonomous movement unit 161d moves the robot 11 to the front of the tea dispenser.
  • the autonomous behavior control unit 132 instructs the task of grasping a cup placed in the tea dispenser to the task module 141c.
  • the autonomous operator 161c causes the robot 11 to grasp the cup.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving to the desk where the user is located.
  • the autonomous movement unit 161d moves the robot 11 to the desk where the user is located.
  • the autonomous behavior control unit 132 instructs the observation module 141b to perform the task of finding the user and checking the user's status.
  • the autonomous observation unit 161b detects the position and orientation of the user's face. However, because the user tends to look down and there is a lot of shadow on the user's face, the autonomous observation unit 161b has a low degree of confidence.
  • the autonomous behavior control unit 132 requests the server 13 to remotely control the task.
  • the server 13 requests the observation operator selected by executing the matching process to remotely operate the task.
  • the remote observation unit 162b discovers the user through remote operation by the observation operator and confirms that there is nothing wrong with the user.
  • the autonomous behavior control unit 132 instructs the dialogue module 141a to execute a task of speaking to the user, such as "I've brought the tea” or "I'll put the cup down.”
  • the autonomous dialogue unit 161a calls out to the user. In response, the user replies, but because the user tends to look down, his/her pronunciation is unclear, and there is a lot of environmental noise, it is difficult for the autonomous dialogue unit 161a to recognize the user's reply.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the selected dialogue operator to remotely operate the task by executing the matching process.
  • the remote dialogue unit 162a speaks to the user through remote operation by the dialogue operator.
  • the remote dialogue unit 162a (remote operator) then confirms with a high degree of certainty that the user is positive.
  • the autonomous behavior control unit 132 instructs the task of placing the cup on the desk to the task module 141c.
  • the autonomous operator 161c places the cup on the desk.
  • the autonomous behavior control unit 132 instructs the dialogue module 141a on the task of interacting with the user.
  • the autonomous dialogue unit 161a executes dialogue with the user.
  • the action planning unit 121 confirms that the action of "carrying a cup of tea to the user" has been completed.
  • the robot 11 can pour tea into a cup and bring it to the user while interacting with the user.
  • the position where the robot 11 is located is the home position P0, where, for example, a charging dock (not shown) is located.
  • a charging dock (not shown) is located.
  • tables 401-1 to 401-8 and chairs 402-1 to 402-7 are located in the living room.
  • Tables 401-1 to 401-8 are arranged in a U-shape. That is, the row of tables 401-1 and 401-2, the row of tables 401-3 to 401-6, and the row of tables 401-7 and 401-8 are arranged in a U-shape.
  • the row of tables 401-3 to 401-6 is arranged in front of the robot 11.
  • the row of tables 401-1 and 401-2 and the row of tables 401-7 and 401-8 are arranged in a direction perpendicular to the row of tables 401-3 to 401-6 and face each other.
  • chairs 402-1 through 402-7 are arranged at tables 401-1 through 401-7, respectively. Note that no chair is arranged at table 401-8.
  • tables 401-1 to 401-7 when there is no need to distinguish between tables 401-1 to 401-7, they will simply be referred to as table 401.
  • chairs 402-1 to 402-7 when there is no need to distinguish between chairs 402-1 to 402-7, they will simply be referred to as chairs 402.
  • FIG. 9 shows a long distance position P3, a medium distance position P2, and a short distance position P1 for each chair 402.
  • the long distance position P3 is set at a position approximately 0 degrees in front of the subject 403 for whom the robot 11 is to execute the application, and approximately 2 m away from the subject 403.
  • the mid-distance position P2 is set at a position approximately 45 degrees in front of the subject 403 and approximately 1.2 m away from the subject 403.
  • the close distance position P1 is set at a position approximately 70 degrees in front of the subject 403 and approximately 0.6 m away from the subject 403.
  • the behavior planning unit 121 approaches the subject 403 in the order of long-distance position P3 ( Figure 10), medium-distance position P2 ( Figure 11), and close-distance position P1 ( Figure 12), speaking to the subject 403 at each position, and finally plans an action to measure the vital signs of the subject 403 at close-distance position P1.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving to the long-distance position P3.
  • the autonomous movement unit 161d moves the robot 11 to the long distance position P3.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the selected mobile operator to remotely operate the task by executing the matching process.
  • the remote movement unit 162d moves the robot 11 to a long distance position P3 by remote operation of the movement operator.
  • the movement operator has better situation recognition and judgment than the robot 11, so the success rate of the task increases. This also applies to the following movement tasks.
  • the autonomous behavior control unit 132 requests the dialogue module 141a to perform the task of dialogue with the subject 403.
  • the autonomous dialogue unit 161a executes a dialogue with the subject 403.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the selected dialogue operator to remotely operate the task by executing the matching process.
  • the remote dialogue unit 162a executes a dialogue with the target person 403 through remote operation by the dialogue operator.
  • the dialogue operator has better situation recognition and judgment than the robot 11, so the success rate of the task increases. This also applies to the following dialogue tasks.
  • the autonomous movement unit 161d moves the robot 11 to the intermediate distance position P2.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the remote movement unit 162d moves the robot 11 to the intermediate position P2 by remote control by the movement operator.
  • the autonomous behavior control unit 132 requests the dialogue module 141a to perform the task of dialogue with the subject 403.
  • the autonomous dialogue unit 161a executes a dialogue with the subject 403.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the dialogue operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the dialogue operator requested last time.
  • the remote dialogue unit 162a executes a dialogue with the target person 403 through remote control by the dialogue operator.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving to the nearby position P1.
  • the autonomous movement unit 161d moves the robot 11 to the close distance position P1.
  • the server 13 requests the movement operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the movement operator that was previously requested.
  • the remote movement unit 162d moves the robot 11 to the close-distance position P1 by remote control of the movement operator.
  • the autonomous dialogue unit 161a executes a dialogue with the subject 403.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the dialogue operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the dialogue operator requested last time.
  • the remote dialogue unit 162a executes a dialogue with the target person 403 through remote control by the dialogue operator.
  • the autonomous observation unit 161b attempts to recognize the position of the subject's 403 face and hands.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the observation operator selected by executing the matching process to remotely operate the task.
  • the remote observation unit 162b recognizes the position of the face and hands of the subject 403 through remote operation by the observing operator.
  • the observing operator has better situation recognition and judgment than the robot 11, so the success rate of the task increases. This also applies to the following observation tasks.
  • the autonomous behavior control unit 132 instructs the task module 141c to hold out the hand 233 of the robot 11 in front of the target person 403.
  • the autonomous operation unit 161c attempts to hold out the hand 233 of the robot 11 in front of the subject 403.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the selected operator to remotely operate the task by executing the matching process.
  • the remote observation unit 162b under remote control by the operator, holds out the hand 233 of the robot 11 in front of the subject 403.
  • the autonomous dialogue unit 161a detects an unexpected situation in which a third party, a staff member, speaks to the robot 11.
  • the autonomous dialogue unit 161a attempts to recognize the content of the staff member's speech.
  • the remote action control unit 133 requests the server 13 to remotely operate a task to recognize the content of the staff member's speech.
  • the server 13 requests the selected dialogue operator to remotely operate the task by executing the matching process.
  • the remote dialogue unit 162a dialogues with the staff member through remote operation by the dialogue operator, and recognizes that the staff member's utterance is "Move aside for a moment.”
  • the action planning unit 121 determines that the task of moving to allow staff to pass has a high priority.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving to allow staff to pass.
  • the autonomous movement unit 161d attempts to move the robot 11 so that the staff member can pass through.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the movement operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the movement operator that was previously requested.
  • the remote movement unit 162d moves the robot 11 by remote control from the movement operator so that staff can pass through.
  • the autonomous behavior control unit 132 instructs the observation module 141b to perform the task of checking for the passage of staff members.
  • the autonomous observation unit 161b attempts to confirm the passage of staff members.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the observing operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the observing operator that was previously requested.
  • the remote observation unit 162b confirms the passage of staff members through remote operation by the observation operator.
  • the autonomous behavior control unit 132 instructs the dialogue module 141a to perform the task of asking the subject 403 to touch the hand 233 of the robot 11.
  • the autonomous dialogue unit 161a asks the subject 403 to touch the hand 233 of the robot 11.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the dialogue operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the dialogue operator requested last time.
  • the remote dialogue unit 162a through remote control by the dialogue operator, requests the target person 403 to touch the hand 233 of the robot 11.
  • the autonomous behavior control unit 132 instructs the observation module 141b to perform the task of confirming that the hand of the subject 403 is in contact with the hand 233 of the robot 11.
  • the autonomous observation unit 161b confirms that the hand of the subject 403 is in contact with the hand 233 of the robot 11.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the observing operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the observing operator that was previously requested.
  • the remote observation unit 162b confirms that the hand of the subject 403 is in contact with the hand 233 of the robot 11 through remote operation by the observation operator.
  • the autonomous behavior control unit 132 may request remote operation in parallel from the work operator, the dialogue operator, and the observation operator via the server 13. Then, the work operator, the dialogue operator, and the observation operator may perform remote operation in parallel.
  • the work module 141c that is remotely controlled by the work operator, the dialogue module 141a that is remotely controlled by the dialogue operator, and the observation module 141b that is remotely controlled by the observation operator can independently execute tasks, so that each task can be smoothly processed in parallel by remote control.
  • the task operator, the dialogue operator, and the observation operator do not necessarily have to be separate operators.
  • one operator with multiple skills may serve as two or more operators.
  • the autonomous behavior control unit 132 instructs the observation module 141b to perform the task of measuring the vital signs of the subject 403.
  • the remote observation unit 162b measures the vital signs of the subject 403.
  • the autonomous behavior control unit 132 instructs the dialogue module 141a to perform a task of requesting the subject 403 to remove his/her hand from the hand 233 of the robot 11.
  • the autonomous dialogue unit 161a requests the subject 403 to remove his/her hand from the robot 11's hand 233.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the dialogue operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the dialogue operator requested last time.
  • the remote dialogue unit 162a requests the dialogue operator to remotely control the target person 403 to remove his/her hand from the robot 11's hand 233.
  • the autonomous behavior control unit 132 instructs the observation module 141b to perform the task of confirming that the hand of the subject 403 is away from the hand 233 of the robot 11.
  • the autonomous observation unit 161b confirms that the subject's 403 hand is away from the robot's 11 hand 233.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the observing operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the observing operator that was previously requested.
  • the remote observation unit 162b confirms that the hand of the subject 403 is removed from the hand 233 of the robot 11 by remote operation of the observation operator.
  • the autonomous behavior control unit 132 instructs the task module 141c to return the hand 233 of the robot 11 to its original position.
  • the autonomous operation unit 161c attempts to return the hand 233 of the robot 11 to its original position.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the selected operator to remotely operate the task by executing the matching process.
  • the remote observation unit 162b returns the hand 233 of the robot 11 to its original position through remote control by the operator.
  • the autonomous behavior control unit 132 requests the dialogue module 141a to perform the task of dialogue with the subject 403.
  • the autonomous dialogue unit 161a executes a dialogue with the subject 403.
  • the remote action control unit 133 requests the server 13 to remotely control the task.
  • the server 13 requests the dialogue operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the dialogue operator requested last time.
  • the remote dialogue unit 162a executes a dialogue with the target person 403 through remote control by the dialogue operator.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving to the next target position.
  • the autonomous movement unit 161d moves the robot 11 to the next target position.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the movement operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the movement operator that was previously requested.
  • the remote movement unit 162d moves the robot 11 to the next target position by remote control from the movement operator.
  • the robot 11 can smoothly measure the vital signs of the subject 403.
  • the behavior planning unit 121 plans a behavior of moving to the front of a door, opening the door, and then passing through the door.
  • the autonomous behavior control unit 132 instructs the movement module 141d to perform the task of moving to the front of the door.
  • the autonomous movement unit 161d moves the robot 11 to the front of the door.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the selected mobile operator to remotely operate the task by executing the matching process.
  • the remote movement unit 162d moves the robot 11 to the front of the door by remote control from the movement operator.
  • the autonomous behavior control unit 132 instructs the observation module 141b to perform the task of recognizing the door handle.
  • the autonomous observation unit 161b attempts to recognize the door handle.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the observation operator selected by executing the matching process to remotely operate the task.
  • the remote observation unit 162b recognizes the door handle through remote operation by the observation operator.
  • the behavior planning unit 121 plans the behavior of opening the door and moving forward based on the shape and position of the door and door handle, etc.
  • the autonomous behavior control unit 132 instructs the task of gripping the door handle to the work module 141c.
  • the autonomous operation unit 161c attempts to grasp the door handle.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the selected operator to remotely operate the task by executing the matching process.
  • the remote observation unit 162b grasps the door handle by remote control of the operator. At this time, the operator has a better ability to recognize the shape and position of the door handle than the robot 11, so the success rate of the task increases.
  • the autonomous behavior control unit 132 instructs the task of rotating the door handle to the work module 141c.
  • the autonomous operation unit 161c attempts to rotate the door handle.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the work operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the work operator that was previously requested.
  • the remote observation unit 162b rotates the door handle by remote control of the operator. At this time, the operator has a better ability than the robot 11 to adjust the direction and force of movement according to the type and shape of the door handle, and this increases the success rate of the task.
  • the autonomous behavior control unit 132 instructs the work module 141c and the movement module 141d to perform the task of pulling the door open.
  • the autonomous work unit 161c and the autonomous movement unit 161d attempt to pull the door open while moving the robot 11 appropriately.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the work operator and the movement operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the work operator requested previously. The server 13 may also request remote operation of the task from one remote operator.
  • the autonomous behavior control unit 132 instructs the task of holding the door to the work module 141c.
  • the autonomous operating unit 161c attempts to hold the door with the hand opposite the hand holding the door handle.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the work operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the work operator that was previously requested.
  • the remote operation unit 162c holds the door by remote control from the operator.
  • the autonomous behavior control unit 132 instructs the work module 141c and the movement module 141d to carry out the task of passing through the door while holding the door.
  • the autonomous work unit 161c and the autonomous movement unit 161d attempt to pass through the door while holding it.
  • the remote action control unit 133 requests the server 13 to remotely operate the task.
  • the server 13 requests the work operator and movement operator selected by executing the matching process to remotely operate the task. At this time, the server 13 may preferentially request remote operation from the work operator and movement operator requested previously. The server 13 may also request remote operation of the task from one remote operator.
  • the remote work unit 162c and the remote movement unit 162d pass through the door while holding the door, under the remote control of the work operator and the movement operator. At this time, the work operator and the movement operator have better situation recognition and judgment than the robot 11, so the success rate of the task increases.
  • the robot 11 can smoothly open the door and pass through.
  • two or more of the work operator, movement operator, and observation operator may perform remote operation in parallel as necessary. This may improve the efficiency of remote operation of each task.
  • the work operator, movement operator, and observation operator do not necessarily have to be separate operators; for example, one operator with multiple skills may serve as two or more operators.
  • the reliability and functionality of the various actions of the robot 11 are improved.
  • by autonomously executing tasks that the robot 11 is good at and remotely executing tasks that it is not good at it is possible to realize a robot 11 with a new architecture that can utilize the brain and sensory organs of a remote operator as components. This makes it possible to increase the reliability of the various actions of the robot 11 to a human level or even higher.
  • each action of the robot 11 is broken down into multiple tasks down to the micro level and remote operation is requested, improving the work efficiency of the remote operator.
  • each task is simplified to a level that is unrelated to the context, making remote operation easier and simpler and reducing the time required. Therefore, for example, each remote operator can remotely operate multiple robots 11 in parallel. Also, for example, each remote operator can participate in remote operation of the robots 11 while performing other tasks or work, or participate in remote operation of the robots 11 during short free time.
  • the robot 11 is modularized for each skill, it becomes possible to execute remote operations related to each skill in parallel. This makes it possible for multiple remote operators to remotely operate one robot 11 in parallel.
  • the skills required of remote operators will be lowered, allowing more remote operators to handle the tasks.
  • each task is simple, it will be easier to realize a user interface with good operability. Therefore, for example, people with physical disabilities will be able to easily operate the robots remotely.
  • many people around the world will be able to participate in and play an active role in remotely operating robots 11 around the world.
  • each remote operator will be able to find work more easily, and the labor shortage will be alleviated.
  • the required level of each task is lowered and operability is improved, the failure rate of remote operation will decrease and the quality of tasks performed by remote operation will improve.
  • FIG. 13 shows an example of a formula for calculating the operational effect R of the remote operation control system 1.
  • the operational effect R is proportional to the total number of robots M, the parallel subdivision coefficient P, and the autonomy coefficient Q.
  • the operational effect R is inversely proportional to the number of operators N, the number of operation modules K, and the required operation skill coefficient J.
  • the total number of robots M is the total number of robots 11 operating in the remote control system 1.
  • the parallel subdivision coefficient P indicates the degree of parallel subdivision of a task. For example, the more a task is subdivided, the larger the parallel subdivision coefficient P.
  • the autonomy coefficient Q indicates the degree of autonomy of the robot 11. The greater the proportion of tasks that the robot 11 can execute autonomously, the greater the autonomy coefficient Q. Typically, the more a task is subdivided, the greater the proportion of tasks that the robot 11 can execute autonomously, and therefore the greater the autonomy coefficient Q.
  • the number of operators N is the total number of remote operators required to smoothly operate the robots 11 out of a total of M robots operating in the remote operation control system 1.
  • the number of operation modules K is the number of task execution modules 141 that the robot 11 has.
  • the robot 11 described above has four task execution modules 141: a dialogue module 141a, an observation module 141b, a work module 141c, and a movement module 141d, so the value of the number of operation modules K is 4.
  • the required operation skill coefficient J indicates the level of skill required for remote operation of a task, and the higher the level of skill required for remote operation of a task, the larger the required operation skill coefficient J. Normally, the more a task is subdivided, the lower the level of skill required for remote operation of the task, and the smaller the required operation skill coefficient J. Normally, the more robots 11 per person, the higher the level of skill required for remote operation of the task, and the larger the required operation skill coefficient J.
  • the number of operators N and the number of operation modules K increase, which causes a decrease in the operational effect R.
  • the parallel subdivision coefficient P increases and the required operation skill coefficient J decreases. This increases the number of robots 11 that each remote operator can operate, making it possible to significantly increase the total number of robots M. Therefore, the overall operational effect R can be increased.
  • Figure 14 shows a specific example of operational effect R.
  • Example 1 shows a case where one remote operator operates one robot 11.
  • the operational effect R is 1, and this value becomes the reference value of the operational effect R.
  • Example 1 there is one remote operator, and it is not necessary to divide the task execution modules of the robot 11, so the number of operation modules K is set to 1.
  • Example 2 shows a case where one remote operator operates five robots 11. In this case, it is necessary to increase the autonomy coefficient Q of the robots 11 in order to suppress an increase in the required operation skill coefficient J. However, even if the autonomy coefficient Q is increased, one remote operator needs to operate five robots all the time, so the required operation skill coefficient J becomes large. And in this example, the value of the operation effect R is 2.
  • Example 2 as in Example 1, there is one remote operator, and it is not necessary to divide the task execution module of the robot 11, so the number of operation modules K is set to 1.
  • Example 3 shows a case where 10 remote operators operate 100 robots 11 in the remote operation control system 1.
  • each robot 11 has four task execution modules 141, so the number of operation modules K is 4.
  • the number of operation modules K increases, the degree of parallel subdivision of tasks increases, the parallel subdivision coefficient P increases, and the required operation skill coefficient J decreases.
  • the value of the operation effect R is improved from 2 to 20 compared to Example 2. In other words, it is now possible to operate a large number of robots with a reduced number of remote operators, improving the operation effect.
  • classification of the robot 11's skills can be changed as appropriate.
  • the server 13 may execute a matching process and control the connection between the robot 11 to be remotely operated and the operation terminal 12 of the remote operator, and then the robot 11 and the operation terminal 12 may communicate directly without going through the server 13 to execute the remote operation.
  • the robot 11 can execute a matching process by itself based on a remote operator database, etc., and request remote operation from a selected remote operator (of the operation terminal 12).
  • the robot 11 can generate the remote operator database, or can acquire the remote operator database from an external source.
  • This technology can be applied to autonomous mobile objects other than robots that can be remotely controlled.
  • this technology can be applied to vehicles, drones, and other vehicles that can be remotely controlled and operate autonomously.
  • this technology can be applied to manipulators that do not move but can remotely control or autonomously move parts such as hands.
  • FIG. 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes using a program.
  • a CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • an input/output interface 1005 Connected to the input/output interface 1005 are an input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010.
  • the input unit 1006 includes an input switch, a button, a microphone, an image sensor, etc.
  • the output unit 1007 includes a display, a speaker, etc.
  • the storage unit 1008 includes a hard disk, a non-volatile memory, etc.
  • the communication unit 1009 includes a network interface, etc.
  • the drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
  • the CPU 1001 loads a program recorded in the storage unit 1008, for example, into the RAM 1003 via the input/output interface 1005 and the bus 1004, and executes the program, thereby performing the above-mentioned series of processes.
  • the program executed by the computer 1000 can be provided by being recorded on a removable medium 1011 such as a package medium, for example.
  • the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
  • the program can be installed in the storage unit 1008 via the input/output interface 1005 by inserting the removable medium 1011 into the drive 1010.
  • the program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008.
  • the program can be pre-installed in the ROM 1002 or storage unit 1008.
  • the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or a program in which processing is performed in parallel or at the required timing, such as when called.
  • a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.
  • this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.
  • each step described in the above flowchart can be executed by a single device, or can be shared and executed by multiple devices.
  • one step includes multiple processes
  • the multiple processes included in that one step can be executed by one device, or can be shared and executed by multiple devices.
  • a matching unit that performs matching between a non-autonomous task, which is a task that is not performed autonomously by the autonomous mobile body, among tasks obtained by decomposing the behavior of the autonomous mobile body, and a remote operator that remotely operates the non-autonomous task; and a remote control unit that controls remote operation of the non-autonomous task by the remote operator.
  • the remote control unit receives remote operation content information indicating the content of the remote operation by the remote operator from an operation terminal operated by the remote operator, and transmits the remote operation content information to the autonomous moving body.
  • the remote control unit provides an operation terminal with a user interface for remotely operating the non-autonomous task.
  • the non-autonomous tasks are classified by skill; The information processing device according to any one of (1) to (7), wherein the matching unit matches the remote operator capable of remote operation of a skill related to the non-autonomous task with the non-autonomous task.
  • the information processing device according to any one of (1) to (10), wherein the non-autonomous task includes the task that the autonomous moving body cannot execute autonomously and the task that the autonomous moving body has failed to execute autonomously.
  • a non-autonomous task which is a task that the autonomous mobile body does not perform autonomously, is matched with a remote operator who remotely operates the non-autonomous task; A remote operation control method for controlling remote operation of the non-autonomous task by the remote operator.
  • the information processing device includes: a matching unit that performs matching between a non-autonomous task, which is a task that is not performed autonomously by the autonomous moving body, among tasks obtained by decomposing the behavior of each of the autonomous moving bodies, and a remote operator that remotely operates the non-autonomous task; and a remote control unit that controls remote operation of the non-autonomous task by the remote operator.
  • a non-autonomous task which is a task that is not performed autonomously by the autonomous moving body, among tasks obtained by decomposing the behavior of each of the autonomous moving bodies, and a remote operator that remotely operates the non-autonomous task
  • a remote control unit that controls remote operation of the non-autonomous task by the remote operator.
  • an action planning unit that breaks down an action to be executed into a plurality of tasks
  • a behavior control unit that controls the autonomous execution of autonomous tasks that are executed autonomously among the tasks, and the execution by remote control of non-autonomous tasks that are not executed

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Manipulator (AREA)
PCT/JP2024/005917 2023-03-10 2024-02-20 情報処理装置、遠隔操作制御方法、遠隔操作制御システム、及び、自律移動体 Ceased WO2024190308A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP24770421.6A EP4679858A4 (en) 2023-03-10 2024-02-20 Information processing device, remote operation control method, remote operation control system, and autonomous mobile body
JP2025506628A JPWO2024190308A1 (https=) 2023-03-10 2024-02-20
CN202480016157.5A CN120958841A (zh) 2023-03-10 2024-02-20 信息处理装置、远程操作控制方法、远程操作控制系统和自主移动体

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023037370 2023-03-10
JP2023-037370 2023-03-10

Publications (1)

Publication Number Publication Date
WO2024190308A1 true WO2024190308A1 (ja) 2024-09-19

Family

ID=92755461

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/005917 Ceased WO2024190308A1 (ja) 2023-03-10 2024-02-20 情報処理装置、遠隔操作制御方法、遠隔操作制御システム、及び、自律移動体

Country Status (4)

Country Link
EP (1) EP4679858A4 (https=)
JP (1) JPWO2024190308A1 (https=)
CN (1) CN120958841A (https=)
WO (1) WO2024190308A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7741598B1 (ja) * 2025-06-20 2025-09-18 範之 澤谷 捜索救助システム及びこれを用いた捜索救助方法

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004243499A (ja) * 2003-02-17 2004-09-02 Matsushita Electric Ind Co Ltd 生活空間用の物品取扱いシステム、物品取扱い方法、及びロボット操作装置
WO2008140011A1 (ja) * 2007-05-09 2008-11-20 Nec Corporation 遠隔操作システム、サーバ、被遠隔操作装置、遠隔操作サービス提供方法
JP2012230506A (ja) 2011-04-25 2012-11-22 Sony Corp 評価装置及び評価方法、サービス提供システム、並びにコンピューター・プログラム
JP2020507160A (ja) * 2017-01-20 2020-03-05 フォロー インスピレーション，エセ．アー． 自律型ロボットシステム
JP2021064064A (ja) * 2019-10-10 2021-04-22 沖電気工業株式会社 ロボットシステム、ロボット及び操作端末
JP2021070140A (ja) * 2019-11-01 2021-05-06 株式会社Ｐｒｅｆｅｒｒｅｄ Ｎｅｔｗｏｒｋｓ 被遠隔操作装置、遠隔操作システム、遠隔操作支援方法、プログラム及び非一時的コンピュータ可読媒体
US20220051165A1 (en) * 2020-08-17 2022-02-17 Teleo, Inc. Systems and methods of assigning microtasks of workflows to teleoperators
JP2022126658A (ja) * 2019-08-01 2022-08-30 オムロン株式会社 作物の栽培管理のための装置、システム、方法、およびプログラム

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004243499A (ja) * 2003-02-17 2004-09-02 Matsushita Electric Ind Co Ltd 生活空間用の物品取扱いシステム、物品取扱い方法、及びロボット操作装置
WO2008140011A1 (ja) * 2007-05-09 2008-11-20 Nec Corporation 遠隔操作システム、サーバ、被遠隔操作装置、遠隔操作サービス提供方法
JP2012230506A (ja) 2011-04-25 2012-11-22 Sony Corp 評価装置及び評価方法、サービス提供システム、並びにコンピューター・プログラム
JP2020507160A (ja) * 2017-01-20 2020-03-05 フォロー インスピレーション，エセ．アー． 自律型ロボットシステム
JP2022126658A (ja) * 2019-08-01 2022-08-30 オムロン株式会社 作物の栽培管理のための装置、システム、方法、およびプログラム
JP2021064064A (ja) * 2019-10-10 2021-04-22 沖電気工業株式会社 ロボットシステム、ロボット及び操作端末
JP2021070140A (ja) * 2019-11-01 2021-05-06 株式会社Ｐｒｅｆｅｒｒｅｄ Ｎｅｔｗｏｒｋｓ 被遠隔操作装置、遠隔操作システム、遠隔操作支援方法、プログラム及び非一時的コンピュータ可読媒体
US20220051165A1 (en) * 2020-08-17 2022-02-17 Teleo, Inc. Systems and methods of assigning microtasks of workflows to teleoperators

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4679858A1

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7741598B1 (ja) * 2025-06-20 2025-09-18 範之 澤谷 捜索救助システム及びこれを用いた捜索救助方法
JP7759154B1 (ja) * 2025-06-20 2025-10-23 範之 澤谷 捜索救助システム及びこれを用いた捜索救助方法

Also Published As

Publication number Publication date
JPWO2024190308A1 (https=) 2024-09-19
EP4679858A4 (en) 2026-05-06
EP4679858A1 (en) 2026-01-14
CN120958841A (zh) 2025-11-14

Similar Documents

Publication Publication Date Title
Rea et al. Still not solved: A call for renewed focus on user-centered teleoperation interfaces
Mišeikis et al. Lio-a personal robot assistant for human-robot interaction and care applications
CN115551681B (zh) 训练和/或利用用于基于自然语言的机器人控制中的机器学习模型
Hoffman et al. Collaboration in human-robot teams
Mitsunaga et al. Robovie-IV: A communication robot interacting with people daily in an office
Breazeal et al. Crowdsourcing human-robot interaction: New methods and system evaluation in a public environment
Jung et al. Engaging robots: easing complex human-robot teamwork using backchanneling
JP5070441B2 (ja) ロボット遠隔操作システム
Hauser et al. Analysis and perspectives on the ana avatar xprize competition
Krishna Sharma et al. Eye gaze controlled robotic arm for persons with severe speech and motor impairment
Hai et al. Remote healthcare for the elderly, patients by tele-presence robot
US12508715B1 (en) Robotic systems and methods
Li et al. Exploring large language model-driven agents for environment-aware spatial interactions and conversations in virtual reality role-play scenarios
Sharma et al. Webcam controlled robotic arm for persons with SSMI
WO2024190308A1 (ja) 情報処理装置、遠隔操作制御方法、遠隔操作制御システム、及び、自律移動体
Krishna Sharma et al. Comparing two safe distance maintenance algorithms for a gaze-controlled HRI involving users with SSMI
Kapadia et al. EchoBot: Facilitating data collection for robot learning with the Amazon echo
Graefe et al. Past, present and future of intelligent robots
JP7548970B2 (ja) ドッペルゲンガー遠隔ロボットシステム
Wilkes et al. Designing for human‐robot symbiosis
WO2024190301A1 (en) Information processing device and nursing care robot control method
Haase et al. Towards context-aware natural language understanding in human-robot-collaboration
Kanda et al. Development of Robovie as a platform for everyday‐robot research
Chung et al. Continuous vocalization control of a full-scale assistive robot
CN120611036A (zh) 机器人交互控制方法、服务器、客户端和机器人

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24770421

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025506628

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025506628

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 2024770421

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2024770421

Country of ref document: EP

Effective date: 20251010

ENP Entry into the national phase

Ref document number: 2024770421

Country of ref document: EP

Effective date: 20251010

ENP Entry into the national phase

Ref document number: 2024770421

Country of ref document: EP

Effective date: 20251010

ENP Entry into the national phase

Ref document number: 2024770421

Country of ref document: EP

Effective date: 20251010

WWP Wipo information: published in national office

Ref document number: 2024770421

Country of ref document: EP