WO2023053521A1 - Information processing device, information processing method, and information processing system - Google Patents

Abstract

The present technology relates to an information processing device, an information processing method, and an information processing system that appropriately operate a robot remotely. The information processing device comprises: an operator monitoring unit that monitors eligibility for operation by an operator; a work monitoring unit that monitors a work status that is a status of work performed by the operator operating a robot remotely by using an operation terminal; and a mediation unit that controls mediation for remote operation between the operation terminal and the robot on the basis of the result of monitoring the eligibility for operation and the work status. The present technology can be applied to a server that mediates remote operation between the operation terminal and the robot, for example.

Description

Information processing device, information processing method, and information processing system

The present technology relates to an information processing device, an information processing method, and an information processing system, and more particularly to an information processing device, an information processing method, and an information processing system suitable for remote control of a robot.

When requesting remote control of a robot, the aptitude of the remote control operator is important.

On the other hand, conventionally, the operator's skill level for remote operation is calculated based on the work history in which the actual work results in the actual work mode, the simulation results in the simulation mode, and the verification results in the verification mode are stored. , to present the skill level (see, for example, Patent Document 1).

In addition, conventionally, one or more robots in charge of remote operation of robots are based on information stored in a management table storage unit that manages the state attribute information of each robot and the skill characteristic information of each operator placed on each console. It has been proposed to determine an operator console for each (see, for example, Patent Document 1).

JP 2019-207570 A Japanese Patent Application Laid-Open No. 2021-62426

However, even if an operator with the required ability is selected, it does not mean that the work will actually be performed according to the operator's ability. For example, it is assumed that the operator will cut corners or not concentrate on the work. For example, it is assumed that the operator falsely reports his abilities. For example, it is assumed that an operator different from the selected operator performs remote operation.

This technology has been developed in view of this situation, and is intended to allow remote control of robots to be performed appropriately.

An information processing apparatus according to a first aspect of the present technology includes an operator monitoring unit that monitors the operational aptitude of an operator; and an intermediary unit for controlling intermediation of remote operation between the operation terminal and the robot based on the operation suitability and the result of monitoring the work situation.

An information processing method according to a first aspect of the present technology monitors the operation suitability of an operator and the work status, which is the status of work performed by the operator remotely controlling a robot using an operation terminal. and controls intermediation of remote operation between the operation terminal and the robot based on the result of monitoring the operation suitability and the work status.

An information processing system according to a second aspect of the present technology includes a robot, an operation terminal used by an operator to remotely operate the robot, and an information processing device that mediates the remote operation between the operation terminal and the robot. The information processing device includes an operator monitoring unit that monitors the operation suitability of the operator, and the status of work performed by the operator remotely controlling the robot using the operation terminal. A work monitoring unit that monitors a work situation, and an intermediary unit that controls mediation of remote operation between the operation terminal and the robot based on the operational suitability and the result of monitoring the work situation.

In the first or second aspect of the present technology, an operator's operational suitability and a work status, which is a status of work performed by the operator remotely controlling a robot using an operation terminal, are monitored. Based on the result of monitoring the operation suitability and the work status, the intermediation of remote operation between the operation terminal and the robot is controlled.

It is a figure which shows an example of the vision aimed at by applying this technique. 1 is a schematic diagram of a first embodiment of an information processing system to which the present technology is applied; FIG. 1 is a block diagram showing a functional configuration example of a first embodiment of an information processing system to which the present technology is applied; FIG. 3 is a block diagram showing a configuration example of functions of a management server; FIG. It is a figure which shows the structural example of work condition DB. FIG. 4 is a sequence diagram for explaining the outline of the first embodiment of processing of the information processing system; 4 is a flowchart for explaining a first embodiment of matching processing; It is a figure which shows the structural example of operator DB. 4 is a flowchart for explaining the details of operator selection processing; 4 is a flowchart for explaining processing of a management server in remote control processing; 4 is a flowchart for explaining processing of a management server in remote control processing; 4 is a flowchart for explaining the details of operator authentication processing; 9 is a flowchart for explaining the details of remote operation mediation processing; FIG. 11 is a flowchart for explaining the details of operation eligibility monitoring processing; FIG. 4 is a flowchart for explaining the details of operator status monitoring processing; 6 is a flowchart for explaining the details of work monitoring processing; FIG. 10 is a flowchart for explaining the details of skill level/prohibited act monitoring processing; FIG. 9 is a flowchart for explaining the details of operating environment monitoring processing; 4 is a flowchart for explaining the details of work environment monitoring processing; 4 is a flowchart for explaining processing of an operation terminal in remote control processing; 9 is a flowchart for explaining details of authentication information transmission processing; 9 is a flowchart for explaining the details of work status presentation processing; 9 is a flowchart for explaining the details of an operator status information transmission process; FIG. 11 is a flowchart for explaining details of an operating environment information transmission process; FIG. 4 is a flowchart for explaining robot processing in remote control processing. 9 is a flowchart for explaining the details of work status information transmission processing; FIG. 10 is a flowchart for explaining the details of work execution processing; FIG. 9 is a flowchart for explaining the details of work environment information transmission processing; 6 is a flowchart for explaining post-work processing; FIG. 11 is a sequence diagram for explaining an outline of a second embodiment of processing of the information processing system; FIG. 11 is a flow chart for explaining a second embodiment of matching processing; FIG. 4 is a flowchart for explaining the details of operator selection processing; FIG. 12 is a sequence diagram for explaining an outline of a third embodiment of processing of the information processing system; FIG. 11 is a flow chart for explaining a third embodiment of matching processing; FIG. FIG. 7 is a block diagram showing a functional configuration example of a second embodiment of an information processing system to which the present technology is applied; FIG. 7 is a block diagram showing a functional configuration example of a second embodiment of an information processing system to which the present technology is applied; FIG. 4 is a diagram showing a configuration example of a skill table for remote control of a surgical robot; It is a figure which shows the structural example of a possession skill table. FIG. 10 is a diagram showing a configuration example of a skill table when remote-controlling an industrial robot; FIG. 4 is a diagram showing a configuration example of a possessed skill table for remote control of a vehicle; It is a figure which shows the structural example of a remuneration table. It is a figure for demonstrating the modification of a matching process. It is a block diagram which shows the structural example of a computer.

Embodiments for implementing the present technology will be described below. The explanation is given in the following order.
1. An example of the vision we want to achieve 2 . First embodiment 3. Second embodiment 4. Third Embodiment 5. Application example 6. Modification 7. others

<<1. An example of the vision we want to achieve>>
The decrease in the productive working population has become a common social issue, and in some areas the situation is becoming serious, such as a decrease in the working population, a shortage of workers, and a shift away from dangerous work. In particular, it is extremely difficult to remote essential workers and skill workers who are involved in actual work on site. Therefore, securing workers and improving productivity through new work styles have become issues. As a solution to this problem, for example, utilization of robot operation by remote control is proposed.

Figure 1 shows an example of a vision that is aimed at by applying this technology. Specifically, FIG. 1 shows an embodiment of a service platform that connects a client (enterprise) and a remote operator (remote worker). The service platform includes a remote robot operation system and a remote operation matching system.

The remote robot operation system is a system that enables remote operation, remote monitoring, equipment management, etc. of each robot, and ensures work quality.

The remote operation matching system is a system that matches operators (remote workers) and clients (business operators) by matching their skills, operating environment, schedule, etc. For example, the remote operation matching system performs matching of operators who match the work request of the client, user management that manages the operator's skills, operating environment and work content, and work evaluation for evaluating the work of the operator. conduct. For example, the remote operation matching system schedules operators so that they can effectively use their free time.

The operator is a worker who remotely controls the robot. For example, skilled workers with high skills such as skilled workers and specialists, part-time workers who use their spare time to provide labor, and senior workers who have physical limitations due to age or illness but are able to provide labor. An example is an operator. These operators remotely operate robots to perform various tasks in the fields of healthcare, construction/civil engineering, infrastructure, agriculture, logistics, and manufacturing/processing, for example, via the service platform.

With this service platform, "3R technology" that remotely realizes reality and real-time technology will be realized, and a new work style will be realized. For example, by providing a new work style, it will be possible to secure workers, improve productivity, and realize a new remote society where everyone who wants to work can work. In this way, this service platform contributes to solving social problems caused by the declining working-age population.

<<2. First Embodiment>>
Next, a first embodiment of the present technology will be described with reference to FIGS. 2 to 34. FIG.

<Configuration example of information processing system 1>
First, a configuration example of an information processing system 1 to which the present technology is applied will be described with reference to FIGS. 2 to 5. FIG.

FIG. 2 is a schematic diagram of the information processing system 1 according to the first embodiment of the present technology.

The information processing system 1 includes communication terminals 11-1 to 11-p, operation terminals 12-1 to 12-q, client terminals 13-1 to 13-m, robots 14-1 to 13-m. 14-n and a management server 15. FIG.

Hereinafter, the communication terminals 11-1 to 11-p are simply referred to as communication terminals 11 when there is no need to distinguish them individually. Hereinafter, the operation terminals 12-1 to 12-q are simply referred to as operation terminals 12 when there is no need to distinguish them individually. Hereinafter, the client terminals 13-1 to 13-m are simply referred to as the client terminal 13 when there is no need to distinguish them individually. Hereinafter, the robots 14-1 to 14-n are simply referred to as robots 14 when there is no need to distinguish them individually.

communication terminals 11-1 to 11-p, operation terminals 12-1 to 12-q, client terminals 13-1 to 13-m, robots 14-1 to 14-n, and The management servers 15 are connected via a network 21 and can communicate with each other.

Each operation terminal 12 and each robot 14 may be connected to the network 21 via an intermediate device. The intermediate device is, for example, an IP converter with communication capabilities.

The information processing system 1 is a system that implements the service platform described above. The information processing system 1 matches, via the management server 15, the operation terminal 12 for remote operation and the robot 14 to be remotely operated. That is, the information processing system 1 implements the above-described remote control matching system. Note that matching between the operation terminal 12 and the robot 14 includes not only matching between the terminals but also matching between the operator performing the remote operation and the robot 14 or the requester requesting the work using the robot 14 . The information processing system 1 also realizes remote control of each robot 14 by each operation terminal 12 via the management server 15 . That is, the information processing system 1 implements the remote robot operation system described above.

In this embodiment, the information processing system 1 realizes matching and remote control of the robot 14 by the operation terminal 12, but the matching system and the remote control system may be separate entities.

The communication terminal 11 is an information processing device used by the operator to accept work. The communication terminal 11 is, for example, a smart phone, a PC (Personal Computer), a tablet terminal, or the like.

The operation terminal 12 is an information processing device capable of remotely controlling the robot 14 . Here, an example is shown in which the operation terminal 12 is a game device (operation terminal 12-1) such as PlayStation 5 (registered trademark), a smartphone (operation terminal 12-2), or a surgical robot console (operation terminal 12-q). It is Alternatively, the operation terminal 12 may be, for example, a PC, a tablet terminal, a dedicated terminal, or the like.

Note that one operator may own multiple communication terminals 11 or multiple operation terminals 12 . Moreover, a plurality of operators may share the communication terminal 11 and the operation terminal 12 . Furthermore, the operating terminal 12 is not necessarily owned by the operator. For example, it is conceivable that the operator borrows the operation terminal 12, or that the operator has the operation terminal 12 owned by a person affiliated with the operator.

Also, for example, the operator can use the operation terminal 12 without using the communication terminal 11 to accept work or the like. That is, the communication terminal 11 and the operation terminal 12 may be one terminal.

The requester terminal 13 is an information processing terminal used by the requester to request work. The client terminal 13 is, for example, a smart phone, a PC, a tablet terminal, or the like.

The robot 14 is a robot that has at least one or more movable parts and can be remotely controlled. That is, the robot 14 is a robot whose movable parts can be operated by remote control.

The robots 14 are, for example, an entertainment robot (robot 14-1), a general-purpose robot (robot 14-2), a surgical robot (robot 14-3), and a picking robot (cooking robot, robot 14-q). In addition, the robot 14 may be, for example, a rescue robot, a cleaning robot, a wall work robot, a security robot, a guidance robot, a wheelchair robot, a sushi robot, a nursing care robot, a medical robot, or the like. Also, the robot 14 may be, for example, a remotely controllable moving object such as a vehicle or a drone.

It should be noted that the robot 14 does not necessarily need to perform all actions by remote control. For example, robot 14 may be a semi-autonomous robot that operates autonomously based on operator input. Specifically, for example, the robot 14 may be a semi-autonomous robot capable of autonomously picking an object designated by the operator.

It should be noted that one client may own multiple client terminals 13 or multiple robots 14 . Also, a plurality of clients may share the client terminal 13 and the robot 14 . Moreover, the robot 14 is not necessarily owned by the client. For example, it is conceivable that the client borrows the robot 14 or that a related party who is affiliated with the client owns the robot 14 .

Also, for example, it is possible for the requester to request work using the robot 14 without using the requester terminal 13 .

The client is not necessarily a natural person, and may be an organization such as a corporation.

The management server 15 is implemented by, for example, one computer or multiple computers cooperating. The management server 15 executes matching processing between the operator or the operating terminal 12 and the client or the robot 14, mediation processing for remote control between the operating terminal 12 and the robot 14, and the like. For example, matching processing, remote control mediation processing, and the like are realized by executing predetermined applications in cloud services such as AWS (Amazon Web Services, registered trademark) and Azure (registered trademark).

The network 21 is a network capable of data communication. For example, the network 21 is configured by the Internet, LAN (Local Area Network), WAN (Wide Area Network), and the like.

FIG. 3 shows an example configuration of the functions of the communication terminal 11, the operation terminal 12, the client terminal 13, and the robot 14 of the information processing system 1. As shown in FIG. 3, only one communication terminal 11, one operation terminal 12, one client terminal 13, and one robot 14 are shown, and the illustration of the network 21 is omitted for the sake of clarity. .

The communication terminal 11 includes an input unit 101, a detection unit 102, a control unit 103, an output unit 104, a communication unit 105, and a storage unit 106.

The input unit 101 includes, for example, various input devices and operation devices. The input unit 101 is used, for example, to operate the communication terminal 11 and to input commands and data to the communication terminal 11 .

The detection unit 102 includes, for example, various sensors, detects a user (eg, an operator), surrounding conditions, etc., and outputs sensor data indicating the detection results. For example, the detection unit 102 includes a camera for capturing a user's image (still image, moving image). For example, the detection unit 102 includes sensors for detecting the position and movement of the communication terminal 11, such as a GNSS (Global Navigation Satellite System) receiver, an acceleration sensor, an angular velocity sensor, and the like. For example, the detection unit 102 includes a sensor that detects biometric information used for user authentication (eg, fingerprint, iris, voice, etc.) and biometric information indicating the user's condition (eg, perspiration amount, heart rate, blood pressure, etc.). Prepare.

The control unit 103 includes a processor such as a CPU (Central Processing Unit). The control unit 103 executes control of the communication terminal 11 and various types of information processing.

The output unit 104 includes, for example, various output devices (eg, display device, speaker, haptics device, etc.). The output unit 104 outputs, for example, various types of information (eg, visual information, auditory information, tactile information, etc.).

The communication unit 105 includes various communication devices, for example. The communication unit 105 communicates with the other communication terminal 11, the operation terminal 12, the client terminal 13, the robot 14, the management server 15, etc. via the network 21 or not via the network 21. FIG.

The storage unit 106 includes, for example, a non-volatile memory and a volatile memory, and stores programs and data necessary for processing of the communication terminal 11.

The operation terminal 12 includes an input unit 121, a detection unit 122, a control unit 123, an output unit 124, a communication unit 125, and a storage unit 126.

The detection unit 122, the output unit 124, the communication unit 125, and the storage unit 126 of the operation terminal 12 are configured similarly to the detection unit 102, the output unit 104, the communication unit 105, and the storage unit 106 of the communication terminal 11, respectively. be.

The input unit 121 includes, for example, various input devices and operation devices. The input unit 121 is used to operate the operation terminal 12 and input commands, data, and the like to the operation terminal 12 . Also, the input unit 121 is used for remote control of the robot 14 .

The control unit 123 includes, for example, a processor such as a CPU. The control unit 103 executes control of the operation terminal 12 and various types of information processing. For example, the control unit 123 executes processing related to remote control of the robot 14 .

The client terminal 13 includes an input unit 141, a detection unit 142, a control unit 143, an output unit 144, a communication unit 145, and a storage unit 146.

The input unit 141 to storage unit 146 of the client terminal 13 are, for example, configured similarly to the input unit 101 to storage unit 106 of the communication terminal 11, respectively.

The robot 14 includes an input section 161 , a detection section 162 , a control section 163 , a movable section 164 , an output section 165 , a communication section 166 and a storage section 167 .

The input unit 161, the output unit 165, the communication unit 166, and the storage unit 167 of the robot 14 are configured similarly to the input unit 101, the output unit 104, the communication unit 105, and the storage unit 106 of the communication terminal 11, respectively. be done.

The detection unit 162 is equipped with various sensors, for example, detects the robot 14 and surrounding conditions, etc., and outputs sensor data indicating the detection results. For example, the detection unit 162 includes a camera for photographing the work site of the robot 14 . For example, the detection unit 162 includes sensors for detecting the position and movement of the robot 14, such as a GNSS receiver, an acceleration sensor, an angular velocity sensor, and the like.

The control unit 163 includes a processor such as a CPU, for example. The control unit 103 executes control of the robot 14 and various types of information processing. Also, the control unit 163 executes processing corresponding to remote operation by the operation terminal 12 .

The movable part 164 is a part that can be moved by the robot 14 to perform various tasks. The operation and configuration of the movable portion 164 are not particularly limited, and differ depending on the application of each robot 14 and the like.

FIG. 4 shows a functional configuration example of the management server 15 . The management server 15 includes an input unit 201 , a control unit 202 , an output unit 203 , a communication unit 204 and a storage unit 205 .

The input unit 201, the output unit 203, and the communication unit 204 are configured similarly to the input unit 101, the output unit 104, and the communication unit 105, respectively, of the communication terminal 11 in FIG.

The control unit 202 includes a matching unit 211, an intermediary unit 212, a monitoring unit 213, an evaluation unit 214, a learning unit 215, and an information processing unit 216.

The matching unit 211 executes matching processing between the operator or the operating terminal 12 and the client or the robot 14.

The mediation unit 212 executes mediation processing for remote control between the operation terminal 12 and the robot 14 .

The monitoring unit 213 includes an operator monitoring unit 221 and a work monitoring unit 222 .

The operator monitoring unit 221 monitors the operation eligibility of each operator based on the information from the operation terminal 12 and the robot 14. That is, the operator monitoring unit 221 monitors whether or not a qualified operator is operating the work requested by the client.

The work monitoring unit 222 monitors the status of work performed by each operator using the operation terminal 12 and the robot 14 based on information from the operation terminal 12 and the robot 14 .

The evaluation unit 214 evaluates the operator and the work performed by the operator's remote control. For example, the evaluation unit 214 evaluates the work content of each operator based on the results of monitoring by the operator monitoring unit 221 and the work monitoring unit 222, and evaluates the work content of each operator based on the evaluation of the work content of each operator. Update the ability information of For example, the evaluation unit 214 updates the operator's evaluation score (described later) based on the client's evaluation. For example, the evaluation unit 214 determines the reward to be given to the operator based on the operator's evaluation of the work content and the requester's evaluation of the work.

The learning unit 215 executes various learning processes. For example, the learning unit 215 executes learning processing for automatic control of each robot 14 . Specifically, the learning unit 215 generates learning data and stores it in the storage unit 205 . The learning unit 215 uses learning data accumulated in the storage unit 205 to generate a learning model for automatically controlling each robot 14 .

The information processing unit 216 executes various types of information processing necessary for the processing of the management server 15.

The storage unit 205 includes, for example, a non-volatile memory and a volatile memory, and stores programs and data necessary for the processing of the management server 15. For example, the storage unit 205 accumulates data regarding work performed by each robot 14 . For example, the storage unit 205 accumulates a work condition DB (database), a work request DB, and the like, which will be described later. The work condition DB is a DB that stores data related to conditions required for remote control of each work. The work request DB is a DB that stores data related to work requested by a requester.

For example, the storage unit 205 accumulates data related to operators and clients who are users of the information processing system 1 . The storage unit 205 stores, for example, an operator DB, a client DB, a skill table, a possessed skill table, a remuneration table, and the like, which will be described later. The operator DB is a DB that stores data on each operator. The client DB is a DB that stores data regarding each client. The skill table is a table that defines skills required for each task. The possessed skill table is a table showing the skills possessed by each operator. The reward table is a table showing rewards given to each operator.

For example, the storage unit 205 accumulates learning data used for learning processing for automatic control of each robot 14 .

<Data configuration example of work condition DB>
FIG. 5 shows a configuration example of data of the work condition DB accumulated in the storage unit 205 of the management server 15. As shown in FIG. The work condition DB is a DB that stores data related to conditions required for remote control of each work.

The work condition DB includes work content, communication volume, communication speed, Ping value, allowable delay, operating terminal, and display.

The work content shows an overview of each work. Here, the work of controlling the robot 14 to be remotely operated is shown as an example.

The amount of communication indicates the amount of communication of the operation terminal 12 required to execute each task. For example, when the operation terminal 12 is a smart phone, the communication volume may be restricted due to a contract with a communication company or the like. Specifically, when the amount of communication on a smartphone exceeds the monthly capacity limit, the communication speed is restricted and may drop sharply. On the other hand, as a guideline for preventing the communication volume of the operation terminal 12 from exceeding the limited capacity, the communication volume is provided as a condition. As a result, it is possible to prevent the amount of communication of the operation terminal 12 from exceeding the limited capacity during remote operation, thereby suppressing a sudden drop in the communication speed, thereby improving the stability of the communication of the operation terminal 12 .

The communication speed indicates the minimum value of the communication speed of the operation terminal 12 required for executing each task.

The Ping value indicates the allowable value (maximum value) of the Ping value between the operation terminal 12 and the robot 14 required for each task. A Ping value is provided as a condition for measuring and evaluating the time required for data transmission and reception between the operation terminal 12 and the robot 14 .

The allowable delay indicates the allowable value (maximum value) of the communication delay time (latency) between the operation terminal 12 and the robot 14 required for each task. The delay time is, for example, the time required for the robot 14 to receive the control signal for controlling the robot 14 via the network 21 and the management server 15 after the control signal for controlling the robot 14 is transmitted from the operation terminal 12 .

The operation terminal indicates the type of operation terminal 12 that can be used for remote control of each task. For example, the type of operation terminal is provided as a condition because the input device and performance required for the operation terminal 12 differ depending on the type and specifications of the robot 14 .

The display shows the display resolution required for remote control of each task. For example, the required resolution of the display differs depending on the accuracy and detail of the work performed by the robot 14, so the resolution of the display is provided as a condition.

<First Embodiment of Processing of Information Processing System 1>
Next, a first embodiment of processing of the information processing system 1 will be described with reference to FIGS. 6 to 29. FIG.

<Overview of First Embodiment of Processing of Information Processing System 1>
First, the outline of the first embodiment of the processing of the information processing system 1 will be described with reference to the sequence diagram of FIG. The sequence diagram of FIG. 6 shows a sequence diagram among the operator, the management server 15, the client, and the robot 14. In FIG. The operator uses the communication terminal 11 and the operation terminal 12 to execute the processing shown in this sequence diagram. The requester uses the requester terminal 13 to execute the processing shown in this sequence diagram.

First, matching processing between the operator (operation terminal 12) and the client (robot 14) is executed.

Specifically, for example, the client sends client information about the client to the management server 15. A specific example of the client information will be described later.

The operator transmits operator information about the operator to the management server 15. A specific example of operator information will be described later.

In response, the management server 15 registers client information and operator information.

Although this diagram shows the flow in which the requester information and the operator information are registered at the same time, the registration of the requester information and the operator information is performed at any timing.

Next, the requester requests the management server 15 to select an operator who will perform the desired work.

The management server 15 selects an operator to request work based on the operator's ability and operating environment, the content of the work, etc., and requests the selected operator to perform the work.

The operator determines whether or not to accept the work, and notifies the management server 15 of the determination result.

When an operator accepts a task, the management server 15 recommends the operator to the client by notifying the client of information on the operator.

It should be noted that, if the operator does not accept the work, the management server 15 starts over from selecting the operator.

Based on the information about the recommended operator, the requester decides whether or not to approve the operator, that is, whether or not to actually request the operator to perform the work. The requester notifies the management server 15 of the operator's approval result.

The management server 15 notifies the operator of the operator's approval result by the requester.

It should be noted that, if the client does not approve the operator, the management server 15 starts over from selecting the operator.

Next, the operator uses the operation terminal 12 to remotely control the robot 14, thereby executing remote control processing for executing the work requested by the requester.

Specifically, the operator transmits the authentication information to the management server 15. At this time, when the operator performs work that requires a qualification, the operator transmits qualification information proving possession of the qualification to the management server 15 .

When the operator is successfully authenticated, the management server 15 mediates connection between the operation terminal 12 operated by the operator and the robot 14 executing the work.

The operator uses the operation terminal 12 to transmit an operation signal for remotely operating the robot 14 to the management server 15 .

The management server 15 transmits the operation signal received from the operation terminal 12 to the robot 14 .

The robot 14 performs the work under the remote control of the operator based on the received operation signal.

The operator (operation terminal 12) constantly transmits monitoring information for monitoring the operator and the operating environment to the management server 15 during execution of the operation.

The robot 14 constantly transmits to the management server 15 monitoring information for monitoring the work by the robot 14 and the work environment during execution of the work.

The management server 15 monitors the operation suitability of the operator and the work status based on the monitoring information received from the operation terminal 12 and the robot 14 . Based on the monitoring results, the management server 15 supports the operation or work, issues a warning to the operator, or suspends the intermediation of the remote operation, if necessary.

Next, post-work processing is executed.

For example, the client evaluates the operator and the work performed by the operator by remote control, and notifies the management server 15 of the evaluation result.

The management server 15 determines the remuneration to be paid to the operator based on the client's evaluation, etc., and notifies the operator of remuneration information regarding the remuneration.

The operator evaluates the client and the work requested by the client, and notifies the management server 15 of the evaluation results.

<Matching process>
Next, details of the matching process executed by the information processing system 1 will be described with reference to the flowchart of FIG. 7 .

In step S101, the client terminal 13 transmits client information. Specifically, the control unit 143 generates client information based on the information or the like input by the client via the input unit 141 .

Client information includes, for example, attribute information and work environment information.

Attribute information is information about the client's attributes. For example, the client information includes personal information such as the name of the client. When the client is an organization such as a corporation other than a natural person, for example, the attribute information includes information about the organization such as the name of the organization. Also, the attribute information includes, for example, the client's bank account information.

The work environment information is information related to the environment in which the work for which the requester requests remote operation is performed. For example, work environment information includes robot information and communication status information.

The robot information is information about the robot 14 that performs the task for which the remote operation is requested. The robot information includes, for example, one or more of identification information, type, specification, position information (for example, installation position, etc.), operation status, and error occurrence status of the robot 14 .

The identification information of the robot 14 is information for identifying the robot 14. The identification information includes, for example, one or more of the serial number of the robot 14 and MAC (Media Access Control) address.

The specifications of the robot 14 include, for example, the functions, performance, and other specifications of the robot 14.

The operating status of the robot 14 includes, for example, information regarding the days of the week and time zones in which the robot 14 can operate.

The error occurrence status of the robot 14 includes, for example, the content and occurrence rate of errors that may occur.

The communication status information is, for example, information about the communication status between the robot 14 and the management server 15. For example, the communication status information includes one or more of communication speed, delay time, communication capacity, communication band, packet loss rate, etc. when the robot 14 is connected to the management server 15 via the network 21 .

The control unit 143 transmits the client information to the management server 15 via the communication unit 145 and requests registration of the client information.

In step S111, the communication terminal 11 transmits operator information. Specifically, the control unit 103 generates operator information based on information input by the operator via the input unit 101, sensor data obtained by the detection unit 102, and the like.

The operator information is information about the operator or the operating terminal 12. The operator information is information including, for example, attribute information, authentication information, and operating environment information.

Attribute information is information about the attributes of the operator. Attribute information is, for example, information including personal information and ability information.

　Personal information is information about the operator. The personal information is information including, for example, the operator's name, sex, age, nationality, occupation, affiliation, and the like.

The ability information is information about the ability of the operator to perform the work. The ability information includes, for example, the operator's possessed qualifications, possessed skills, and the skill level of the skills possessed by the operator. For the skill level, for example, the result of a predetermined skill test, years of experience of the skill, or the like can be used.

　Authentication information is information used to authenticate the operator. Authentication information includes, for example, at least one of biometric information and a password.

The biometric information is information used for biometric authentication of the operator, and is extracted from sensor data acquired by the detection unit 102, for example. For example, the biometric information includes facial image data obtained by photographing the operator's face, fingerprint image data obtained by photographing the fingerprint, vein image data obtained by photographing the vein, iris image data obtained by photographing the iris, and voice data obtained by recording the operator's voice. etc. is assumed.

For example, instead of the operator's face image data, it is possible to use a copy image of a personal certificate (eg, passport, driver's license, etc.) containing the operator's face photo.

Also, in consideration of security, the control unit 103 may anonymize the biometric information and transmit it to the management server 15 . For example, the control unit 103 may use, as biometric information, feature amounts extracted from image data or audio data representing the biometric information of the operator.

An example of using biometric information as authentication information will be described below.

The operating environment information is information related to the environment in which the operator performs remote operation. For example, operating environment information includes operating terminal information and communication status information.

The operating terminal information is information related to the operating terminal 12 used by the operator. The operation terminal information includes, for example, one or more of identification information, type, specification, state, position information, and connection status with the communication terminal 11 of the operation terminal 12 .

The identification information of the operation terminal 12 is information for identifying the operation terminal 12 . The identification information includes, for example, one or more of the serial number, MAC address, etc. of the operation terminal 12 .

The specifications of the operation terminal 12 include, for example, the functions, performance, and other specifications of the operation terminal 12.

The state of the operation terminal 12 includes, for example, whether or not an abnormality has occurred in the operation terminal 12.

The communication status information is, for example, information about the communication status between the operation terminal 12 and the management server 15. For example, the communication status information includes one or more of communication speed, delay time, communication capacity, communication band, packet loss rate, etc. when the operation terminal 12 is connected to the management server 15 via the network 21 .

The control unit 103 transmits the operator information to the management server 15 via the communication unit 105 and requests registration of the operator information.

On the other hand, in step S121, the management server 15 registers client information and operator information.

Specifically, the communication unit 204 receives client information from the client terminal 13 and operator information from the communication terminal 11 . The information processing unit 216 adds the received client information to the client DB accumulated in the storage unit 205 . The information processing unit 216 adds the received operator information to the operator DB accumulated in the storage unit 205 .

FIG. 8 shows a part of the configuration example of the data of the operator DB. The operator DB includes an operator ID, ability information, and operating environment information for each operator.

The operator ID is an ID for identifying each operator.

The ability information includes, for example, whether or not each robot 14 has each qualification necessary for remote control.

The operating environment information includes, for example, the type of operating terminal owned by each operator.

Note that the operator DB also includes, for example, each operator's evaluation score. The evaluation score is, for example, a score indicating the evaluation of each operator by the client who requested the work. An operator with a higher client evaluation has a higher evaluation score, and an operator with a lower client evaluation has a lower evaluation score.

The ability information and operating environment information of each operator may be stored in one DB as shown in FIG. 8, or may be stored in different DBs.

When the ability information and operating environment information are stored in one DB, the information is unified, so the access speed to each operator's information increases, but the risk of information leakage increases. On the other hand, when the ability information and the operating environment information are stored in different DBs, the information is distributed, so the speed of each operator's access to the information slows down, while the risk of information leakage decreases.

In order to simplify the explanation, the management server 15 shows an example in which client information and operator information are registered at the same time. transmits client information and operator information at arbitrary timings. Each time the management server 15 receives client information or operator information, it registers each piece of information.

In step S102, the client terminal 13 requests selection of an operator. Specifically, the control unit 143 generates work request information based on information or the like input by the requester via the input unit 141 .

The work request information includes, for example, the identification information of the robot 14 to be operated, the requested work content and work time, delivery date, remuneration, conditions necessary for the operator, requested work quality, and the like.

The conditions necessary for the operator include, for example, the ability required for the operator. The abilities required for the operator are indicated by one or more of qualifications, skills, and skill levels, for example.

The conditions required for the operator may include, for example, the operator's attributes such as gender, age, nationality, occupation, etc., as necessary.

The required quality of work is indicated, for example, by one or more of work speed, work accuracy, and quality of deliverables (eg, products, etc.) obtained by work.

The control unit 143 transmits work request information to the management server 15 via the communication unit 145 and requests selection of an operator.

On the other hand, in step S122, the management server 15 executes operator selection processing.

Here, the details of the operator selection process will be described with reference to the flowchart of FIG.

In step S151, the management server 15 generates selection conditions based on the content of the work request.

Specifically, the communication unit 204 receives work request information from the client terminal 13 .

The matching unit 211 generates operator selection conditions based on the received work request information and the work condition DB ( FIG. 7 ) accumulated in the storage unit 205 . The operator selection conditions include, for example, operator-related conditions and operational environment conditions.

Conditions related to the operator include, for example, conditions related to the ability of the operator. The conditions regarding the operator's ability include, for example, at least one of the operator's possessed qualifications, possessed skills, and skill level.

The operating environment conditions include the operating environment required to remotely operate the robot 14 requested by the client to perform the work. Conditions related to the operating environment are set based on one or more of, for example, the type of robot 14, the specifications of the robot 14, and the details of the work performed by the robot 14. FIG.

More specifically, the operating environment conditions include, for example, at least one of conditions relating to the operating terminal 12 used by the operator and conditions relating to the communication status of the operating terminal 12 . The conditions regarding the operating terminal 12 include, for example, the types of usable operating terminals 12 and specifications (eg, functions, performance, etc.). The condition regarding the communication status of the operation terminal 12 is, for example, the communication speed, the allowable delay time, the communication capacity, the communication band, and the allowable packet loss rate when the operation terminal 12 is connected to the management server 15 via the network 21. At least one.

The selection condition generation process may be executed using, for example, a machine learning model obtained by machine learning such as a neural network, or may be executed based on the conditions input by the requester. good too.

In step S152, the matching unit 211 selects an operator to be judged. Specifically, the matching unit 211 selects one of the operators registered in the operator DB who has not yet determined whether or not the selection condition is satisfied.

In step S153, the matching unit 211 determines whether the operator's ability and operating environment satisfy the selection conditions.

For example, if there is an essential condition among the selection conditions, the matching unit 211 determines whether the ability and operating environment of the selected operator satisfy the essential condition. For example, qualifications, the type of the operation terminal 12 used for remote operation, and the like are set as essential conditions.

If the operator satisfies the required conditions or if the required conditions do not exist, the matching unit 211 calculates the degree of matching between the operator's ability and operating environment and the selection conditions generated in step S151.

At this time, the matching unit 211 may, for example, set priorities in the selection conditions and calculate the degree of matching according to the priorities. For example, when calculating the degree of matching, the matching unit 211 increases the weight of the degree of matching for selection conditions with high priority, and decreases the weight of the degree of matching with respect to selection conditions with low priority.

If the calculated matching degree is equal to or greater than the predetermined threshold, the matching unit 211 determines that the selection condition is satisfied, and the process proceeds to step S154.

In step S154, the matching unit 211 puts the selected operator into the candidates for the operator to request the work.

After that, the process proceeds to step S156.

On the other hand, in step S153, the matching unit 211 determines that the selection condition is not satisfied if the matching degree of the operator is less than the predetermined threshold value, or if the operator does not satisfy the essential condition, and the process proceeds to step S153. Proceed to S155.

In step S155, the matching unit 211 excludes the selected operator from candidates for the operator to request the work.

After that, the process proceeds to step S156.

In step S156, the matching unit 211 determines whether or not all operators have been determined. If it is determined that all operators have not yet been determined, the process returns to step S152.

After that, in step S156, the processing of steps S152 to S156 is repeatedly executed until it is determined that all operators have been determined. As a result, it is determined whether or not all operators registered in the operator DB satisfy the selection condition.

It should be noted that it is not necessary to determine all operators, and the matching unit 211 may narrow down the number of operators to be determined. For example, the matching unit 211 may narrow down the number of operators to be judged based on conditions such as age, or may narrow down the number of operators to be judged by randomly selecting operators. .

On the other hand, if it is determined in step S156 that all operators have been determined, the process proceeds to step S157.

In step S157, the matching unit 211 determines the operator to whom the work is requested. For example, the matching unit 211 determines the operator with the highest matching degree among the operator candidates extracted using the selection conditions as the operator to request the work.

It should be noted that, for example, if there are multiple operators with the highest degree of matching, the operator is selected based on the distance between the operating terminal 12 to be used and the robot 14 to be remotely operated. For example, an operator using an operation terminal 12 located closer to the robot 14 to be remotely operated is preferentially selected. This is because the closer the position of the operating terminal 12 is to the position of the robot 14, the more likely it is that the communication time between the operating terminal 12 and the robot 14 will be shorter.

If there is no operator who satisfies the selection conditions, for example, the matching unit 211 notifies the client terminal 13 that there is no operator who satisfies the selection conditions.

Returning to FIG. 7, in step S123, the matching unit 211 asks the selected operator to accept the work. Specifically, the matching unit 211 transmits work request information to the communication terminal 11 of the selected operator via the communication unit 204, and inquires whether or not the work can be accepted.

On the other hand, in step S112, the communication terminal 11 presents a work request. Specifically, the communication unit 105 receives work request information from the management server 15 . The output unit 104 presents work request information to the operator under the control of the control unit 103 .

In step S113, the communication terminal 11 acquires and notifies whether or not the work can be accepted.

Specifically, the operator determines whether or not to accept the presented work, and uses the input unit 101 to input the result of the determination into the communication terminal 11 .

When the control unit 103 determines that the operator will accept the requested work, the control unit 103 extracts the operator's authentication information from the sensor data from the detection unit 122, for example. The authentication information extracted at this time is, for example, the same type of authentication information as the authentication information registered in the operator DB held by the management server 15, or information based on the authentication information of the operator DB. . For example, when the authentication information of the operator DB is the feature amount of the face image data, the information on which the authentication information of the operator DB is based corresponds to the face image data.

The control unit 103 transmits the authentication information to the management server 15 via the communication unit 105 and notifies that the operator accepts the work.

On the other hand, when the control unit 103 determines that the operator will not accept the requested work, it notifies the management server 15 via the communication unit 105 that the operator will not accept the work.

On the other hand, in step S124, the management server 15 authenticates the operator. Specifically, when the communication terminal 11 notifies that the operator accepts the work, the communication unit 204 receives the authentication information transmitted at the same time from the communication terminal 11 . The matching unit 211 performs personal authentication of the operator based on the received authentication information and the operator authentication information registered in the operator DB. If the personal authentication of the operator is successful, that is, if the operator selected by the matching unit 211 matches the operator who accepted the task, the process proceeds to step S125.

It should be noted that, for example, when the communication terminal 11 notifies that the operator does not accept the work, or when the operator's personal authentication fails, the processes after step S122 are executed again. As a result, a new operator is selected, and the selected operator is asked to accept the work.

In step S125, the matching unit 211 transmits information about the operator who accepted the task. Specifically, the matching unit 211 generates information (hereinafter referred to as mandated operator information) including information on the ability and operating environment of the operator who has accepted the work. The communication unit 204 transmits the mandated operator information to the client terminal 13 .

On the other hand, in step S103, the client terminal 13 presents information about the operator who accepted the work. Specifically, the communication unit 145 receives mandated operator information from the management server 15 .

Under the control of the control unit 143, the output unit 144 presents information about the operator who accepted the work to the requester based on the accepted operator information. As a result, the operator selected by the management server 15 is recommended to the client.

In step S104, the client terminal 13 acquires and transmits the approval result for the operator.

Specifically, the requester determines whether or not to approve the operator based on the presented information about the operator, and uses the input unit 141 to input the determined result to the requester terminal 13. .

The communication unit 145 notifies the management server 15 of the approval result of the requester's operator.

In response to this, in step S126, the management server 15 notifies the operator of the approval result. Specifically, the communication unit 204 notifies the communication terminal 11 of the approval result for the operator notified from the client terminal 13 .

It should be noted that, for example, if the requester does not approve the operator, the processes from step S122 onward are executed again. As a result, a new operator is selected, and the selected operator is asked to accept the work.

On the other hand, in step S114, the communication terminal 11 presents the approval result to the operator. Specifically, under the control of the control unit 103 , the output unit 104 presents the requester with the approval result for the operator notified from the management server 15 .

<Remote control processing>
Next, remote control processing executed by the information processing system 1 will be described with reference to FIGS. 10 to 28. FIG.

<Processing of management server 15>
First, the processing of the management server 15 in the remote control processing will be described with reference to the flowcharts of FIGS. 10 and 11. FIG.

In step S201, the management server 15 requests transmission of the operator's authentication information. Specifically, the intermediary unit 212 requests, via the communication unit 204, the operation terminal 12 used by the operator to transmit the authentication information of the operator who will execute the requested work.

In response to this, the operation terminal 12 transmits the operator's authentication information to the management server 15, as will be described later. The authentication information transmitted at this time is, for example, the same information as the authentication information transmitted from the communication terminal 11 in the process of step S113 in FIG. 7 described above.

In step S202, the management server 15 executes operator authentication processing.

Here, the details of the operator authentication process will be described with reference to the flowchart of FIG.

In step S231, the management server 15 acquires the operator's authentication information. That is, the communication unit 204 receives operator authentication information from the operation terminal 12 .

In step S232, the operator monitoring unit 221 determines whether or not the work requires qualification. If it is determined that the work requires qualification, the process proceeds to step S223.

In step S233, the operator monitoring unit 221 determines whether or not it is time to confirm the qualification. If it is determined that it is time to confirm the qualification, the process proceeds to step S234.

For example, the start of work is set as the timing for confirming qualifications. For example, the work performed by the operator is divided into a plurality of work units, and the division between the work units is set at the timing of confirming the qualification.

In step S234, the operator monitoring unit 221 acquires the operator's qualification information. Specifically, the communication unit 204 requests the operation terminal 12 used by the operator to transmit the qualification information related to the qualification necessary for executing the work.

In response, the operation terminal 12 transmits the requested credential information.

　Qualification information is information that indicates that the operator has the qualifications necessary to perform the work. The qualification information is, for example, information recorded in an IC card containing an IC chip given to the holder of the qualification. For example, the qualification information is assumed to be a copy of a certificate certifying possession of the qualification, an ID given to the holder of the qualification, or the like.

In response, the communication unit 204 receives the qualification information transmitted from the operation terminal 12.

After that, the process proceeds to step S234.

On the other hand, if it is determined in step S233 that the work does not require qualification, the process of step S234 is skipped and the process proceeds to step S235.

Also, if it is determined in step S232 that the work does not require qualification, the processing of steps S233 and S234 is skipped, and the processing proceeds to step S235.

In step S235, the operator monitoring unit 221 executes authentication processing. Specifically, the operator monitoring unit 221 performs personal authentication of the operator based on the acquired authentication information and the operator authentication information registered in the operator DB. When it is time to confirm the qualification, the operator monitoring unit 221 confirms whether or not the operator has the qualification necessary for the work based on the qualification information.

When it is time to confirm the operator's qualifications, the operator monitoring unit 221 can confirm that the operator's personal authentication has succeeded and that the operator has the qualifications necessary for the work. Determine that the authentication has succeeded. On the other hand, the operator monitoring unit 221 checks the qualification when it is time to confirm the qualification, when the personal authentication of the operator fails, or when it is not possible to confirm that the operator has the qualification necessary for the work. , it is determined that the authentication of the operator has failed. Further, when it is not the timing to confirm the qualification, the operator monitoring unit 221 determines that the operator has been successfully authenticated when the personal authentication of the operator is successful.

After that, the operator authentication process ends.

Returning to FIG. 10, in step S203, the mediation unit 212 determines whether or not to permit remote operation. Specifically, when the operator is successfully authenticated in the process of step S202, the intermediary unit 212 determines to permit the remote operation, and the process proceeds to step S204.

In step S204, the mediation unit 212 connects the operation terminal 12 and the robot 14. Specifically, the intermediary unit 212 instructs the operation terminal 12 and the robot 14 to be remotely operated to connect to each other via the communication unit 204 . Then, the intermediary unit 212 connects the operation terminal 12 and the robot 14 to each other via the network 21 and the management server 15, and puts them in a communicable state.

In step S205, the management server 15 executes remote operation mediation processing.

Here, the details of the remote operation intermediation process will be described with reference to the flowchart of FIG.

In step S251, the intermediary section 212 receives work status information from the robot 14 via the communication section 204.

Work status information includes, for example, work image data and work sensor data. The work image data is image data obtained by photographing a work site where the robot 14 is working, and is, for example, image data obtained by photographing how the robot 14 performs the work. The work sensor data is, for example, sensor data required for remote control of the robot 14 .

In step S<b>252 , the intermediary section 212 transmits the work status information received from the robot 14 to the operation terminal 12 via the communication section 204 .

Here, the intermediary unit 212 processes the work image data to be transmitted to the operation terminal 12 as necessary.

For example, the intermediary unit 212 performs information protection processing on the work image data.

Specifically, for example, the intermediary unit 212 presents only the work area by blurring the area other than the work area specified by the client or replacing it with a CG (Computer Graphics) image. visible) to generate working image data. The work area is an area that the operator needs to visually recognize in order to work using the robot 14 by remote control.

For example, the intermediary unit 212 blurs the confidential area specified by the client, replaces it with a CG image, or erases it so that the confidential area cannot be visually recognized, thereby concealing the confidential information. Generate data. The confidential area is, for example, an area including pre-release products, blueprints, design drawings, etc., an area including an undisclosed manufacturing process, or the like.

For example, the intermediary unit 212 applies game processing to the work image data. The gaming processing is processing such as effect processing so that the working image data becomes an image that imitates a game screen. As a result, an effect can be expected that the operator will continue to concentrate on the work without getting bored.

It should be noted that the robot 14 may process these work image data and transmit the work image data after processing to the management server 15 . This can further reduce the risk of leakage of confidential information. Further, for example, the management server 15 or the robot 14 may perform processing for concealing confidential information also for work sensor data.

In step S253, the intermediary section 212 receives an operation signal from the operation terminal 12 via the communication section 204.

In step S<b>254 , the intermediary section 212 transmits the operation signal received from the operation terminal 12 to the robot 14 via the communication section 204 .

In step S255, the learning unit 215 starts storing remote control data. Specifically, the learning unit 215 arranges the operation signals received from the operation terminal 12 in chronological order and starts a process of storing them in the storage unit 205 as remote operation data. This remote control data is used as learning data, as will be described later.

Returning to FIG. 10, in step S206, the management server 15 executes operation eligibility monitoring processing.

Here, the details of the operation eligibility process will be described with reference to the flowchart of FIG.

In step S271, operator authentication processing is executed in the same manner as the processing in step S202 of FIG.

In step S272, the management server 15 executes operator status monitoring processing, and then ends the operation eligibility monitoring processing.

Here, the details of the operator state monitoring process will be described with reference to the flowchart of FIG.

In step S291, the management server 15 receives operator status information. That is, the operator monitoring unit 221 receives operator status information from the operation terminal 12 via the communication unit 204 .

The operator state information includes, for example, image data obtained by photographing the operator (hereinafter referred to as operator image data) and biometric information indicating the state of the operator (hereinafter referred to as operator biometric information).

The operator's biometric information includes, for example, the operator's perspiration, heart rate, blood pressure, and the like.

In step S292, the operator monitoring unit 221 executes operator status monitoring processing based on the operator status information. For example, the operator monitoring unit 221 monitors movements, concentration, arousal, tension, physical condition, etc. of each part of the operator (eg, head, line of sight, arms, hands, fingers, etc.). For example, the operator monitoring unit 221 monitors the operator's remote control method, speed, procedure, etc. based on the movement of each part of the operator. The operator monitoring unit 221 determines whether there is an abnormality in the state of the operator based on the results of the monitoring.

After that, the operator status monitoring process ends.

Returning to FIG. 10, in step S207, the operator monitoring unit 221 determines whether or not the operator is qualified to operate. For example, if the operator's authentication fails or if the operator's condition is abnormal, the operator monitoring unit 221 determines that the operator is not qualified for operation, and the process proceeds to step S208.

In step S208, the management server 15 issues a warning that there is no operational eligibility. For example, the intermediary unit 212 generates warning information (hereinafter referred to as "operation unqualified warning information") for notifying the operator of the lack of operational qualification and the reason thereof. The intermediary unit 212 transmits operation unqualified warning information to the operation terminal 12 via the communication unit 204 .

In response to this, the operation terminal 12 receives the operation ineligibility warning information and warns the operator that he is not competent to operate.

In step S209, the operation eligibility monitoring process is executed in the same manner as in step S206. That is, after the warning that the operator is not qualified for operation is issued, the presence or absence of the operator's qualification for operation is determined again.

In step S210, the operator monitoring unit 221 determines whether or not the operator's operational eligibility has been confirmed. For example, when the operator monitoring unit 221 determines in the process of step S209 that the operator has been successfully authenticated and that the operator is in a normal state, the operator monitoring unit 221 determines that the operator's operational eligibility has been confirmed. , the process proceeds to step S211.

On the other hand, in step S207, for example, the operator monitoring unit 221 determines that the operator is qualified to operate when the operator is successfully authenticated and the operator is in a normal state in the process of step S206. Then, the processing from step S208 to step S210 is skipped, and the processing proceeds to step S211.

In step S211, the management server 15 executes work monitoring processing.

Here, the details of the work monitoring process will be described with reference to the flowchart of FIG.

In step S311, the management server 15 executes skill level/prohibited act monitoring processing.

Here, the details of the skill level/prohibited act monitoring process will be described with reference to the flowchart of FIG.

In step S331, the management server 15 acquires the operation signal, operator status information, and work environment information. Specifically, the communication unit 204 receives operation signals and operator state information transmitted from the operation terminal 12 . The communication unit 204 also receives working environment information transmitted from the robot 14 .

The work environment information received at this time is, for example, the same information as the work environment information transmitted from the client terminal 13 in the process of step S101 in FIG.

In step S332, the work monitoring unit 222 executes the skill level monitoring process of the operator based on the operation signal, the operator status information, and the work situation information.

For example, the work monitoring unit 222 detects the operator's action based on the operator image data and operator biometric information included in the operation state information. Specifically, for example, the work monitoring unit 222 detects movements of each part of the operator (eg, head, line of sight, arms, hands, fingers, etc.). Also, for example, the work monitoring unit 222 detects the operator's remote control method, speed, procedure, etc. based on the movement of each part of the operator.

For example, the work monitoring unit 222 detects the operation of the robot 14 remotely controlled by the operator based on the operation signal and the work image data and work sensor data included in the work situation information. Specifically, for example, the work monitoring unit 222 detects the content, speed, accuracy, method, procedure, etc. of the work of the robot 14 .

The work monitoring unit 222 determines whether the operator has the required skill level based on the detection result of the operator's motion and the robot's 14 motion. The skill level is recognized based on, for example, the operation method of the operation terminal 12 and the work procedure, speed, accuracy, etc. of the robot 14 .

In step S333, the work monitoring unit 222 executes monitoring processing for prohibited actions. For example, the work monitoring unit 222 detects whether or not a prohibited action is performed based on the detection result of the motion of the robot 14 .

Here, a prohibited act is, for example, an act that lowers the progress or quality of work. In particular, for example, actions that significantly lower the progress or quality of work, or actions that intentionally lower the progress or quality of work fall under prohibited actions. For example, when a product is being assembled, it is assumed that the wrong part is installed, the part is installed in the wrong position, or the product or part is damaged or destroyed due to falling or the like.

Note that the work monitoring unit 222 determines that the operator has performed a prohibited action when, for example, an action that lowers the progress or accuracy of the work is repeated a predetermined number of times or more or continues for a predetermined time or longer. can be This makes it possible to identify whether the operator has mistakenly executed a prohibited act or intentionally executed a prohibited act with malicious intent.

After that, the skill level/prohibited act monitoring process ends.

Returning to FIG. 16, in step S312, the management server 15 executes operating environment monitoring processing.

Here, the details of the operating environment monitoring process will be described with reference to the flowchart of FIG.

In step S351, the management server 15 acquires operating environment information. Specifically, the communication unit 204 receives operating environment information transmitted from the operating terminal 12 .

The operational environment information transmitted and received at this time is, for example, the same information as the operational environment information transmitted from the communication terminal 11 in the process of step S111 in FIG. 7 described above.

In step S352, the work monitoring unit 222 executes an operating environment monitoring process based on the operating environment information. For example, the work monitoring unit 222 detects whether or not there is an abnormality in the operating terminal 12 based on the operating environment information. For example, the work monitoring unit 222 detects the communication status between the operation terminal 12 and the management server 15 and the presence or absence of communication failure. The communication status includes, for example, at least one of communication speed, delay time, communication capacity, communication band, and packet loss rate.

The work monitoring unit 222 determines whether there is an abnormality in the operating environment based on the presence or absence of an abnormality in the operation terminal 12, the communication status between the operation terminal 12 and the management server 15, and the presence or absence of a communication failure. For example, the work monitoring unit 222 satisfies the conditions that the operation terminal 12 is normal, the communication status between the operation terminal 12 and the management server 15 is required, and If no communication failure has occurred, it is determined that no abnormality has occurred in the operating environment. For example, the work monitoring unit 222 detects when an abnormality occurs in the operation terminal 12, when the communication status between the operation terminal 12 and the management server 15 does not meet the required conditions, or when the operation terminal 12 and the management server 15 If a communication failure occurs with the management server 15, it is determined that an abnormality has occurred in the operating environment.

If an abnormality occurs in the operation terminal 12, for example, if the operation terminal 12 that does not meet the required conditions is used (for example, if the display resolution does not meet the required conditions) etc.).

After that, the operating environment monitoring process ends.

Returning to FIG. 16, in step S313, the work monitoring unit 222 executes work environment monitoring processing, after which the work monitoring processing ends.

Here, the details of the work environment monitoring process will be described with reference to the flowchart of FIG.

In step S371, the management server 15 acquires work environment information. Specifically, the communication unit 204 receives work environment information transmitted from the robot 14 .

In step S372, the work monitoring unit 222 executes work environment monitoring processing based on the work environment information. For example, the work monitoring unit 222 detects whether there is an abnormality in the robot 14 based on work environment information. For example, the work monitoring unit 222 detects the communication status between the robot 14 and the management server 15 and the presence or absence of communication failure. The communication status includes, for example, at least one of communication speed, delay time, communication capacity, communication band, and packet loss rate.

The work monitoring unit 222 determines whether there is an abnormality in the work environment based on the presence or absence of an abnormality in the robot 14, the communication status between the robot 14 and the management server 15, and the presence or absence of a communication failure. For example, the work monitoring unit 222 determines that the robot 14 is normal, the communication status between the robot 14 and the management server 15 satisfies the required conditions, and there is no communication failure between the robot 14 and the management server 15. If no problem has occurred, it is determined that no problem has occurred in the work environment. For example, the work monitoring unit 222 detects when an abnormality occurs in the robot 14, when the communication status between the robot 14 and the management server 15 does not meet the required conditions, or when the robot 14 and the management server 15 If there is a communication failure between the

After that, the working environment monitoring process ends.

Returning to FIG. 11, in step S212, the work monitoring unit 222 determines whether or not a problem has occurred based on the result of the processing in step S211.

For example, the work monitoring unit 222 detects when the operator does not have the required skill level, when a prohibited act is detected, when an abnormality occurs in the operating environment, or when an abnormality occurs in the work environment. If so, it is determined that a problem has occurred in the work status. On the other hand, the work monitoring unit 222 confirms that the operator has the required skill level, that no prohibited act has been detected, that the operating environment is normal, and that the work environment is normal. If so, it is determined that no problem has occurred in the work situation.

Then, if it is determined that a problem has occurred, the process proceeds to step S213.

In step S213, the intermediary unit 212 supports operations or work as necessary.

For example, when a problem occurs in the operator or in the operating environment, the intermediation unit 212 generates operation support information that assists the operator in solving the problem, and sends it to the operation terminal 12 via the communication unit 204. Send.

Operation support information includes, for example, information that notifies the occurrence and content of problems. This makes it possible for the operator to recognize the occurrence and details of the problem and to deal with it.

The operation support information includes, for example, information indicating how to solve the problem. For example, the operation assistance information includes assistance information for assisting the remote operation of the operator when the work quality is below a predetermined standard. The assist information includes, for example, information that guides remote control methods and procedures. For example, the operator can improve the quality of work by performing remote control according to the assist information.

For example, the operation support information includes information that calls the operator's attention when the operator's attention is low, for example, when the operator falls asleep or is not concentrating on the work. Specifically, the operation support information includes information for vibrating an operation device included in the input unit 121 of the operation terminal 12 and outputting a warning sound from the output unit 124 . As a result, the operator's attention can be called and the quality of the work can be improved.

The intermediary unit 212 transmits operation support information to the operation terminal 12 via the communication unit 204 .

For example, if there is a problem in the work environment, the intermediary unit 212 generates work support information that helps the client or the worker at the work site to solve the problem.

Work support information includes, for example, information that notifies the occurrence and content of problems. This makes it possible for the client or worker to recognize the occurrence and content of the problem and to deal with it.

The work support information includes, for example, information indicating how to solve the problem. For example, the work support information includes assist information indicating a method for resolving obstacles of the robot 14 . Thereby, for example, the requester or the worker can solve the obstacle of the robot 14 according to the assist information.

The intermediary section 212 transmits work support information to the client terminal 13 or the robot 14 via the communication section 204 .

In step S214, work monitoring processing is executed in the same manner as in step S211. That is, after the operation or work is supported, the work monitoring process is executed again.

In step S215, the work monitoring unit 222 determines whether or not the work situation has improved based on the result of the processing in step S214. For example, if the problem recognized in step S211 is resolved, the work monitoring unit 222 determines that the work situation has improved, and the process proceeds to step S216.

On the other hand, if it is determined in step S212 that no problem has occurred, the processing of steps S213 to S215 is skipped, and the processing proceeds to step S216.

In step S216, the intermediary unit 212 determines whether or not a request has been made to stop the remote operation. If it is determined that there is no request to stop the remote operation, the process proceeds to step S217.

In step S217, the work monitoring unit 222 determines whether or not the work has been completed. If it is determined that the work has not been completed, the process returns to step S205.

After that, in step S210, it is determined that the operator's operational eligibility could not be confirmed, in step S215, it is determined that the work situation has not improved, or in step S216, it is determined that a request to stop the remote operation has been made. The processing from step S205 to step S217 is repeatedly executed until it is determined or until it is determined in step S217 that the work has been completed.

As a result, while the remote operation of the robot 14 is being performed by the operation terminal 12, intermediation of the remote operation is executed, monitoring of the operator's operational qualification (operator authentication processing and monitoring of the state), and , work status monitoring is continuously performed.

On the other hand, in step S217, the work monitoring unit 222 determines that the work is completed when, for example, all of the scheduled work is finished, or when the end time of the work is reached, and the process proceeds to step S218. .

In addition, in step S216, when the operation terminal 12 or the robot 14 instructs to stop the remote operation via the communication unit 204, the intermediary unit 212 determines that the stop of the remote operation is requested, and the process proceeds to step S218. proceed to

Further, in step S215, the work monitoring unit 222 determines that the work situation has not improved if, for example, the work problem continues even after a predetermined period of time has elapsed since the work problem occurred, and performs the process. goes to step S218.

Also, if it is determined in step S210 that the operation eligibility could not be confirmed, the process proceeds to step S218.

In step S218, the management server 15 stops mediation of remote control. Specifically, the intermediation unit 212 notifies the operation terminal 12 and the robot 14 of stopping intermediation of remote operation via the communication unit 204, and stops intermediation of remote operation.

After that, the processing of the management server 15 ends.

On the other hand, if the authentication of the operator fails in step S203, the intermediary section 212 determines that remote operation is not permitted, and the process proceeds to step S219.

In step S219, the intermediary section 212 notifies the operation terminal 12 and the robot 14 of non-permission of remote control via the communication section 204.

After that, the processing of the management server 15 ends.

<Processing of operation terminal 12>
Next, the processing of the operation terminal 12 executed corresponding to the processing of the management server 15 of FIG. 10 will be described with reference to the flowchart of FIG.

This process is started, for example, when the control unit 123 of the operation terminal 12 receives a request for transmission of operator authentication information from the management server 15 via the communication unit 125 in step S201 of FIG. be done.

In step S401, the operating terminal 12 executes authentication information transmission processing.

Here, the details of the authentication information processing will be described with reference to the flowchart of FIG.

At step S431, the operating terminal 12 acquires the operator's authentication data. For example, the control unit 123 acquires authentication data including biometric information used for biometric authentication of the operator from the sensor data output from the detection unit 122 . For example, facial image data obtained by photographing the face of the operator, fingerprint image data obtained by photographing the fingerprint, vein image data obtained by photographing the vein, iris image data obtained by photographing the iris, voice data obtained by recording the voice of the operator, etc. Acquired as data. These authentication data are acquired by a camera for photographing the operator, a biosensor attached to the operator, or the like.

In step S432, the control unit 123 extracts authentication information from the authentication data. For example, the control unit 123 extracts biometric information used for biometric authentication of the operator from the authentication data as authentication information.

In step S433, the communication unit 125 transmits the authentication information to the management server 15.

In step S434, the control unit 123 determines whether or not transmission of the credential information has been requested. When the control unit 123 receives a request for transmission of the credential information from the management server 15 through the processing of step S234 of FIG. Proceed to S435.

In step S435, the control unit 123 acquires the operator's qualification information. For example, the control unit 123 acquires the qualification information input by the user via the input unit 121 or the qualification information stored in the storage unit 126 .

In step S436, the communication unit 125 transmits the credential information to the management server 15.

After that, the authentication information transmission process ends.

On the other hand, if it is determined in step S434 that the transmission of the credential information has not been requested, the processing of steps S435 and S436 is skipped, and the authentication information transmission processing ends.

Returning to FIG. 20, in step S402, the control unit 123 determines whether or not an instruction to connect to the robot 14 has been given. When the controller 123 receives an instruction to connect to the robot 14 from the management server 15 through the process of step S203 in FIG. goes to step S403.

In step S403, the operating terminal 12 connects to the robot 14 via the management server 15. Specifically, the communication unit 125 executes connection processing with the robot 14 via the network 21 and the management server 15 under the control of the control unit 123 . As a result, the communication unit 125 is connected to the robot 14 via the network 21 and the management server 15 and is ready to communicate with the robot 14 .

In step S404, the operating terminal 12 executes work status presentation processing.

Here, the details of the work situation presentation process will be described with reference to the flowchart of FIG.

In step S451, the operating terminal 12 acquires work status information. Specifically, the communication unit 125 receives the work status information transmitted from the management server 15 in step S252 of FIG.

At step S452, the operation terminal 12 presents the work status. For example, the output unit 124 displays an image of the work site of the robot 14 under the control of the control unit 123 based on the work image data included in the work situation information. Further, for example, the output unit 124 displays sensor values required for remote control of the robot 14 under the control of the control unit 123 based on work sensor data included in the work situation information.

In step S453, the control unit 123 determines whether or not it has received a warning of lack of operational eligibility. Specifically, when the control unit 123 receives the operation unqualified warning information transmitted by the management server 15 in step S208 of FIG. , the process proceeds to step S454.

In step S454, the operation terminal 12 warns that there is no operational eligibility. For example, the output unit 124, under the control of the control unit 123, warns the operator that the operator is not qualified for operation based on the warning information for unqualified operation. At this time, the operator may be presented with information such as the reason why it is determined that the operator is not eligible for operation, information necessary for confirming the operator's suitability for operation, and the like.

After that, the process proceeds to step S455.

On the other hand, if it is determined in step S453 that the warning of lack of operation eligibility has not been received, the process of step S454 is skipped and the process proceeds to step S455.

In step S455, the control unit 123 determines whether or not the operation support information has been acquired. Specifically, when the control unit 123 receives the operation support information transmitted by the management server 15 in step S213 of FIG. goes to step S456.

In step S456, the operation terminal 12 provides operation support. For example, when a problem occurs in the operator or in the operating environment, the output unit 124 outputs, under the control of the control unit 123, information notifying the occurrence and content of the problem, and information indicating how to solve the problem ( For example, the above-mentioned assist information) is presented to the operator by means of images and sounds. For example, the output unit 124 outputs a warning to draw the operator's attention under the control of the control unit 123 when the operator's attention is low. For example, when the attention of the operator is low, the input unit 121 vibrates the operation device used by the operator to remotely operate the robot 14 under the control of the control unit 123 .

After that, the work status presentation process ends.

On the other hand, if it is determined in step S455 that the operation support information has not been acquired, the process of step S456 is skipped and the work situation presentation process ends.

Returning to FIG. 20, in step S405, the operating terminal 12 transmits an operation signal in accordance with the operator's operation. For example, the control unit 123 generates an operation signal for remotely operating the robot 14 when the operator uses the input unit 121 to perform an operation for remotely operating the robot 14 . The communication unit 125 transmits operation signals to the management server 15 .

In step S406, authentication information transmission processing is executed in the same manner as in step S401.

In step S407, the operating terminal 12 executes operator status information transmission processing.

Here, the details of the operator status information transmission process will be described with reference to the flowchart of FIG.

In step S471, the operating terminal 12 acquires the captured image of the operator. Specifically, for example, the detection unit 122 captures an image of the operator operating the operation terminal 12 and supplies the obtained operator image data to the control unit 123 .

In step S472, the operating terminal 12 acquires the biometric information of the operator. For example, the control unit 123 extracts operator biometric information indicating the state of the operator from the sensor data output from the detection unit 122 .

In step S473, the control unit 123 generates operator state information. Specifically, the control unit 123 generates operator state information including the operator image data and the operator's biological information.

In step S474, the communication unit 125 transmits the operator status information to the management server 15.

After that, the operator status information transmission process ends.

Returning to FIG. 20, in step S408, the operating terminal 12 executes the operating environment information transmission process.

Here, the details of the operating environment information transmission process will be described with reference to the flowchart of FIG.

In step S491, the control unit 123 acquires operating environment data. For example, the control unit 123 acquires data on the identification information, type, and specifications of the operation terminal 12 from the storage unit 126 . For example, the control unit 123 acquires data regarding the position of the operation terminal 12 from the detection unit 122 . For example, the control unit 123 acquires data regarding the state of the operation terminal 12 . For example, the communication unit 125 actually performs a communication test with the management server 15 via the network 21, acquires data regarding the communication status between the operation terminal 12 and the management server 15, and supplies the data to the control unit 123. do.

In step S492, the control unit 123 extracts operating environment information from the operating environment data. The authentication information extracted at this time is, for example, the same information as the operating environment information transmitted from the communication terminal 11 in the process of step S111 in FIG.

In step S493, the communication unit 125 transmits the operating environment information to the management server 15.

After that, the operating environment information transmission process ends.

Returning to FIG. 20, in step S409, the control unit 123 determines whether or not the intermediation of remote operation has been stopped. If it is determined that the intermediation of remote control has not been stopped, the process proceeds to step S410.

In step S410, the control unit 123 determines whether or not to request termination of remote operation. If it is determined not to request the cancellation of the remote operation, the process returns to step S404.

After that, the processing from step S404 to step S410 is repeatedly executed until it is determined in step S409 that the intermediation of the remote operation has been stopped, or until it is determined in step S410 that a request to stop the remote operation is requested.

On the other hand, in step S410, the control unit 123, for example, when the operator uses the input unit 121 to perform an operation to stop the remote operation, or when a failure occurs and the remote operation of the robot 14 cannot be continued. If it is difficult, it is determined to stop the remote operation, and the process proceeds to step S411.

In step S411, the control unit 123 requests the management server 15 via the communication unit 125 to stop the remote operation.

After that, the process proceeds to step S412.

On the other hand, in step S409, when control unit 123 receives a notification to stop intermediation of remote operation from management server 15 through communication unit 125 in the process of step S218 in FIG. 11, intermediation of remote operation is stopped. and the process proceeds to step S412.

In step S412, the control unit 123 stops remote operation.

After that, the processing of the operation terminal 12 ends.

On the other hand, in step S402, when the management server 15 notifies the control unit 123 of the non-permission of remote operation through the communication unit 204 by the process of step S219 in FIG. It is determined that there is no operation terminal 12, and the processing of the operation terminal 12 ends.

<Processing of robot 14>
Next, the processing of the robot 14 executed in correspondence with the processing of the management server 15 of FIG. 10 will be described with reference to the flowchart of FIG.

This process is started, for example, when the control unit 163 of the robot 14 receives a connection support instruction with the operation terminal 12 from the management server 15 via the communication unit 166 in step S204 of FIG.

In step S501, the robot 14 connects to the operation terminal 12 via the management server 15. Specifically, the communication unit 166 executes connection processing with the operation terminal 12 via the network 21 and the management server 15 under the control of the control unit 163 . As a result, the communication unit 166 is connected to the operation terminal 12 via the network 21 and the management server 15 and becomes communicable with the operation terminal 12 .

In step S502, the robot 14 executes work status information transmission processing.

Here, the details of the work status information transmission process will be described with reference to the flowchart of FIG.

In step S531, the robot 14 acquires a photographed image of the work site. Specifically, the detection unit 162 photographs the work site of the robot 14 and supplies the obtained work image data to the control unit 163 .

At step S532, the robot 14 acquires sensor data. Specifically, the control unit 163 acquires work sensor data required for remote control of the robot 14 from the detection unit 162 .

In step S533, the control unit 163 generates work status information. Specifically, the control unit 163 generates work situation information including work image data and work sensor data.

In step S534, the communication unit 166 transmits work status information to the management server 15.

After that, the work status information transmission process ends.

Returning to FIG. 25, in step S503, the robot 14 executes work execution processing.

Here, the details of the work execution process will be described with reference to the flowchart of FIG.

In step S551, the control unit 163 determines whether or not an operation signal has been acquired. Specifically, when the control unit 163 receives the operation signal transmitted from the management server 15 in step S254 of FIG. 13 via the communication unit 166, it determines that the operation signal has been acquired, and the process proceeds to step S552. proceed to

At step S552, the robot 14 performs the work based on the operation signal. For example, the control section 163 controls the movable section 164 based on the operation signal, and causes the robot 14 to execute processing corresponding to the operation signal. As a result, the robot 14 is remotely operated using the operation terminal 12, and the work requested by the client is executed.

After that, the process proceeds to step S553.

On the other hand, if it is determined in step S551 that the operation signal has not been acquired, the process of step S552 is skipped and the process proceeds to step S553.

In step S553, the control unit 163 determines whether work support information has been acquired. Specifically, when the control unit 163 receives the work support information transmitted from the management server 15 in step S213 of FIG. 11 via the communication unit 166, it determines that the work support information has been acquired. The process proceeds to step S554.

In step S554, the robot 14 provides work assistance based on the work assistance information. For example, when a problem occurs in the robot 14 or in the working environment, the output unit 165 outputs, under the control of the control unit 163, information notifying the occurrence and content of the problem, and information indicating how to solve the problem ( For example, the above-mentioned assist information) is presented to the requester or the worker at the work site by image or sound.

After that, the work execution process ends.

On the other hand, if it is determined in step S553 that the work support information has not been acquired, the process of step S554 is skipped and the work execution process ends.

Returning to FIG. 25, in step S504, the robot 14 executes work environment information transmission processing.

Here, the details of the work environment information transmission process will be described with reference to the flowchart of FIG.

In step S571, the robot 14 acquires work environment data. For example, the control unit 163 acquires data regarding the identification information, type, and specifications of the robot 14 from the storage unit 167 . For example, the control unit 163 acquires data regarding the position of the robot 14 from the detection unit 162 . For example, the control unit 163 acquires data regarding the operation status of the robot 14, the error occurrence status, and the like. For example, the control unit 163 acquires data regarding the communication status between the robot 14 and the management server 15 from the communication unit 166 .

In step S572, the control unit 163 extracts working environment information from the working environment data. The work environment information extracted at this time is, for example, the same information as the work environment information transmitted from the client terminal 13 in the process of step S101 in FIG.

In step S573, the communication unit 166 transmits the working environment information to the management server 15.

After that, the work environment information transmission process ends.

Returning to FIG. 25, in step S505, the control unit 163 determines whether or not the intermediation of remote operation has been stopped. If it is determined that the mediation of remote control has not been stopped, the process proceeds to step S506.

In step S506, the control unit 163 determines whether or not to request termination of remote operation. If it is determined not to request the termination of the remote operation, the process returns to step S502.

After that, the processing of steps S502 to S506 is repeatedly executed until it is determined in step S505 that the intermediation of the remote operation has stopped, or until it is determined in step S506 that a request to stop the remote operation is requested.

On the other hand, in step S506, the control unit 163, for example, when the requester uses the input unit 161 to perform an operation to stop the remote operation, or when a failure occurs and it is difficult to continue the operation of the robot 14. If so, it is determined to stop the remote operation, and the process proceeds to step S507.

In step S507, the control unit 163 requests the management server 15 to stop the remote operation.

After that, the process proceeds to step S508.

On the other hand, in step S505, when control unit 163 receives a notification to stop intermediation of remote operation from management server 15 via communication unit 166 in the process of step S218 in FIG. 11, intermediation of remote operation is stopped. and the process proceeds to step S508.

In step S508, the control unit 163 stops processing corresponding to remote operation.

After that, the processing of the robot 14 ends.

<Post-work processing>
Next, post-work processing executed by the information processing system 1 will be described with reference to the flowchart of FIG. 29 .

This process is executed, for example, after the above-described processes of FIGS. 10 to 28 are completed.

In step S601, the management server 15 evaluates the work content. For example, the evaluation unit 214 evaluates the work performed by the operator through remote control based on the results of monitoring during work by the operator monitoring unit 221 and the work monitoring unit 222 . The work content is evaluated, for example, by one or more of the work speed, accuracy, achievement rate, and the quality of the product obtained by the work.

Note that, for example, when a defective product is generated, the evaluation unit 214 adds the information to the record of the target operator in the operator DB so that the operator who generated the defective product and the details of the operation can be identified. You may make it This improves traceability in investigating the cause of defective products.

In step S602, the evaluation unit 214 of the management server 15 requests the client terminal 13 via the communication unit 204 for the client's evaluation of the current work.

On the other hand, in step S611, the client terminal 13 acquires the client's evaluation of the current work. For example, the output unit 144 displays, under the control of the control unit 143, an input screen for inputting the client's evaluation of the current work.

The client's evaluation of this work includes, for example, an evaluation of the operator and an evaluation of the work performed by the operator. The evaluation of the operator includes, for example, evaluation of the operator's response, ability (for example, skill level), and the like. The evaluation of work content includes, for example, evaluation of work quality and the like.

In response to this, the requester uses the input unit 141 to input the evaluation of the current work into the requester terminal 13 .

In step S612, the control unit 143 of the client terminal 13 notifies the management server 15 of the client's evaluation of the current work via the communication unit 145.

On the other hand, in step S603, the management server 15 determines the reward based on the evaluation of the work content and the client's evaluation, and updates the evaluation of the operator.

For example, when the work achievement rate is less than 100%, that is, when the work requested of the operator is not completed, the evaluation unit 214 gives the operator a reward based on the work achievement rate, Subtract from the remuneration presented at the time of requesting the work.

For example, if the work achievement rate is 100%, the evaluation unit 214 determines the reward based on the client's evaluation of the work. For example, when the requester's evaluation is 100% or less, the evaluation unit 214 determines the remuneration to be given to the operator as the remuneration presented at the time of requesting the work. For example, when the requester's evaluation exceeds 100%, the evaluation unit 214 determines the remuneration given to the operator to be the remuneration presented at the time of requesting the work×120%.

For example, the evaluation unit 214 updates the ability information (for example, skill level, etc.) of the target operator in the operator DB based on the evaluation of the work content.

For example, the evaluation unit 214 updates the evaluation score of the target operator in the operator DB based on the client's evaluation.

In step S604, the evaluation unit 214 notifies the communication terminal 11 of the reward via the communication unit 204.

In response to this, in step S621, the communication terminal 11 presents a reward. For example, the output unit 104 presents the reward notified from the management server 15 to the operator under the control of the control unit 103 .

In step S622, the communication terminal 11 acquires the operator's evaluation of the current work. Specifically, the output unit 104 displays, under the control of the control unit 103, a screen for inputting the operator's evaluation of the current work.

The operator's evaluation of the current work includes, for example, the evaluation of the requester and the evaluation of the requested work. The evaluation of the client includes, for example, evaluation of the client's response, remuneration, and the like. The evaluation of the requested work includes, for example, evaluation of the difficulty of the work, work environment, and the like.

In response to this, the operator uses the input unit 101 to input an evaluation of the current work into the communication terminal 11 .

In step S623, the control unit 103 notifies the management server 15 of the operator's evaluation of the current work via the communication unit 105.

On the other hand, in step S605, the management server 15 generates learning data. Specifically, the learning unit 215 includes, as input data, the remote operation data accumulated in the storage unit 205 in the current work, and learning data labeled based on the client's evaluation of the work and the operator's evaluation. to generate For example, the learning unit 215 assigns a label indicating whether the work was successful or unsuccessful or a label quantifying the quality of the work to the learning data based on the client's evaluation and the operator's evaluation of the work. The learning unit 215 accumulates the generated learning data in the storage unit 205 .

As a result, the storage unit 205 accumulates learning data including remote operation data when the work is successful, or learning data including remote operation data when the work is unsuccessful.

The learning unit 215 uses learning data accumulated in the storage unit 205 to learn a learning model for automating the work of the robot 14 . By applying the obtained learning model to the robot 14, the robot 14 can automatically perform a predetermined work without requesting remote control.

As described above, matching between the operator who performs the remote operation and the robot 14 is properly performed. As a result, the quality of work performed by remote control is improved, and the client's satisfaction is improved. In addition, since the operator is matched with work according to his/her ability, the degree of satisfaction is improved.

In addition, since the operation eligibility and work status are constantly monitored, the remote control of the robot 14 can be properly executed. For example, the operator will be able to perform remote operation while fully demonstrating his or her ability without cutting corners or changing hands, which will further improve the quality of work. Furthermore, if the quality of the work cannot be ensured due to the operator's lack of ability or the like, countermeasures such as changing the operator can be taken.

In addition, learning data based on remote operation is accumulated every time each operator performs work by remote operation. Then, learning of the learning model is performed using the accumulated learning data, and automation of the work of the robot 14 can be achieved using the generated learning model.

<Second Embodiment of Processing of Information Processing System 1>
Next, a second embodiment of processing of the information processing system 1 will be described with reference to FIGS. 30 to 32. FIG.

The second embodiment of the processing of the information processing system 1 differs from the first embodiment in matching processing.

<Overview of Second Embodiment of Processing of Information Processing System 1>
First, the outline of the second embodiment of the processing of the information processing system 1 will be described with reference to the sequence diagram of FIG. The sequence diagram of FIG. 30 shows an overview of the processing of the information processing system 1, like the sequence diagram of FIG.

First, matching processing between the operator (operation terminal 12) and the client (robot 14) is executed.

Specifically, for example, the client sends client information about the client to the management server 15.

The operator transmits operator information about the operator to the management server 15.

In response, the management server 15 registers client information and operator information.

The processing up to this point is the same as in the first embodiment.

Next, the requester requests the management server 15 to select an operator who will perform the desired work.

The management server 15 extracts candidates for the operator to whom the work is requested, and generates an operator candidate list listing the operator candidates. The management server 15 transmits the operator candidate list to the client.

The requester selects an operator to request work from among the operator candidates shown in the operator candidate list. The operator notifies the management server 15 of the operator's selection result.

The management server 15 requests the operator selected by the requester to perform the work.

The operator will decide whether or not to accept the requested work. The operator notifies the management server 15 of acceptance or rejection of the work.

The management server 15 notifies the requester whether or not the operator selected by the requester can accept the work.

When the operator accepts the work, remote control processing and post-work processing are subsequently executed. Since the remote control processing and post-work processing are the same as those in the first embodiment, description thereof will be omitted.

On the other hand, if the operator does not accept the work, the requester starts over from selecting an operator from the operator candidate list.

<Second Embodiment of Matching Processing>
Next, a second embodiment of matching processing will be described with reference to the flowchart of FIG.

In step S701, the requester terminal 13 transmits the requester information to the management server 15, similar to the process of step S101 in FIG.

In step S711, the communication terminal 11 transmits the operator information to the management server 15, similar to the process of step S111 in FIG.

In step S721, the management server 15 registers client information and operator information in the same manner as in the process of step S121 in FIG.

In step S702, the client terminal 13 requests the management server 15 to select an operator, as in the process of step S102 in FIG.

In response to this, in step S722, the management server 15 executes operator selection processing.

Here, the details of the operator selection process will be described with reference to the flowchart of FIG.

In steps S751 to S756, the same processing as steps S151 to S156 in FIG. 9 is executed. As a result, operator candidates are extracted from the operators registered in the operator DB.

In step S757, the matching unit 211 generates an operator candidate list. That is, the matching unit 211 generates an operator candidate list in which the extracted operator candidates are listed.

Note that, for example, the matching unit 211 extracts a predetermined number of operators from the highest matching degree or operators whose matching degree is equal to or greater than a predetermined threshold, and lists the extracted operator candidates. A list may be generated.

The operator candidate list includes, for example, information on the abilities and operating environment of each operator candidate. Note that, for example, the candidate operator list may include the degree of matching and the evaluation score of each candidate operator.

After that, the operator selection process ends.

Returning to FIG. 31, in step S723, the matching unit 211 transmits the operator candidate list to the client terminal 13 via the communication unit 204. Through this processing, the matching unit 211 controls the presentation of the extracted operator candidates to the requester.

In response to this, in step S703, the client terminal 13 presents an operator candidate list. Specifically, the communication unit 145 receives the operator candidate list from the management server 15 . The output unit 144 presents the operator candidate list to the client under the control of the control unit 143 . As a result, the operator candidate extracted by the management server 15 is recommended to the client.

In step S704, the client terminal 13 acquires and notifies the operator's selection result. Specifically, the requester selects an operator to whom the work is requested from the operator candidate list, and uses the input unit 141 to input the operator's selection result to the requester terminal 13 . The control unit 143 notifies the management server 15 of the operator's selection result via the communication unit 145 .

In response to this, in step S724, the management server 15 asks the selected operator to accept the work. Specifically, the matching unit 211 transmits work request information to the communication terminal 11 of the operator selected by the operator via the communication unit 204, and inquires whether or not the work can be accepted.

On the other hand, in step S712, the communication terminal 11 presents a work request in the same manner as in step S112 of FIG.

In step S713, the communication terminal 11 acquires whether or not the work can be accepted, and notifies the management server 15 of it, in the same manner as in the process of step S113 in FIG.

On the other hand, in step S725, the management server 15 authenticates the operator in the same manner as in step S124 of FIG.

In step S726, the management server 15 notifies whether or not the operator's work can be accepted. Specifically, when the operator accepts the work and the operator is successfully authenticated, the matching unit 211 notifies the requester terminal via the communication unit 204 that the operator can accept the work. Notify 13. On the other hand, if the operator does not accept the work, or if the operator authentication fails, the matching unit 211 notifies the requester terminal via the communication unit 204 that the operator cannot accept the work. Notify 13.

In response to this, in step S705, the client terminal 13 presents whether or not the operator can accept the work. Specifically, under the control of the control unit 143, the output unit 144 presents to the client whether or not the operator's work can be accepted.

If the operator selected by the requester cannot accept the work, for example, the processing from step S703 onwards is executed. As a result, the requester reselects an operator from the operator candidate list, and asks the selected operator to accept the work.

<Third Embodiment of Processing of Information Processing System 1>
Next, a third embodiment of processing of the information processing system 1 will be described with reference to FIGS. 33 and 34. FIG.

The third embodiment of the processing of the information processing system 1 differs in matching processing from the first and second embodiments.

<Overview of Third Embodiment of Processing of Information Processing System 1>
First, with reference to the sequence diagram of FIG. 33, the outline of the third embodiment of the processing of the information processing system 1 will be described. The sequence diagram of FIG. 30 shows an outline of the processing of the information processing system 1, like the sequence diagrams of FIGS.

First, matching processing between the operator (operation terminal 12) and the client (robot 14) is executed.

Specifically, for example, the client sends client information about the client to the management server 15.

The operator transmits operator information about the operator to the management server 15.

In response, the management server 15 registers client information and operator information.

The processing up to this point is the same as in the first and second embodiments.

Next, the requester requests the management server 15 to select an operator who will perform the desired work.

In response to this, the management server 15 registers a request for work for which operator selection is requested.

Also, the operator inquires of the management server 15 about work requests currently recruiting operators.

On the other hand, the management server 15 extracts work requests that can be handled by the operator based on the operator's ability and operating environment, and generates a work request list listing the extracted work requests. The management server 15 transmits the work request list to the operator.

In response, the operator selects the desired work request from the work request list. The operator notifies the management server 15 of the selection result of the work request.

In response to this, the management server 15 generates an operator candidate list that lists the operators who have applied for the work request for which the requester has requested the operator to be selected. The management server 15 transmits the operator candidate list to the client.

In response, the requester selects an operator to request the work from the operator candidate list. The requester notifies the management server 15 of the operator's selection result.

In response to this, the management server 15 notifies each operator who has applied for a work request whether or not there is a work request, based on the operator's selection result by the requester. That is, when the requester selects, the operator is notified that the work is requested, and when the requester does not select, the operator is notified that the work is not requested.

When the operator requests work, remote control processing and post-work processing are subsequently executed. Since remote control processing and post-work processing are the same as those in the first and second embodiments, description thereof will be omitted.

As described above, the management server 15 can appropriately recommend operator candidates to the client. Also, the requester can select a desired operator from among a plurality of operator candidates recommended by the management server 15 .

<Third Embodiment of Matching Processing>
Next, a third embodiment of matching processing will be described with reference to the flowchart of FIG.

In step S801, the requester terminal 13 transmits the requester information to the management server 15, similar to the process of step S101 in FIG.

In step S811, the communication terminal 11 transmits the operator information to the management server 15, similar to the process of step S111 in FIG.

In step S821, the management server 15 registers client information and operator information in the same manner as in step S121 of FIG.

In step S802, the requester terminal 13 requests the management server 15 to select an operator, as in the process of step S102 in FIG.

On the other hand, in step S822, the management server 15 registers the work request. Specifically, the communication unit 204 receives work request information from the client terminal 13 . The matching unit 211 registers newly received work request information in the work request DB accumulated in the storage unit 205 .

At step S812, the communication terminal 11 inquires about the work request being solicited.

For example, the operator uses the input unit 101 to input to the communication terminal 11 an instruction to inquire about a work request that is being recruited.

In response to this, the control unit 103 generates work request inquiry information for inquiring about a work request that is being solicited. The work request inquiry information includes, for example, operator identification information and authentication information. The authentication information is generated by a process similar to that of step S113 in FIG. The communication unit 105 transmits work request inquiry information to the management server 15 .

On the other hand, in step S823, the management server 15 identifies the operator's ability and operating environment. Specifically, the communication unit 204 receives work request inquiry information from the communication terminal 11 . The matching unit 211 authenticates the operator based on the authentication information included in the work request inquiry information and the authentication information registered in the operator DB. When the operator is successfully authenticated, the matching unit 211 identifies the operator's ability and operating environment based on the operator DB.

In step S824, the matching unit 211 extracts work requests that can be handled by the operator and generates a work request list. Specifically, based on the work request DB, the matching unit 211 extracts work requests that can be dealt with by the operator's ability and operating environment from among the work requests that are recruiting operators. In other words, the matching unit 211 extracts a work request that satisfies the required selection conditions for the operator's ability and operating environment. The matching unit 211 generates a work request list in which work request information corresponding to the extracted work request is listed.

In step S825, the matching unit 211 transmits the work request list to the communication terminal 11 via the communication unit 204. Through this processing, the matching unit 211 controls presentation of the extracted work request to the operator.

In response to this, in step S813, the communication terminal 11 presents a work request list. Specifically, the communication unit 105 receives the work request list from the management server 15 . The output unit 104 presents the work request list to the operator under the control of the control unit 103 . As a result, the work request extracted by the management server 15 is recommended to the operator.

In step S814, the communication terminal 11 acquires and notifies the selection result of the work request.

Specifically, the operator uses the input unit 101 to select a work request to apply from the presented work request list.

In response, the control unit 103 notifies the management server 15 of the work request selection result via the communication unit 105 .

In step S826, the management server 15 generates an operator candidate list based on the application status of operators. Specifically, the matching unit 211 generates an operator candidate list listing all operators who have applied for the work request selected by the operator in the process of step S814.

The content of the operator candidate list is the same as the operator candidate list generated in the process of step S757 in FIG.

In step S827, the management server 15 transmits the operator candidate list to the client terminal 13, similar to the process of step S723 in FIG.

On the other hand, in step S803, the client terminal 13 presents an operator candidate list in the same manner as in the process of step S703 in FIG.

In step S804, the requester terminal 13 acquires the operator's selection result and notifies it to the management server 15 in the same manner as in the process of step S804 in FIG.

In response to this, in step S828, the management server 15 notifies whether there is a work request. Specifically, based on the operator's selection result notified from the requester terminal 13, the matching unit 211 sends the communication terminal 11 of each operator who applied for the target work whether or not there is a work request (work request). request or not).

In response to this, in step S815, the communication terminal 11 presents the presence or absence of a work request. Specifically, the output unit 104 presents the presence or absence of a work request to the operator under the control of the control unit 103 .

As described above, the management server 15 can appropriately recommend work requests that can be handled by the operator. Also, the operator can select a desired work request from the work requests that he/she can handle. Furthermore, the requester can select a desired operator from among multiple operator candidates who have applied for the work request.

<<3. Second Embodiment>>
Next, a second embodiment of the present technology will be described with reference to FIG.

<Configuration example of information processing system 301>
FIG. 35 illustrates a configuration example of functions of an information processing system 301 according to the second embodiment of the present technology. In the figure, parts corresponding to those of the information processing system 1 shown in FIG.

The information processing system 301 differs from the information processing system 1 in that an intermediary device 311a and an intermediary device 311b are added.

The mediation device 311a is arranged between the operation terminal 12 and the network 21 (not shown), and mediates communication between the operation terminal 12 and the management server 15. In other words, the operation terminal 12 communicates with the management server 15 via the intermediary device 311a. Also, for example, the intermediary device 311a acts as a proxy for part of the processing of the operation terminal 12 . The intermediary device 311a includes a control unit 321a and a communication unit 322a.

The control unit 321a includes a processor such as a CPU, for example. The control unit 321a controls the intermediary device 311a and executes various types of information processing.

The communication unit 322a communicates with the operation terminal 12 and the management server 15.

The mediation device 311b is placed between the robot 14 and the network 21 (not shown) and mediates communication between the robot 14 and the management server 15. In other words, the robot 14 communicates with the management server 15 via the intermediary device 311b. Also, for example, the intermediary device 311b acts as a proxy for part of the processing of the robot 14 . The mediation device 311b includes a control unit 321b and a communication unit 322b, like the mediation device 311a.

For example, the intermediary device 311b performs compression processing or feature amount extraction processing on the image data generated by the detection unit 162 of the robot 14, and then transmits the data to the management server 15. This reduces the amount of communication between the robot 14 and the management server 15 .

For example, the mediation device 311a and the mediation device 311b perform security authentication with the management server 15. This eliminates the need for the operation terminal 12 and the robot 14 to be directly connected to the platform system on the management server 15, making it easier to ensure the security of the operation terminal 12, the robot 14, and the management server 15.

<<4. Third Embodiment>>
Next, with reference to FIG. 36, a third embodiment of the present technology will be described.

<Configuration example of information processing system 351>
FIG. 36 shows a configuration example of functions of an information processing system 351 according to the third embodiment of the present technology. In the figure, the parts corresponding to those of the information processing system 1 shown in FIG. 3 and the information processing system 301 shown in FIG.

The information processing system 351 differs from the information processing system 301 in that an intermediary device 361a and an intermediary device 361b are provided instead of the intermediary device 311a and the intermediary device 311b.

The mediation device 361a and the mediation device 361b have substantially the same functions as the mediation device 311a and the mediation device 311b in FIG. However, the mediation device 361a and the mediation device 361b differ from the mediation device 311a and the mediation device 311b in that the mediation device 361a and the mediation device 361b can communicate with each other without going through the management server 15 .

For example, after the management server 15 matches the mediation device 361a and the mediation device 361b, the mediation device 361a and the mediation device 361b communicate directly without going through the management server 15. This reduces communication delay between the operation terminal 12 and the robot 14 .

In this case, the mediation devices 361a and 361b and the management server 15 may be connected via a global network (for example, the Internet), and the mediation device 361a and the mediation device 361b may be connected via a closed network. desirable. This enables high-speed or low-delay communication.

For example, the intermediary device 361a of hospital A and the intermediary device 361b of hospital B communicate and are matched via a platform system running on the management server 15, which is a cloud server. After that, the intermediation device 361a and the intermediation device 361b communicate via a closed network connecting the hospitals. In addition, the mediation device 361a and the mediation device 361b are controlled by the management server 15 to set which mediation device to communicate with, and do not accept communication other than mediation devices set by the management server 15. FIG.

<<5. Application example >>
Next, application examples of the present technology will be described with reference to FIGS. 37 to 41. FIG.

<Example of application to remote control of surgical robot>
The present technology can be applied, for example, when performing remote control of a surgical robot.

Here, with reference to FIGS. 37 and 38, a configuration example of a table for managing the skills required for remote operation of the surgical robot and the skills possessed by each operator will be described.

FIG. 37 shows a configuration example of a skill table indicating skills required for remote operation of a surgical robot. The skill table includes items of skill code, clinical department, procedure, and skill.

A skill code is a code for identifying each skill.

The clinical department indicates the clinical department where the target skill is used in the hospital.

A procedure is a procedure performed by a surgical robot and indicates the type of procedure in which the target skill is used.

The skill indicates the skill used in the target procedure. Note that when a plurality of skills are used in the same procedure, different skill codes are assigned to each skill.

FIG. 38 shows a configuration example of a possessed skill table showing the skills possessed by each operator. This possessed skill table is generated for each possessor. This possessed skill table includes skill codes, presence/absence of skills, and skill levels.

The skill code corresponds to the skill code in the skill table in FIG.

The presence or absence of skills indicates whether or not the operator possesses the target skill.

The skill level indicates the operator's level for the target skill. Skill levels are, for example, classified into four levels from A to D, with A, B, C, and D in descending order of level. Note that the skill level is set to D when the operator does not possess the target skill.

It should be noted that, for example, the operator's skill level for the surgical robot is evaluated based on the movement of the operator's line of sight on the monitor displaying the surgical site. For example, a high skill level is set for an operator who can smoothly and quickly move his/her line of sight to an area to be watched, and an operator who can continue working without moving his/her line of sight from the area to be watched.

For example, the operator's skill level for the surgical robot is evaluated based on the operator's hand movements with respect to the operation terminal. For example, a high skill level is set for an operator who can move his hands appropriately and quickly, and an operator who makes less useless hand movements.

For example, the operator's skill level for the surgical robot is evaluated based on the operator's remote control of the surgical robot's instruments and camera movements. For example, a high skill level is set for an operator who can quickly move an instrument to an appropriate position, or an operator who can quickly move the imaging range of a camera to an appropriate position.

When the operator remotely operates the surgical robot, for example, operator image data (live video) captured by the operator is used as the operator authentication information in the operation qualification monitoring process. For the operator's qualification information, for example, a copy image of the operator's doctor's license is used.

Also, in the work monitoring process, the operator's skill level is monitored based on, for example, the operator's technique and whether or not the robot 14 has made a surgical error.

Furthermore, low latency, for example, is required as a condition regarding the communication status of the operating environment.

<Example of application to remote control of industrial robots>
The present technology can be applied, for example, to remotely control an industrial robot.

FIG. 39 shows an example of a skill table when applying this technology to an industrial robot. This skill table includes skill codes, steps, and skills.

A skill code is a code for identifying each skill.

The process indicates the process in which the target skill is used. For example, when remote-controlling an industrial robot in a vehicle factory, processes such as car body painting and car body assembly are assumed.

Skills indicate the skills used in the target process. When multiple skills are used in the same process, a different skill code is assigned to each skill.

Also when remotely controlling an industrial robot, a possessed skill table having the same configuration as the possessed skill table of FIG. 38 described above is used.

In addition, the skill level for each skill of each operator is, for example, at least one of skill certification test, total past remote operation time in each process of each operator, and years of experience in each process of each operator. evaluated based on

When an operator remotely operates an industrial robot, operator image data (live video) captured by the operator is used as the operator authentication information in the operation qualification monitoring process, for example. As the operator's qualification information, for example, a certificate of qualification necessary for operating an industrial robot, a copy image of an IC card, or the like is used.

Also, in the work monitoring process, the operator's skill level is monitored based on, for example, the operator's alertness and concentration, and the accuracy of the work performed by the robot 14 .

Furthermore, low latency (delay time) is required as a condition regarding the communication status of the operating environment.

<Example of application to remote control of vehicle>
The present technology can also be applied, for example, when performing remote control (remote driving) of a vehicle.

The type of vehicle that is the target of remote control is not particularly limited, but for example, delivery vehicles, fixed-route buses, taxis, vehicles for transport or work in factories or warehouses, etc. are assumed.

FIG. 40 shows an example of a skill table when applying this technology to remote control of a vehicle. This skill table includes skill codes, car classes, and skills.

A skill code is a code for identifying each skill.

The car class code is a code for identifying the car class of the vehicle to be operated. Vehicle classes are classified into, for example, small-sized special vehicles, ordinary vehicles, medium-sized vehicles, large-sized vehicles, large-sized special vehicles, and the like. It should be noted that the classification method of vehicle class is not limited to this.

Skills indicate the skills used in remote control of the vehicle of the target vehicle class. Note that when a plurality of skills are used in remote control of a vehicle of the same class, a different skill code is assigned to each skill.

It should be noted that a possessed skill table having the same configuration as the possessed skill table of FIG. 38 described above is also used when remotely controlling a vehicle.

FIG. 41 shows an example of a reward table showing the reward rank of each operator. This remuneration table includes items of operator code, skill level, and remuneration rank.

The operator code is a code for identifying each operator.

The skill level indicates the skill level of each operator for each vehicle class. This skill level is manually or automatically evaluated, for example, based on the past remote operation (driving operation) history for each vehicle class of each operator. For example, a higher skill level is set for a vehicle class with a larger number of past driving operations or a longer past driving operation time.

The reward rank indicates the reward rank of each operator. The remuneration ranks are classified into, for example, four levels A to D, and are in the order of A, B, C, and D from the highest remuneration rank. For example, a reward to be paid to the operator is set based on this reward rank.

It should be noted that, for example, this technology can also be applied to remote control of flying cars such as eVOLT (electric Vertical Take-Off and Landing). In this case, for example, records corresponding to flight skills are registered in the skill table.

When an operator remotely operates an industrial robot, operator image data (live video) captured by the operator is used as the operator authentication information in the operation qualification monitoring process, for example. For the operator's qualification information, for example, a copy image of the operator's driver's license or the like is used.

Also, in the work monitoring process, the operator's skill level is monitored, for example, based on the operator's alertness and concentration and the safety of driving the robot 14 (vehicle).

Furthermore, low latency is required as a condition regarding the communication status of the operating environment.

<Example of application to personnel placement within a company>
The present technology can be applied, for example, when assigning personnel between a plurality of factories located at remote locations.

For example, if one company has multiple factories in remote locations and each factory manufactures different products, there may be differences in the operating rates of each factory due to factors such as product sales. As a result, for example, there may be a shortage of workers in a factory with a high operating rate, and an excess of workers in a factory with a low operating rate.

In response, by applying this technology, workers in factories with low operating rates can perform work in factories with high operating rates through remote control, thereby increasing or decreasing the number of workers in each factory and relocating them. It is possible to optimize the number of workers in each factory without

<<6. Modification>>
Modifications of the embodiment of the present technology described above will be described below.

<Modified example of matching processing>
For example, the evaluation unit 214 of the management server 15 may cause the operation terminal 12 of each operator to execute a simulator or game simulating actual work to measure the skill level of each operator. Then, the matching unit 211 may select the operator to whom the work is requested based on the measured skill level.

For example, the matching unit 211 of the management server 15 causes the selected operator to execute the above-described simulator or game. The operator may be requested to do the work.

For example, the matching unit 211 may select the operator based on the operator's desired condition in addition to the selection condition.

The operator's desired conditions may include, for example, the type of work desired, remuneration, remuneration system, date and time when the work can be performed, etc.

For example, if the operator desires a predetermined type of work, the matching unit 211 may preferentially assign the desired type of work to the operator.

For example, if the operator desires a higher reward, the matching unit 211 may preferentially allocate to the operator work with a higher reward. For example, as shown in FIG. 42, the matching unit 211 may allocate the work with the highest reward to the operator for each time slot so as to maximize the reward. In this example, work A is assigned in the time slot from 9:00 to 12:00, work B is assigned in the time slot from 13:00 to 15:00, and work B is assigned in the time slot from 16:00 to 19:00. In the band, task C is assigned.

For example, the matching unit 211 may preferentially allocate to the operator the work of the remuneration system desired by the operator. For example, the remuneration system is defined by payment forms such as daily payment, weekly payment, and monthly payment, types of remuneration such as cash, virtual currency, points, and the like.

<Modified example of collaborative work>
The present technology can also be applied, for example, when a plurality of operators work together.

For example, the present technology can also be applied when a different operator remote-controls the robot 14 for each work process to perform one work. In this case, the matching unit 211 of the management server 15 executes the matching process described above for each work process, and selects an operator suitable for each work process.

For example, the present technology can also be applied when a plurality of operators remote-control one robot 14 or a plurality of robots 14 at the same time to jointly perform one task. A single robot 14 is remotely operated by a plurality of operators at the same time, for example, when the robot 14 has a plurality of movable parts 164 and a different operator remotely operates each movable part 164 . In this case, the matching unit 211 of the management server 15 executes the matching process described above and selects a plurality of operators for one task.

When a plurality of operators work together, for example, when the work situation of a certain operator is deteriorating, the intermediary unit 212 of the management server 15 may, for example, delay work or reduce accuracy. When the operator is working, the communication terminal 11 or the operation terminal 12 of another operator may be notified of the worsening of the working situation of the operator. As a result, for example, another operator can support the operator or prompt the operator to improve the work. Also, for example, when a plurality of operators perform work in turn, it becomes possible for the operator in the downstream process to grasp the delay in the work in the upstream process in advance and to deal with it appropriately.

<Modified example of handover process>
For example, the matching unit 211 of the management server 15 determines that it is necessary to change the operator when the intermediation of the remote operation is stopped in the middle of the work by the process of step S218 in FIG. Processing may be performed.

For example, the matching unit 211 executes matching processing again and selects a new operator. Alternatively, the matching unit 211 selects backup candidates in advance. The intermediary unit 212 executes intermediary processing for remote operation between the operation terminal 12 of the new operator and the robot 14 to be operated. This makes it possible to properly select a replacement operator and continue work.

Also, for example, the work may be handed over to a pre-selected backup operator during the operator's break time. Further, when the operator ends the operation due to the end of working hours or the like, the work may be handed over to a pre-selected backup operator. As a result, the robot 14 is prevented from being left in a non-operating state, and the robot 14 can be used efficiently.

In addition, when the takeover process is necessary, for example, the intermediary unit 212 of the management server 15 can be used by a management operator (for example, an operator such as a factory management worker) who manages a plurality of operators to communicate with the communication terminal 11 or the operation The terminal 12 may be notified. As a result, for example, until a new operator takes over or until the operator returns from a break or the like, the managing operator can temporarily perform remote control of the robot 14 on behalf of the operator. becomes possible.

Also, if a handover process is required, for example, the intermediary unit 212 of the management server 15 may notify the requester's client terminal 13 that the operator's handover is required.

<Modified Example Regarding Sharing of Processing in Information Processing System 1>
In the information processing system 1, it is possible to change the allocation of each process as necessary.

For example, the operation terminal 12 may execute part of the processing of the communication terminal 11 described above. For example, the operation terminal 12 may also serve as the communication terminal 11 and execute all the processing of the communication terminal 11 .

For example, the robot 14 may execute part of the processing of the client terminal 13 described above. For example, the robot 14 may also serve as the client terminal 13 and execute all the processing of the client terminal 13 .

For example, the operation terminal 12 may execute part of the processing of the management server 15 described above.

For example, the operation terminal 12 may perform operator authentication processing and notify the management server 15 of the authentication result. Then, the monitoring unit 213 or the like of the management server 15 may use the result of the authentication processing of the operation terminal 12 to determine whether or not the operator has been successfully authenticated.

For example, the operation terminal 12 may monitor the state of the operator and notify the management server 15 of the state of the operator. For example, the operating terminal 12 may monitor the state of the operating environment and the operating environment, and notify the management server 15 of the result of the monitoring process. Then, the monitoring unit 213 of the management server 15 may monitor the work using the result of the monitoring process by the operation terminal 12 .

Note that these monitoring processes may be executed by, for example, another device provided around the operation terminal 12, or may be executed jointly by the operation terminal 12 and another device.

For example, the robot 14 may execute part of the processing of the management server 15 described above. For example, the robot 14 may execute part of the processing of the monitoring unit 213 of the management server 15 . Specifically, for example, the robot 14 may monitor the state, motion, and communication status of the robot 14 and notify the management server 15 of the results of the monitoring process. Then, the monitoring unit 213 of the management server 15 may use the results of the monitoring process by the robot 14 to monitor the work.

Note that these monitoring processes may be executed, for example, by another device provided around the robot 14, or may be jointly executed by the robot 14 and another device.

For example, devices around the robot 14 may capture work image data.

<Other Modifications>
For example, in step S203 of FIG. 10, the intermediation unit 212 acquires from the operation terminal 12 information about the communication status between the operation terminal 12 and the management server 15, and furthermore, based on the communication status, whether to permit remote operation or not. You may make it determine whether or not. For example, the intermediary unit 212 may determine that remote operation is not permitted when the communication environment does not meet the required conditions.

<<7. Other>>
<Computer configuration example>
The series of processes described above can be executed by hardware or by software. When executing a series of processes by software, a program that constitutes the software is installed in the computer. Here, the computer includes, for example, a computer built into dedicated hardware and a general-purpose personal computer capable of executing various functions by installing various programs.

FIG. 43 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

In computer 1000 , CPU (Central Processing Unit) 1001 , ROM (Read Only Memory) 1002 and RAM (Random Access Memory) 1003 are interconnected by bus 1004 .

An input/output interface 1005 is further connected to the bus 1004 . An input unit 1006 , an output unit 1007 , a storage unit 1008 , a communication unit 1009 and a drive 1010 are connected to the input/output interface 1005 .

The input unit 1006 consists of input switches, buttons, a microphone, an imaging device, and the like. The output unit 1007 includes a display, a speaker, and the like. Storage unit 1008 includes a hard disk, a nonvolatile memory, or the like. A communication unit 1009 includes a network interface and the like. A drive 1010 drives a removable medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

In the computer 1000 configured as described above, the CPU 1001 loads, for example, a program recorded in the storage unit 1008 into the RAM 1003 via the input/output interface 1005 and the bus 1004, and executes the program. A series of processes are performed.

The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on removable media 1011 such as package media, for example. Also, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer 1000 , the program can be installed in the storage unit 1008 via the input/output interface 1005 by loading the removable medium 1011 into the drive 1010 . Also, the program can be received by the communication unit 1009 and installed in the storage unit 1008 via a wired or wireless transmission medium. In addition, programs can be installed in the ROM 1002 and the storage unit 1008 in advance.

The program executed by the computer may be a program that is processed in chronological order according to the order described in this specification, or may be executed in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

Also, in this specification, a system means a set of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device housing a plurality of modules in one housing are both systems. .

Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

For example, this technology can take the configuration of cloud computing in which one function is shared by multiple devices via a network and processed jointly.

In addition, each step described in the flowchart above can be executed by a single device, or can be shared by a plurality of devices.

Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device or shared by multiple devices.

<Configuration example combination>
This technique can also take the following configurations.

(1)
an operator monitoring unit that monitors the operational eligibility of the operator;
a work monitoring unit that monitors a work status, which is the status of work performed by the operator remotely controlling the robot using an operation terminal;
An information processing apparatus comprising: an intermediation unit that controls intermediation of remote operation between the operation terminal and the robot based on the operation suitability and the monitoring result of the work situation.
(2)
The information processing apparatus according to (1), wherein the intermediation unit performs at least one of warning the operator and stopping intermediation of the remote operation when the operator does not have the operation qualification. .
(3)
The intermediation unit performs at least one of supporting the operator and stopping intermediation of the remote operation when a problem occurs in the work situation. The information processing according to (1) or (2). Device.
(4)
The operator monitoring unit determines whether or not the operator has the operation qualification based on at least one of a result of authentication processing of the operator and a result of monitoring the state of the operator. The information processing apparatus according to any one of (1) to (3).
(5)
The operator monitoring unit performs personal authentication of the operator as the authentication process, and confirms whether or not the operator has the qualification when the work requires qualification. The information processing apparatus according to (4).
(6)
The information processing apparatus according to (4) or (5), wherein the operator monitoring unit executes the authentication process while the robot is being remotely operated by the operation terminal.
(7)
Any one of (4) to (6) above, wherein the operator monitoring unit monitors at least one of the operator's alertness, concentration, tension, and physical condition as the operator's state. The information processing device described.
(8)
The work monitoring unit monitors, as the work status, at least one of the operator's skill level, whether or not the operator has performed a prohibited act, and the operator's operating environment. The information processing device according to any one of (7).
(9)
The information processing apparatus according to (8), wherein the work monitoring unit monitors the operator's skill level based on at least one of the operator's motion and the robot's motion.
(10)
The information processing apparatus according to (8) or (9), wherein the prohibited action is an action that reduces at least one of progress and quality of the work.
(11)
The information processing apparatus according to any one of (8) to (10), wherein the operating environment includes at least one of the operating terminal and a communication status of the operating terminal.
(12)
The mediation unit according to any one of (1) to (11) above, when the work situation is deteriorating, notifying a co-worker who works jointly with the operator that the work situation is deteriorating. Information processing equipment.
(13)
The intermediation unit mediates remote operation between an operation terminal of a substitute operator and the robot when intermediation of the remote operation of the operator is stopped based on the result of monitoring the operation suitability and the work situation. The information processing apparatus according to any one of (1) to (12).
(14)
The intermediary unit connects the operation terminal and the robot via a network, transmits an operation signal from the operation terminal to the robot, and transmits a work image of the operation of the robot to the operation terminal. The information processing apparatus according to any one of (1) to (13) above.
(15)
The information processing apparatus according to (14), wherein the intermediation unit generates an image in which confidential information is hidden in the work image, and transmits the image to the operation terminal.
(16)
The information processing apparatus according to (14) or (15), wherein the intermediation unit generates an image presenting only a region to be remotely operated in the work image, and transmits the image to the operation terminal.
(17)
The information processing apparatus according to any one of (1) to (16), further comprising a matching unit that selects the operator based on at least one of the operator's ability and operating environment.
(18)
The information processing apparatus according to (17), further comprising an evaluation unit that updates the evaluation of the operator's ability based on the work content of the operator.
(19)
Monitor the operation suitability of the operator and the work status, which is the status of the work performed by the operator remotely controlling the robot using the operation terminal,
An information processing method, comprising: controlling mediation of remote operation between the operation terminal and the robot based on the result of monitoring the operation suitability and the work status.
(20)
robot and
an operation terminal used by an operator to remotely operate the robot;
an information processing device that mediates remote control between the operation terminal and the robot;
The information processing device is
an operator monitoring unit that monitors the operational eligibility of the operator;
a work monitoring unit that monitors a work status, which is a status of work performed by the operator remotely controlling the robot using the operation terminal;
An information processing system comprising: an intermediation unit that controls intermediation of remote operation between the operation terminal and the robot based on the operational suitability and the result of monitoring the work status.

It should be noted that the effects described in this specification are only examples and are not limited, and other effects may be provided.

1 information processing system, 11-1 to 11-m communication terminals, 12-1 to 12-n operation terminals, 13-1 to 13-p client terminals, 14-1 to 14-q robots, 15 management server, 21 network, 103 control unit, 122 detection unit, 123 control unit, 124 output unit, 143 control unit, 162 detection unit, 163 control unit, 164 movable unit, 202 control unit, 211 matching unit, 212 mediation unit, 213 monitoring unit, 214 evaluation unit, 215 learning unit, 216 information processing unit, 221 operator monitoring unit, 222 work monitoring unit, 301 information processing system, 311a, 311b mediation device, 351 information processing system, 361a, 361b mediation device

Claims (20)

1. an operator monitoring unit that monitors the operational eligibility of the operator;
   a work monitoring unit that monitors a work status, which is the status of work performed by the operator remotely controlling the robot using an operation terminal;
   An information processing apparatus comprising: an intermediation unit that controls intermediation of remote operation between the operation terminal and the robot based on the operation suitability and the monitoring result of the work situation.
2. The information processing apparatus according to claim 1, wherein, when the operator does not have the operation qualification, the mediation unit performs at least one of warning the operator and stopping mediation of the remote operation.
3. The information processing apparatus according to claim 1, wherein, when a problem occurs in the work situation, the intermediation unit performs at least one of supporting the operator and stopping intermediation of the remote operation.
4. The operator monitoring unit determines whether or not the operator has the operation qualification based on at least one of a result of authentication processing of the operator and a result of monitoring the state of the operator. The information processing apparatus according to claim 1, wherein the determination is made.
5. The operator monitoring unit performs personal authentication of the operator as the authentication process, and confirms whether or not the operator has the qualification when the work requires qualification. The information processing apparatus according to claim 4.
6. The information processing apparatus according to claim 4, wherein the operator monitoring unit executes the authentication process while the robot is being remotely operated by the operation terminal.
7. The information processing apparatus according to claim 4, wherein the operator monitoring unit monitors at least one of alertness, concentration, tension, and physical condition of the operator as the operator's state.
8. 2. The work monitoring unit according to claim 1, wherein the work monitoring unit monitors at least one of the operator's skill level, whether or not the operator has performed a prohibited act, and the operator's operating environment as the work status. information processing equipment.
9. The information processing apparatus according to claim 8, wherein the work monitoring unit monitors the skill level of the operator based on at least one of the motion of the operator and the motion of the robot.
10. The information processing apparatus according to claim 8, wherein the prohibited action is an action that reduces at least one of progress and quality of the work.
11. The information processing apparatus according to claim 8, wherein the operating environment includes at least one of the operating terminal and a communication status of the operating terminal.
12. The information processing apparatus according to claim 1, wherein, when the work situation is deteriorating, the intermediation unit notifies a co-worker who works together with the operator that the work situation is deteriorating.
13. The intermediation unit mediates remote operation between an operation terminal of a substitute operator and the robot when intermediation of the remote operation of the operator is stopped based on the result of monitoring the operation suitability and the work situation. The information processing apparatus according to claim 1 .
14. The intermediary unit connects the operation terminal and the robot via a network, transmits an operation signal from the operation terminal to the robot, and transmits a work image of the operation of the robot to the operation terminal. The information processing apparatus according to claim 1, wherein the transmission is performed to.
15. The information processing apparatus according to claim 14, wherein the intermediary unit generates an image in which confidential information is hidden in the work image and transmits the image to the operation terminal.
16. 15. The information processing apparatus according to claim 14, wherein the intermediary unit generates an image presenting only a remote operation target area in the work image, and transmits the image to the operation terminal.
17. The information processing apparatus according to claim 1, further comprising a matching unit that selects the operator based on at least one of the operator's ability and operating environment.
18. 18. The information processing apparatus according to claim 17, further comprising an evaluation unit that updates the evaluation of the operator's ability based on the work content of the operator.
19. Monitor the operation suitability of the operator and the work status, which is the status of the work performed by the operator remotely controlling the robot using the operation terminal,
    An information processing method, comprising: controlling mediation of remote operation between the operation terminal and the robot based on the result of monitoring the operation suitability and the work status.
20. robot and
    an operation terminal used by an operator to remotely operate the robot;
    an information processing device that mediates remote control between the operation terminal and the robot;
    The information processing device is
    an operator monitoring unit that monitors the operational eligibility of the operator;
    a work monitoring unit that monitors a work status, which is a status of work performed by the operator remotely controlling the robot using the operation terminal;
    An information processing system comprising: an intermediation unit that controls intermediation of remote operation between the operation terminal and the robot based on the operational suitability and the result of monitoring the work status.

Images