JP2019139733A - Information processing system, information processing device, information processing method and program - Google Patents

Abstract

To make it possible to conduct learning more readily, avoiding collisions more surely.SOLUTION: An information processing device includes: measurement means that measures a degree of a risk of contact with a subject reflected in a moving picture by machine learning from the moving picture transmitted from a robot autonomously traveling on a floor in a building, and capturing the moving picture at a traveling destination; display control means that controls a display to display means of the moving picture; acquisition means that acquires an instruction of stopping the robot by an operation from a monitor monitoring the displayed moving picture; command means that, when the measured degree of the risk is more than a prescribed threshold, or the stoppage is instructed from the monitor, command the robot to stop; and recording control means that, when the stoppage of the robot is instructed from the monitor, causes recording means to record the moving picture of a time with a prescribed length including a time when the stoppage is instructed.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an information processing system, an information processing apparatus, an information processing method, and a program, and more particularly, to an information processing system, an information processing apparatus, an information processing method, and a program used for controlling the running of a robot.

  Robots have come to be used for cleaning floors in buildings. When cleaning areas with people, such as public buildings and hospitals, collisions with people must be avoided. Various proposals have been made for avoiding a collision with a person on a traveling vehicle.

  In a vehicle running, there is a type in which whether or not an object is a pedestrian is determined from an image by using a neural network, and a collision risk is determined (for example, see Patent Document 2).

  Some images are used to determine whether a pedestrian is entering the road using a neural network (see, for example, Patent Document 2).

JP 2007-148835 A JP 2008-21034 A

  However, there is a possibility of mistakes only by the discrimination by the machine. Particularly in hospitals and the like, it is necessary to reliably avoid collisions with patients whose movement may be restricted. In addition, machine learning is often used for the determination of the collision risk. However, it is necessary to prepare learning data to learn and to make a more reliable determination, which takes time.

  The present invention has been made in view of such a situation, and makes it possible to more easily avoid collision and more easily learn.

  An information processing system according to a first aspect of the present invention is an information processing system that includes a robot that autonomously travels on a floor in a building and captures a moving image of the travel destination, and an information processing device that controls the robot. A measuring means for measuring the degree of danger of contact between the subject in the moving image and the robot by machine learning from the moving image, a display control means for controlling the display on the moving image display means, An acquisition means for acquiring an instruction to stop the robot by an operation from the monitoring person who is monitoring the displayed moving image, and when the measured degree of danger is equal to or greater than a predetermined threshold or when the monitoring person stops. When instructed, the instruction means for instructing the robot to stop, and when the supervisor instructs the stop of the robot, the recording means records a moving image for a predetermined length of time including the time when the stop is instructed. Recording system And means.

  From the moving image captured by each of the plurality of robots, the measuring unit is caused to measure the degree of danger of contact with the subject reflected in the moving image for each of the plurality of robots by machine learning. The degree of danger measured for any of the plurality of robots by causing the command means to obtain an instruction to stop any of the plurality of robots by an operation from the supervisor who is monitoring the displayed moving image Is equal to or greater than a predetermined threshold value, or when a stop is instructed by any of a plurality of robots, the robot is instructed to stop if the degree of danger is equal to or greater than a predetermined threshold value, When the recording controller is instructed to stop any of the plurality of robots by the monitoring person, it is a moving image transmitted from the robot instructed to stop. Te, can be made to be recorded in the recording means a moving image of a predetermined length of time, including when the stop is instructed.

  It is possible to cause the measurement unit to learn using data indicating that stop has been instructed and a moving image recorded in the recording unit.

  The information processing apparatus according to the second aspect of the present invention autonomously travels on a floor in a building and converts a moving image transmitted from a robot that captures a moving image to a moving image by machine learning. From the measuring means for measuring the degree of danger of contact between the subject being photographed and the robot, the display control means for controlling the display of the moving image on the display means, and the monitor who is monitoring the displayed moving image An acquisition means for acquiring an instruction to stop the robot by the operation of the above, and an instruction means for instructing the robot to stop when the measured degree of danger is equal to or greater than a predetermined threshold or when a stop is instructed by a supervisor; And a recording control unit that causes the recording unit to record a moving image for a predetermined length of time including when the stop is instructed by the supervisor.

  Let the measuring means measure the degree of danger of contact with the subject in the moving image for each of the plurality of robots by using machine learning from the moving images transmitted from each of the plurality of robots. , By acquiring an instruction to stop any of the plurality of robots by an operation from the monitoring person who is monitoring the displayed moving image, and instructing the command means to detect the risk measured for any of the plurality of robots. When the degree is greater than or equal to a predetermined threshold or when a stop is instructed by any of a plurality of robots, the robot with the degree of danger exceeding the predetermined threshold or instructed to stop is instructed to stop. When the stop is instructed to any one of the plurality of robots by the recording control means, the moving image is transmitted from the robot instructed to stop. Te, can be recorded in the recording means a moving image of a predetermined length of time, including when the stop is instructed.

  It is possible to cause the measurement unit to learn using data indicating that stop has been instructed and a moving image recorded in the recording unit.

  The information processing method according to the third aspect of the present invention is a method of autonomously running on a floor in a building and converting a moving image transmitted from a robot that captures a moving image to a moving image by machine learning. From a measurement step for measuring the degree of danger of contact between the subject being photographed and the robot, a display control step for controlling the display of the moving image on the display means, and a supervisor who is monitoring the displayed moving image An acquisition step for acquiring an instruction to stop the robot by the operation of, and a command step for instructing the robot to stop when the measured degree of danger is equal to or greater than a predetermined threshold or when a stop is instructed by a supervisor; A recording control step of causing the recording means to record a moving image for a predetermined length of time including when the stop is instructed by the supervisor.

  The program according to the fourth aspect of the present invention is captured on a moving image by machine learning from a moving image transmitted from a robot that autonomously travels on a floor in a building and picks up a moving image. Measurement step for measuring the degree of danger of contact between the subject and the robot, a display control step for controlling the display of the moving image on the display means, and an operation from the supervisor who is monitoring the displayed moving image An acquisition step for acquiring an instruction to stop the robot, a command step for instructing the robot to stop if the measured degree of danger is equal to or greater than a predetermined threshold value, or a stop is instructed by a supervisor, Information processing including a recording control step for causing the recording means to record a moving image for a predetermined length of time including when the stop is instructed by a person. To perform.

  As described above, according to the present invention, it is possible to more easily avoid a collision and learn more easily.

It is a figure which shows an example of a structure of the cleaning system. It is a figure which shows an example of the external shape of the cleaning robot. 3 is a block diagram illustrating an example of a hardware configuration of a cleaning robot 22. FIG. It is a block diagram explaining an example of the structure of the function implement | achieved when the control module 42 runs a control program. 2 is a block diagram illustrating an example of a hardware configuration of an AI server 21. FIG. It is a block diagram which shows an example of a structure of the function implement | achieved when the AI server 21 runs a program. It is a flowchart explaining the process of starting. It is a flowchart explaining the detail of the process of traveling control. 5 is a flowchart for explaining a process of running control of the cleaning robot 22 by the AI server 21. 5 is a flowchart for explaining a process of running control of the cleaning robot 22 by the AI server 21. It is a flowchart explaining the detail of the measurement process of a risk degree. 6 is a diagram showing an example of a moving image displayed on the touch panel monitor 25. FIG. 6 is a diagram showing an example of a moving image displayed on the touch panel monitor 25. FIG. 6 is a diagram showing an example of a moving image displayed on the touch panel monitor 25. FIG. 6 is a diagram showing an example of a moving image displayed on the touch panel monitor 25. FIG. It is a flowchart explaining the process according to the instruction | indication from a monitoring person. 6 is a diagram showing an example of a moving image displayed on the touch panel monitor 25. FIG. 6 is a diagram showing an example of a moving image displayed on the touch panel monitor 25. FIG. It is a flowchart explaining the process of completion | finish. It is a flowchart explaining the process of scheduling of additional learning. It is a flowchart explaining the process of additional learning.

  Embodiments of the present invention will be described below with reference to FIGS.

  FIG. 1 is a diagram illustrating an example of the configuration of the cleaning system 11. The cleaning system 11 cleans indoor floors of buildings such as hospitals and public facilities. The cleaning system 11 runs autonomously, controls a cleaning robot that performs cleaning, and causes the cleaning robot to perform cleaning. The cleaning system 11 includes an AI (Artificial Intelligence) server 21, cleaning robots 22-1 and 22-2, a hub 23, access points 24-1 to 24-3, and touch panel monitors 25-1 and 25-2. .

  The AI server 21 measures the degree of danger in the scene of the moving image from the moving image. Here, the scene of a moving image is a segment in the moving image, which is a segment where a predetermined place where a predetermined subject in a predetermined state or a predetermined motion is present is photographed. For example, the AI server 21 measures the degree of danger of contact between the subject shown in the moving image and the cleaning robots 22-1 and 22-2 from the moving image. The AI server 21 instructs the cleaning robots 22-1 and 22-2 to stop or decelerate when the measured degree of danger is equal to or greater than a predetermined threshold. The cleaning robots 22-1 and 22-2 travel autonomously and clean the floor surface. The cleaning robots 22-1 and 22-2 capture a moving image of the traveling destination.

  That is, the AI server 21 processes the moving image imaging data transmitted from the cleaning robots 22-1 and 22-2 that captures the moving image of the traveling destination, and makes contact with the subject reflected in the moving image. The degree of danger is measured, and if the measured degree of danger is equal to or greater than a predetermined threshold, the cleaning robots 22-1 and 22-2 are instructed to stop or decelerate.

  The hub 23 is a line concentrator in a network that employs a star-type network topology (physical topology). The hub 23 connects the AI server 21 and each of the access points 24-1 to 24-3.

  The access points 24-1 to 24-3 are wireless communication devices for connecting to the cleaning robots 22-1 and 22-2, respectively, by wireless communication of a predetermined standard of a wireless local area network (LAN). . The cleaning robots 22-1 and 22-2 capture the moving image of the traveling destination, and transmit imaging data for displaying the moving image on the AI server 21 via the hub 23 and the access points 24-1 to 24-3. To do.

  The touch panel monitors 25-1 and 25-2 have a display function and an input function. The touch panel monitors 25-1 and 25-2 are also referred to as touch screens, and display predetermined images based on image data from the AI server 21. In addition, a touch panel which is a position input device that transmits a displayed image and inputs a touched position is provided on the front surface of each screen of the touch panel monitors 25-1 and 25-2. When the touch panels of the touch panel monitors 25-1 and 25-2 are touched and operated, a signal indicating the touched position is sent to the AI server 21.

  The AI server 21 displays moving images on the touch panel monitors 25-1 and 25-2 based on the imaging data transmitted from the cleaning robots 22-1 and 22-2. The supervisor who monitors the running of the cleaning robots 22-1 and 22-2 monitors the moving images displayed on the touch panel monitors 25-1 and 25-2. When the supervisor determines from the moving images displayed on the touch panel monitors 25-1 and 25-2 that there is a risk that the cleaning robot 22-1 or 22-2 will come into contact with a person such as a patient or a passerby, Touching the touch panel monitor 25-1 or 25-2, the cleaning robot 22-1 or 22-2 is stopped or decelerated.

  Since the cleaning robot 22 travels indoors in buildings such as hospitals and public facilities, the speed is only about walking, and is slower than a car traveling on a road. If a stop or deceleration is instructed by looking at, dangers such as contact with the subject can be avoided.

  Hereinafter, the cleaning robots 22-1 and 22-2 are simply referred to as the cleaning robot 22 when it is not necessary to distinguish them individually. Hereinafter, when it is not necessary to individually distinguish the access points 24-1 to 24-3, they are simply referred to as access points 24. Further, hereinafter, when it is not necessary to distinguish between the touch panel monitors 25-1 and 25-2, they are simply referred to as a touch panel monitor 25.

  Although two cleaning robots 22 are shown in FIG. 1, any number of cleaning robots 22 or a plurality of cleaning robots 22 can be used. Further, although three access points 24 are shown in FIG. 1, an arbitrary number of access points 24 can be provided. Furthermore, although two touch panel monitors 25 are shown in FIG. 1, one or a plurality of arbitrary numbers of touch panel monitors 25 can be provided.

  FIG. 2 is a diagram illustrating an example of the outer shape of the cleaning robot 22. The main body 41 of the cleaning robot 22 is provided with wheels and motors for traveling, a brush for cleaning, a motor for driving the brushes, and a power source such as a battery. A control module 42 is provided in front of the upper surface of the main body 41 of the cleaning robot 22. The control module 42 is composed of a tablet personal computer or a dedicated computer. The control module 42 communicates with the AI server 21 via the access point 24 and the hub 23. The control module 42 sends various commands such as a stop to the main body 41 to control the operation of the main body 41.

  A camera 43 is provided on the front side of the main body 41 of the cleaning robot 22. The camera 43 captures a moving image in front of the cleaning robot 22. That is, the camera 43 captures a moving image where the cleaning robot 22 travels. The moving image image data captured by the camera 43 is transmitted to the AI server 21 by the control module 42.

  Next, the hardware configuration of the cleaning robot 22 will be described.

  FIG. 3 is a block diagram illustrating an example of a hardware configuration of the cleaning robot 22. The cleaning robot 22 includes a control module 42, a camera 43, a travel operation control unit 51, a sensor 52, a drive unit 53, a motor 54, wheels 55, a cleaning operation control unit 61, a sensor 62, a drive unit 63, a motor 64, and a brush 65. Consists of. The control module 42 includes a speaker 71, a display unit 72, a touch panel 73, a communication unit 74, and a storage unit 75. The speaker 71 is a so-called loudspeaker, and outputs (releases) various sounds. For example, the speaker 71 outputs a voice indicating a message to a person who is traveling indoors in a building such as a hospital or public facility. The display unit 72 includes a liquid crystal display device, an organic EL (Electroluminescence) display device, and the like, and displays various images and characters. For example, the display unit 72 displays an image or a character indicating a message for a person who is traveling indoors in a building such as a hospital or public facility. The touch panel 73 is provided on the display surface of the display unit 72, transmits an image displayed on the display unit 72, and inputs a touched position. For example, the touch panel 73 inputs an instruction from an operator at the time of activation. The communication unit 74 performs wireless communication with the access point 24 in accordance with a predetermined standard for wireless LAN. The storage unit 75 includes a nonvolatile memory such as a flash memory, and stores various data.

  The traveling operation control unit 51 includes a dedicated processor and the like, and controls the traveling of the cleaning robot 22 by controlling the driving unit 53 by a signal from the sensor 52 or the control module 42. The sensor 52 includes a distance sensor using infrared rays or ultrasonic waves, a three-dimensional acceleration sensor, and the like. The sensor 52 detects the posture of the cleaning robot 22 and surrounding people and objects, and outputs a signal indicating the detection result to the traveling operation control unit. 51. The drive unit 53 is an inverter or the like, and drives the motor 54 by a power source such as a battery (not shown). The motor 54 rotates wheels 55 provided at the four corners of the bottom surface of the cleaning robot 22 or changes the direction of the wheels 55.

  The cleaning operation control unit 61 includes a dedicated processor and the like, and controls the cleaning operation by the cleaning robot 22 by controlling the drive unit 63 by a signal from the sensor 62. The sensor 62 is composed of a microphone that senses the sound during cleaning, a piezo sensor that detects the presence or amount of dust, an optical sensor, and the like, and senses the state of the floor dust and cleaning operation and indicates the detection result. A signal is supplied to the cleaning operation control unit 61. The drive unit 63 is an inverter or the like, and drives the motor 64 by a power source such as a battery (not shown). The motor 64 rotates a brush 65 provided on the bottom surface of the cleaning robot 22 or operates a suction device that sucks in dust.

  Next, the function of the control module 42 will be described.

  FIG. 4 is a block diagram illustrating an example of a functional configuration realized by the control module 42 executing a control program. When the control module 42 executes the control program, an imaging control unit 81, an encoding unit 82, a communication control unit 83, a flag setting unit 84, a storage control unit 85, an instruction unit 86, and an audio output control unit 87 are realized.

  The imaging control unit 81 controls imaging of moving images by the camera 43. The encoding unit 82 encodes a moving image captured by the camera 43 using a method such as H.262 / MPEG (Moving Picture Experts Group) -2 Part 2 or MPEG-4 AVC (Advanced Video Coding) /H.264. To generate imaging data. The communication control unit 83 controls communication by the communication unit 74. The communication control unit 83 includes a transmission control unit 91 and a reception control unit 92. The transmission control unit 91 controls transmission of various data to the AI server 21 via the access point 24 such as transmission of moving image captured data by controlling the communication unit 74. The reception control unit 92 controls reception of various data transmitted from the AI server 21 via the access point 24, such as reception of a control signal instructing stoppage, by controlling the communication unit 74.

  The flag setting unit 84 sets various flags such as a stop flag indicating that the cleaning robot 22 is stopped by setting a bit of data at a predetermined address in the storage unit 75 to 0 or 1. The storage control unit 85 controls storage of audio data in the storage unit 75. The instruction unit 86 instructs the travel operation control unit 51 to stop or decelerate according to a signal according to an operation from the touch panel 73 or a control signal for instructing a stop from the AI server 21. Instruction unit 86 includes a determination unit 93. The determination unit 93 determines whether the control signal transmitted from the AI server 21 is an instruction to stop or an instruction to decelerate.

  When the voice output control unit 87 receives a control signal transmitted from the AI server 21 and instructing voice output, the voice output control unit 87 reads the voice data stored in the storage unit 75 and reads the voice data. Is output to the speaker 71.

  Next, the hardware of the AI server 21 will be described.

  FIG. 5 is a block diagram illustrating an example of a hardware configuration of the AI server 21.

  In the AI server 21, a CPU (Central Processing Unit) 121, a ROM (Read Only Memory) 122, and a RAM (Random Access Memory) 123 are connected to each other by a bus 124.

  An input / output interface 125 is further connected to the bus 124. The input / output interface 125 includes an input unit 126 including a keyboard or a mouse connected as necessary, an output unit 127 including a display connected as necessary, and a recording unit including a hard disk and a non-volatile memory. 128, a communication unit 129 including a network interface, and a drive 130 for driving a removable medium 131 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory are connected.

  The touch panel monitor 25 is connected to the input / output interface 125 via a GPU (Graphics Processing Unit) (not shown).

  In the AI server 21 (computer) configured as described above, the CPU 121 loads, for example, a program recorded in the recording unit 128 to the RAM 123 via the input / output interface 125 and the bus 124 and executes the program. Thus, a series of processes described later is performed.

  The programs executed by the AI server 21 (CPU 121) are, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, Alternatively, it is recorded on a removable medium 131 which is a package medium made of a semiconductor memory, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the computer by mounting the removable medium 131 in the drive 130 and recording it in the recording unit 128 via the input / output interface 125. Further, the program can be installed in a computer by being received by the communication unit 129 via a wired or wireless transmission medium and recorded in the recording unit 128. In addition, the program can be installed in the computer in advance by storing or recording the program in the ROM 122 or the recording unit 128 in advance.

  That is, in the AI server 21 (computer), the functions described later with reference to FIG. 6 are realized by executing the installed program.

  FIG. 6 is a block diagram illustrating an example of a functional configuration realized by the AI server 21 executing a program. When the AI server 21 executes the program, the risk measuring unit 161, the imaging data buffer 162, the decoding unit 163, the learning data recording control unit 164, the learning scheduler 165, the determination unit 166, the command unit 167, the communication control unit 168, An input control unit 169 and a display control unit 170 are realized. The danger measuring unit 161 measures the degree of danger of a moving image scene from the moving image captured data transmitted from the cleaning robot 22 by machine learning. Further, the danger measuring unit 161 measures the degree of danger of contact with the subject in the moving image by machine learning from the moving image captured data transmitted from the cleaning robot 22. Further, the risk measuring unit 161 measures the distance from the cleaning robot 22 to the subject and the speed of the subject with respect to the cleaning robot 22. The risk level measurement unit 161 includes an object detection AI unit 171, an image processing unit 172, a risk level measurement AI unit 173, and a drawing unit 174.

  The risk measuring unit 161 may cause a GPU (not shown) to perform calculation and measure the degree of danger of a moving image scene. Further, the danger measuring unit 161 may cause an AI chip (not shown) to perform calculation to measure the degree of danger of a moving image scene. Note that the risk level measuring units 161 may be realized by the number of cleaning robots 22. For example, as illustrated in FIG. 1, when the cleaning robots 22-1 and 22-2 perform cleaning in the cleaning system 11, the AI server 21 that executes the program is for the cleaning robot 22-1 although illustration is omitted. The risk measuring unit 161-1 and the risk measuring unit 161-2 for the cleaning robot 22-2 are realized.

  The object detection AI unit 171 determines whether or not a subject is a person or something other than a person in a moving image by machine learning using a neural network from moving image captured data captured by the cleaning robot 22. And the position of the subject in the moving image is detected.

  For example, the object detection AI unit 171 performs object detection using deep learning, such as YOLO (You Only Look Once), R-CNN (Regions with Convolutional Neural Network), Faster R-CNN or FCN (Fully Convolutional Network). By machine learning using a convolutional neural network, it is detected whether the subject is a person or something other than a person, and the position of the subject in the moving image is detected. For example, the object detection AI unit 171 employing YOLO infers the class probability and the bounding box coordinates of the object from the entire moving image. For example, the object detection AI unit 171 adopting R-CNN inputs an image in the object region candidate obtained from the selective search method to a CNN (Convolutional Neural Network) that has been learned in advance, and the image feature And the extracted image features are input to a linear support vector machine to predict the object class.

  The object detection AI unit 171 uses the learned model to detect whether the subject is a person or something other than a person, and detects the position of the subject in the moving image.

  In addition, the object detection AI unit 171 causes a GPU (not shown) to perform calculation, detects whether the subject is a person or something other than a person in the moving image, and The position of the subject may be detected. The object detection AI unit 171 causes an AI chip (not shown) to perform calculation to detect whether the subject is a person or a person other than a person in the moving image. Alternatively, the position of the subject may be detected. Furthermore, the object detection AI units 171 may be realized by the number of cleaning robots 22. For example, as illustrated in FIG. 1, when the cleaning robots 22-1 and 22-2 perform cleaning in the cleaning system 11, the AI server 21 that executes the program is for the cleaning robot 22-1 although illustration is omitted. The object detection AI unit 171-1 and the object detection AI unit 171-2 for the cleaning robot 22-2 are realized.

  The image processing unit 172 obtains the distance from the cleaning robot 22 to the subject and the speed of the subject relative to the cleaning robot 22 from the moving image and the position of the subject in the moving image by image processing. Further, when the object detection AI unit 171 detects that the subject is a person by image processing, the image processing unit 172 obtains the orientation of the subject with respect to the cleaning robot 22 or the orientation of the face of the subject with respect to the cleaning robot 22. For example, the image processing unit 172 obtains the distance from the cleaning robot 22 to the subject by plane projection conversion or resection processing from the moving image and the position of the subject in the moving image. For example, the image processing unit 172 obtains the speed of the subject with respect to the cleaning robot 22 from the moving image and the position of the subject in the moving image by the background difference method or the optical flow.

  For example, the image processing unit 172 detects the six points of the left eye end point, right eye end point, nose apex, left mouth end, right mouth end, and chin tip from the feature points (regions) such as the eyes, nose and mouth, By obtaining the rotation angle, the orientation of the subject's face relative to the cleaning robot 22 is obtained. Further, for example, the image processing unit 172 extracts a feature amount of the subject's body in the moving image, and detects the orientation of the subject's body from the normalized feature amount using a support vector machine (SVM) and a class identifier.

  Note that the object detection AI unit 171 or the image processing unit 172 determines whether the human subject is a healthy person, a patient, or a child, whether the human subject is talking, and whether the human subject is a crutch. It may be detected whether or not the subject is a person and whether or not the subject as a person has an infusion stand.

  The risk measurement AI unit 173 detects the moving image, the position of the subject in the moving image, the distance from the cleaning robot 22 to the subject, and the cleaning robot 22 as a result of detecting whether the subject is a person or something other than a person. The degree of danger of a moving image scene is measured from the speed of the subject by machine learning using a neural network. In addition, the risk measurement AI unit 173 detects the moving image, the position of the subject in the moving image, the distance from the cleaning robot 22 to the subject, and the cleaning robot as a result of detecting whether the subject is a person or something other than a person. The degree of danger of contact between the cleaning robot 22 and the subject is measured from the speed of the subject with respect to 22 by machine learning using a neural network.

  For example, the risk measurement AI unit 173 detects whether the subject is a person or something other than a person by machine learning using a convolutional neural network such as Deep CNN. From the distance from the subject 22 to the subject and the speed of the subject with respect to the cleaning robot 22, the degree of danger of the moving image scene is measured. For example, the risk measurement AI unit 173 detects whether the subject is a person or something other than a person by machine learning using a convolutional neural network such as Deep CNN. The degree of danger of contact between the cleaning robot 22 and the subject is measured from the distance from the subject 22 to the subject and the speed of the subject with respect to the cleaning robot 22.

  The degree-of-risk measurement AI unit 173 measures the degree of danger of the moving image scene and the degree of danger of contact between the cleaning robot 22 and the subject using the learned model.

  The drawing unit 174 writes a character or an image indicating the degree of danger of the moving image scene in the moving image. The drawing unit 174 writes a character or an image indicating the degree of danger of contact between the cleaning robot 22 and the subject in the moving image. That is, the drawing unit 174 displays, in the moving image imaging data, characters or images that indicate the degree of danger of the moving image scene and characters or images that indicate the degree of danger of contact between the cleaning robot 22 and the subject. Write. More specifically, the drawing unit 174 detects, in the moving image, the position of the subject in the moving image, the distance from the cleaning robot 22 to the subject, the cleaning, as a result of detecting whether the subject is a person or something other than a person. Characters or images indicating the speed of the subject with respect to the robot 22 or the degree of danger of the moving image scene or the degree of danger of contact between the cleaning robot 22 and the subject are drawn.

  Further, the drawing unit 174 writes a character or an image indicating an instruction to the cleaning robot 22 such as stop or deceleration according to the degree of danger of the scene of the moving image or the degree of danger of contact with the subject in the moving image.

  The imaging data buffer 162 temporarily stores imaging data from the cleaning robot 22 using the storage area of the RAM 123 or the recording area of the recording unit 128. For example, the imaging data buffer 162 is a ring buffer. The decoding unit 163 decodes the imaging data from the cleaning robot 22 and encoded by the encoding unit 82. The learning data recording control unit 164 acquires the imaging data from the imaging data buffer 162 when the supervisor instructs the stop or deceleration of the cleaning robot 22 by an operation on the touch panel monitor 25, and indicates when the stop is instructed. The moving image pickup data of a predetermined length including the time is recorded in the recording unit 128. The learning scheduler 165 reserves execution of learning by the risk measuring unit 161 using moving image captured data recorded in the recording unit 128. The learning scheduler 165 makes a reservation by scheduling a learning job of the risk measuring unit 161. In addition, the learning scheduler 165 determines whether or not learning can be executed and learning in the scheduling of the learning job of the risk measuring unit 161 in accordance with an instruction by an operation to the touch panel monitor 25 or the input unit 126 from a supervisor or an operator. Determine the time of execution.

  The determination unit 166 determines whether to instruct the cleaning robot 22 to stop or whether to instruct the cleaning robot 22 to decelerate based on the degree of danger of contact between the cleaning robot 22 and the subject measured by the risk measurement AI unit 173. Determine whether or not. The command unit 167 instructs the cleaning robot 22 to stop, decelerate, or accelerate by transmitting a control signal from the communication unit 129 to the cleaning robot 22. The communication control unit 168 controls communication by the communication unit 129. Communication control unit 168 includes a transmission control unit 175 and a reception control unit 176. The transmission control unit 175 controls transmission of various data to the cleaning robot 22 via the access point 24 such as transmission of a control signal instructing stop by controlling the communication unit 129. The reception control unit 176 controls reception of various data transmitted from the cleaning robot 22 via the access point 24, such as reception of moving image imaging data, by controlling the communication unit 129.

  The input control unit 169 controls the touch panel of the touch panel monitor 25 and acquires a signal corresponding to the operation transmitted from the touch panel monitor 25 when the monitor operates the touch panel of the touch panel monitor 25. The display control unit 170 controls the monitor of the touch panel monitor 25 and displays the moving image transmitted from the cleaning robot 22 on the touch panel monitor 25.

  Next, the process performed in the cleaning system 11 is demonstrated with reference to a flowchart.

  FIG. 7 is a flowchart for explaining the activation process. In step S101, the control module 42 is activated by the operation of the supervisor or operator. In step S <b> 102, the control module 42 connects to the AI server 21. In step S <b> 201, the AI server 21 is connected to the control module 42. For example, the communication control unit 83 of the control module 42 establishes a connection with the communication unit 129 controlled by the communication control unit 168 via the hub 23 and the access point 24 via the transmission control protocol (TCP). Let The control module 42 and the AI server 21 may be connected by UDP (User Datagram Protocol).

  In step S <b> 103, the determination unit 93 of the instruction unit 86 acquires status information indicating whether or not a connection is established from the communication control unit 83, refers to the status information, and the control module 42 sends the information to the AI server 21. Determine if connected. When it is determined in step S103 that the control module 42 is not connected to the AI server 21, the procedure proceeds to step S104, and the determination unit 93 starts from the procedure of step S102 by a timer built in the control module 42. It is determined whether or not a timeout has occurred, for example, depending on whether or not one second has elapsed. For example, in step S104, the determination unit 93 determines that a time-out has not occurred if one second has not elapsed since the procedure in step S102, and determines that a time-out has occurred if one second or more has elapsed since the procedure in step S102. To do.

  If it is determined in step S104 that the timeout has not occurred, the procedure returns to step S103, and the above-described processing is repeated. If it is determined in step S104 that the time-out has occurred, the procedure proceeds to step S105, and the determination unit 93 is a stop flag set by the flag setting unit 84 and indicates that the cleaning robot 22 is stopped. It is determined whether or not a flag is set. If it is determined in step S105 that the stop flag is set, the procedure returns to step S102, and the above-described processing is repeated.

  If it is determined in step S105 that the stop flag is not set, the procedure proceeds to step S106, and the control module 42 stops the cleaning robot 22. That is, the instruction unit 86 of the control module 42 instructs the travel operation control unit 51 to stop. After step S105, the procedure returns to step S102, and the above-described processing is repeated.

  If it is determined in step S103 that the control module 42 is connected to the AI server 21, the procedure proceeds to step S107, and the imaging control unit 81 controls the camera 43 to capture a moving image in the camera 43. Let it begin. The camera 43 supplies moving image data to the control module 42. In step S108, the driving control process by the control module 42 is started, and the starting process is ended.

  FIG. 8 is a flowchart illustrating details of the travel control process in step S108. In step S <b> 11, the instruction unit 86 acquires data indicating the operation state of the cleaning robot 22 from the travel operation control unit 51 and the cleaning operation control unit 61 (hereinafter referred to as operation state data). In step S12, the determination unit 93 of the instruction unit 86 refers to the operation state data to determine whether or not the cleaning robot 22 is stopped. If it is determined in step S12 that the cleaning robot 22 is stopped, the procedure proceeds to step S13, and the flag setting unit 84 sets a stop flag. If it is determined in step S12 that the cleaning robot 22 has not stopped, the procedure proceeds to step S14, and the flag setting unit 84 defeats the stop flag.

  After step S13 or step S14, the procedure proceeds to step S15, and the reception control unit 92 of the communication control unit 83 causes the communication unit 74 to receive a control signal transmitted from the AI server 21. In step S <b> 16, the determination unit 93 of the instruction unit 86 determines whether stop is instructed by the received control signal. If it is determined in step S16 that the stop has been instructed, the procedure proceeds to step S17, and the instruction unit 86 instructs the travel operation control unit 51 to stop and stops the cleaning robot 22, and the procedure proceeds to step S11. Returning, the process described above is repeated.

  If it is determined in step S16 that stop is not instructed, the procedure proceeds to step S18, and the determination unit 93 of the instruction unit 86 determines whether deceleration is instructed by the received control signal. If it is determined in step S18 that deceleration has been instructed, the procedure proceeds to step S19, and the instruction unit 86 instructs the traveling operation control unit 51 to decelerate, causing the cleaning robot 22 to decelerate, and the procedure proceeds to step S11. Returning, the process described above is repeated.

  If it is determined in step S18 that deceleration is not instructed, the procedure proceeds to step S20, and the determination unit 93 of the instruction unit 86 determines whether acceleration is instructed by the received control signal. If it is determined in step S20 that acceleration has been instructed, the procedure proceeds to step S21, and the instruction unit 86 instructs the traveling operation control unit 51 to perform acceleration and normal traveling, and accelerates the cleaning robot 22 to perform normal traveling. The procedure returns to step S11, and the above-described processing is repeated.

  If it is determined in step S20 that acceleration is not instructed, the procedure proceeds to step S22, and the determination unit 93 of the instruction unit 86 determines whether or not a stop flag is requested by the received control signal. If it is determined in step S22 that the stop flag has been requested, the procedure proceeds to step S23, the flag setting unit 84 reads the stop flag, and the transmission control unit 91 of the communication control unit 83 sends the communication unit 74 to the communication unit 74. The read stop flag is transmitted to the AI server 21. After step S23, the procedure returns to step S11, and the above-described processing is repeated.

  When it is determined in step S22 that the stop flag is not requested, the procedure proceeds to step S24, and the determination unit 93 of the instruction unit 86 determines whether or not the output of the sound is requested by the received control signal. . If it is determined in step S24 that output of audio has been requested, the procedure proceeds to step S25, where the audio output control unit 87 reads out audio data from the storage unit 75, and outputs audio to the speaker 71 using the read out audio data. Output. If it is determined after step S23 or in step S24 that no audio output is requested, the procedure returns to step S11 and the above-described processing is repeated.

  Next, with reference to the flowcharts of FIGS. 9 and 10, the travel control processing of the cleaning robot 22 by the AI server 21 will be described. In step S121, the cleaning robot 22 transmits the imaging data to the AI server 21 by streaming. That is, in step S121, the encoding unit 82 encodes moving image data obtained by imaging with the camera 43 to generate imaging data that is streaming data, and the transmission control unit 91 of the communication control unit 83 The communication unit 74 transmits imaging data that is streaming data to the AI server 21.

  In step S221, the AI server 21 receives imaging data that is streaming data. More specifically, the reception control unit 176 of the communication control unit 168 causes the communication unit 129 to receive imaging data transmitted from the cleaning robot 22. The imaging data buffer 162 temporarily stores the received imaging data. The risk measurer 161 reads the image data sequentially from the image data buffer 162.

  In step S222, risk level measurement processing is performed.

  Here, with reference to the flowchart of FIG. 11, the details of the process of measuring the risk will be described. In step S <b> 51, the object detection AI unit 171 of the risk level measurement unit 161 uses the moving image image data captured by the cleaning robot 22 to perform object learning on the object imaged in the moving image by machine learning using a neural network. Type and the position of the subject in the image. In other words, the object detection AI unit 171 determines whether a subject is a person from a moving image captured by the cleaning robot 22 that is a moving body by a machine learning using a neural network. Whether the object is other than a person is detected, and the position of the subject in the moving image is detected.

  In step S52, the image processing unit 172 of the risk measurement unit 161 performs subject processing such as imaging data, the type of subject, the position of the subject in the image, the distance from the cleaning robot 22 to the subject, and the subject's moving speed by image processing. Find the attributes of. That is, the image processing unit 172 obtains the distance from the cleaning robot 22 to the subject and the speed of the subject relative to the cleaning robot 22 from the moving image and the position of the subject in the moving image by image processing. When the image processing unit 172 detects that the subject is a person in step S <b> 51 by image processing, the image processing unit 172 obtains the orientation of the subject with respect to the cleaning robot 22 or the orientation of the face of the subject with respect to the cleaning robot 22.

  In step S53, the risk measurement AI unit 173 of the risk measurement unit 161 measures the risk from the imaging data, the type of the subject, the position of the subject in the image, and the attribute of the subject. For example, the risk level measurement AI unit 173 measures a risk level that is the degree of danger of a moving image scene from the imaging data, the type of the subject, the position of the subject in the image, and the attribute of the subject. Also, for example, the risk measurement AI unit 173 measures the risk that is the degree of danger of contact between the cleaning robot 22 and the subject from the imaging data, the type of the subject, the position of the subject in the image, and the attribute of the subject. . Here, for example, the degree of risk is expressed in a range of 0% to 100%. A risk level of 0% indicates that there is no risk of contact between the cleaning robot 22 and the subject. A risk level of 100% indicates that the possibility of contact between the cleaning robot 22 and the subject is extremely high. For example, the degree of danger indicates the degree of danger that the cleaning robot 22 falls from a step such as a staircase. Further, for example, the degree of danger indicates a degree of danger that a door near the robot 22 opens and a person behind the door comes into contact with the door. Further, for example, the degree of danger indicates a degree of danger that the robot 22 may involve an object (falling object) dropped on the floor surface by cleaning.

  In other words, the risk measurement AI unit 173 detects the moving image, the position of the subject in the moving image, the distance from the cleaning robot 22 to the subject, and the cleaning result as a result of detecting whether the subject is a person or something other than a person. From the speed of the subject with respect to the robot 22, the degree of danger of the moving image scene is measured by machine learning using a neural network. The risk measurement AI unit 173 detects the moving image, the position of the subject in the moving image, the distance from the cleaning robot 22 to the subject, and the cleaning robot 22 as a result of detecting whether the subject is a person or something other than a person. From the subject speed, the degree of danger of contact between the cleaning robot 22 and the subject and the degree of danger of the moving image scene are measured by machine learning using a neural network.

  In step S54, the drawing unit 174 of the risk level measurement unit 161 adds a frame indicating the position of the subject, the type of the subject, the attribute of the subject, the risk level as a result of the measurement, and a control signal corresponding to the risk level to the imaging data. Data for drawing the image or character to be written is written, and the process of measuring the degree of risk ends. That is, in step S54, the drawing unit 174 detects the position of the subject in the moving image, the distance from the cleaning robot 22 to the subject, the cleaning, as a result of detecting whether the subject is a person or something other than a person. Characters or images indicating the speed of the subject with respect to the robot 22 or the degree of danger of contact between the cleaning robot 22 and the subject are drawn.

  Returning to FIG. 9, in step S <b> 223, the AI server 21 causes the touch panel monitor 25 to display a moving image with the imaging data in which the result of the risk degree measurement is written.

  FIG. 12 is a diagram illustrating an example of a moving image displayed on the touch panel monitor 25. The touch panel monitor 25 indicates a moving image, a frame 201 indicating the position of the subject where it is detected whether the subject is a person or something other than a person, the type of subject, and the probability of the correctness of the type of subject. A type display 202 such as “Person: 0.89” and a distance display 203 such as “70” indicating the distance from the cleaning robot 22 to the subject in cm are displayed. Further, the touch panel monitor 25 displays a status monitor 204 that is an image indicating the state of the cleaning robot 22 on the upper right of the screen. The status monitor 204 indicates the name of the cleaning robot 22 such as “ROBOT1”, the current position of the cleaning robot 22 such as “East Bldg 1F”, and the degree of danger of the moving image scene such as “Degree of Risk 20%”. Is displayed. The status monitor 204 also includes a lamp display 211 indicating that the cleaning robot 22 is in a normal traveling state, a lamp display 212 indicating that the cleaning robot 22 is decelerating, and the cleaning robot 22 being stopped. Is displayed. The lamp display 211, the lamp display 212, and the lamp display 213 are images that simulate traffic signals, and are images that are displayed so that any one of them becomes active (the luminance increases). For example, the lamp display 211, the lamp display 212, and the lamp display 213 are displayed in green, yellow, and red, respectively.

  For example, as shown in FIG. 12, when the lamp display 211 is displayed actively and the lamp display 212 and the lamp display 213 are displayed inactive (the brightness is reduced), the lamp display 211, the lamp display 212, and The lamp display 213 indicates that the cleaning robot 22 is in a normal traveling state.

  On the touch panel monitor 25, “Stop” is displayed at the lower left of the screen, a button 205 for instructing the stop and “Attention” are displayed, and an audio output such as “Please note. The robot passes.” A button 206 for instructing is displayed. Further, the touch panel monitor 25 displays “stop: 0%: 70 cm” or the like on the distance display 207 at the lower right of the screen.

  Returning to FIG. 9, in step S224, the determination unit 166 determines whether the degree of risk measured in step S53 is 40% or more and less than 60%. For example, in step S224, the determination unit 166 determines whether or not the risk level of the moving image scene is 40% or more and less than 60%. If it is determined in step S224 that the degree of risk is 40% or more and less than 60%, the procedure proceeds to step S225, and the command unit 167 transmits a control signal instructing deceleration to the cleaning robot 22. In this case, the transmission control unit 175 of the communication control unit 168 causes the communication unit 129 to transmit a control signal instructing deceleration to the cleaning robot 22. Further, in this case, the command unit 167 causes the output unit 127, which is a speaker, to output a sound such as a predetermined beep sound to inform the supervisor that the vehicle has decelerated.

  If it is determined in step S224 that the degree of risk is not 40% or more and less than 60%, the procedure proceeds to step S226, and the determination unit 166 determines whether or not the degree of risk measured in step S53 is 60% or more. judge. For example, in step S226, the determination unit 166 determines whether or not the risk level of the moving image scene is 60% or more. If it is determined in step S226 that the degree of risk is 60% or more, the procedure proceeds to step S227, and the command unit 167 transmits a control signal instructing the stop to the cleaning robot 22. In this case, the transmission control unit 175 of the communication control unit 168 causes the communication unit 129 to transmit a control signal for instructing the stop to the cleaning robot 22.

  In this case, for example, as shown in FIG. 13, the AI server 21 causes the touch panel monitor 25 to display the button 205 and the lamp display 213 in an active manner and to display the lamp display 211 and the lamp display 212 in an inactive state. The cleaning robot 22 is stopped. Also, the AI server 21 causes the touch panel monitor 25 to display the periphery of the moving image in the same red color as that of the button 205 and the lamp display 213 that are actively displayed.

  Further, the AI server 21 causes the touch panel monitor 25 to display a message 231 such as “AUTO: STOP” and “Please TOUCH the monitor to release the robot if AI Server is mistaking”.

  In step S228, the determination unit 166 determines whether or not the subject is a person based on the result of detection of whether the subject is a person or something other than a person in step S51. If it is determined in step S228 that the subject is a person, the procedure proceeds to step S229, and the command unit 167 transmits a control signal instructing output of sound to the cleaning robot 22. In this case, the transmission control unit 175 of the communication control unit 168 causes the communication unit 129 to transmit a control signal for instructing output of audio to the cleaning robot 22. When the cleaning robot 22 receives a control signal for instructing voice output, the voice output control unit 87 of the cleaning robot 22 reads the voice data from the storage unit 75, and “cautions the speaker 71 with the read voice data. "The robot passes."

  When the vehicle decelerates, a sound such as “Please be careful. The robot passes” may be output. In this case, for example, as illustrated in FIG. 14, the AI server 21 causes the touch panel monitor 25 to actively display the button 206 and the lamp display 212 and to display the lamp display 211 and the lamp display 213 inactive. , It displays that the cleaning robot 22 is decelerating. In addition, the AI server 21 causes the touch panel monitor 25 to display the periphery of the moving image in the same yellow as the yellow of the button 206 and the lamp display 212 that are actively displayed.

  Furthermore, the AI server 21 causes the touch panel monitor 25 to display a message 231 such as “AUTO: Attention please. The robot will pass.” And “Please TOUCH the monitor to release the robot if AI Server is mistaking”.

  In addition, when the supervisor touches the button 206 displayed on the touch panel monitor 25, the cleaning robot 22 can be decelerated and a sound such as “Please be careful. The robot passes” can be output. . In this case, for example, as shown in FIG. 15, the AI server 21 displays the button 206 and the lamp display 212 on the touch panel monitor 25 in an active manner, and displays the lamp display 211 and the lamp display 213 in an inactive state. , It displays that the cleaning robot 22 is decelerating. In addition, the AI server 21 causes the touch panel monitor 25 to display the periphery of the moving image in the same yellow as the yellow of the button 206 and the lamp display 212 that are actively displayed.

  Also, the AI server 21 displays a message 231 such as ““ MANUAL: Attention please. The robot will pass ”on the touch panel monitor 25.

  In step S230, the determination unit 166 determines whether or not the sound output has been repeated three times, and when it is determined that the sound output has been repeated three times, the control is returned to the cleaning robot 22 and the cleaning robot 22 is returned. In order to avoid a person, the procedure proceeds to step S231, and the command unit 167 transmits a control signal instructing acceleration to the cleaning robot 22. In this case, the transmission control unit 175 of the communication control unit 168 causes the communication unit 129 to transmit a control signal instructing acceleration to the cleaning robot 22.

  After step S231, if it is determined in step S230 that the sound output has not been repeated three times, or after step S225, the process of running control of the cleaning robot 22 by the AI server 21 is performed in step S121 and step S231. Returning to S221, the above-described procedure is repeated.

  In the cleaning system 11, the monitor looks at the image displayed on the touch panel monitor 25 in the procedure of step S 223, monitors the running of the cleaning robot 22, and feels danger in the current situation as a moving image scene. In case of touching the screen of the touch panel monitor 25, that is, the touch panel, the stop or deceleration of the cleaning robot 22 is instructed.

  With reference to the flowchart of FIG. 16, the process according to the instruction | indication from a supervisor is demonstrated. In step S81, the input control unit 169 obtains a signal transmitted from the touch panel of the touch panel monitor 25, thereby touching the button 205, touching the button 206, touching any position on the screen, or the like. The instruction by the operation of the monitor on the touch panel of the touch panel monitor 25 is acquired.

  In step S <b> 82, the determination unit 166 determines whether or not the inactive state button 205 has been touched or any position on the screen has been touched to stop the monitor. If it is determined in step S82 that a stop has been instructed by the supervisor, the procedure proceeds to step S83, and the command unit 167 transmits a control signal instructing the stop to the cleaning robot 22. In this case, the transmission control unit 175 of the communication control unit 168 causes the communication unit 129 to transmit a control signal for instructing the stop to the cleaning robot 22.

  In this case, for example, as shown in FIG. 17, the AI server 21 causes the touch panel monitor 25 to display the button 205 and the lamp display 213 in an active manner and to display the lamp display 211 and the lamp display 212 in an inactive state. The cleaning robot 22 is stopped. Also, the AI server 21 causes the touch panel monitor 25 to display the periphery of the moving image in the same red color as that of the button 205 and the lamp display 213 that are actively displayed.

  Further, the AI server 21 displays a message 231 such as “MANUAL: STOP” on the touch panel monitor 25.

  In addition, when a plurality of people or a plurality of objects are captured in a moving image scene, the type of the subject and the position of the subject in the image are detected for each subject, and cleaning is performed for each subject. The attributes of the subject such as the distance from the robot 22 to the subject and the moving speed of the subject are obtained, and the degree of danger that is the degree of danger of the moving image scene is measured.

  FIG. 18 is a diagram illustrating an example of a moving image displayed on the touch panel monitor 25 in the case where a plurality of people and a plurality of objects are captured in the moving image. In addition, the same code | symbol is attached | subjected to the part similar to FIG. 17, and the description is abbreviate | omitted.

  In the example shown in FIG. 18, the touch panel monitor 25 includes a moving image and frames 201-1 to 201-5 indicating the positions of the subjects where the subject is detected to be a person or something other than a person, Type display 202-1 to 202-5 indicating the probability of the correctness of the type and the type of subject, and distance displays 203-1 to 203-5 indicating the distance from the cleaning robot 22 to the subject in cm. For example, in the touch panel monitor 25, a frame 201-1 indicating the position of a subject detected as a person, the type of the subject at the position indicated by the frame 201-1 is a person, and the type of the subject is a person. "Person: 0.89" indicating the probability of correctness, type display 202-1, "260" indicating the distance from the cleaning robot 22 to the subject at the position indicated by the frame 201-1 in cm The distance display 203-1 is displayed. Further, for example, on the touch panel monitor 25, a frame 201-2 indicating the position of a subject detected as a person, the type of the subject at the position indicated by the frame 201-2 is a person, and the type of the subject is a person. “Person: 0.93” indicating the probability of being correct, “270” indicating the distance from the cleaning robot 22 to the subject at the position indicated by the frame 201-2 in cm. ”Or the like is displayed.

  Further, for example, in the touch panel monitor 25, a frame 201-3 indicating the position of the subject that is detected to be a trash can, and the type of the subject at the position indicated by the frame 201-3 is the trash can. The type display 202-3 “Bin: 0.82” indicating the probability of the correctness of being a waste bin, and the distance from the cleaning robot 22 to the subject that is the waste bin at the position indicated by the frame 201-3 in cm A distance display 203-3 such as “180” shown in FIG. Furthermore, for example, in the touch panel monitor 25, the frame 201-4 indicating the position of the subject detected as the door, the type of the subject at the position indicated by the frame 201-4 is the door, and the type of the subject is A type display 202-4 “Door: 0.79” indicating the probability of the correctness of being a door, and a distance from the cleaning robot 22 to a subject at the position indicated by the frame 201-4 in cm. A distance display 203-4 such as “320” is displayed.

  Further, for example, in the touch panel monitor 25, the frame 201-5 indicating the position of the subject detected to be illumination, the subject type at the position indicated by the frame 201-5 is illumination, and the subject type is illumination. "Light: 0.69" indicating the probability of correctness, and "300: 300" indicating the distance from the cleaning robot 22 to the subject that is the illumination at the position indicated by the frame 201-5. ”Or the like is displayed.

  The status monitor 204 indicates the name of the cleaning robot 22 such as “ROBOT1”, the current position of the cleaning robot 22 such as “East Bldg 1F”, and the degree of danger of the moving image scene such as “Degree of Risk 87%”. Is displayed.

  As shown in FIG. 18, when a moving image includes a plurality of people or a plurality of objects, a subject frame 201 having a large influence on the degree of danger of the moving image scene is indicated by a thick line or a double line. Or highlighted, such as red. That is, the drawing unit 174 highlights the subject frame 201 that greatly affects the degree of danger of the moving image scene, and draws it on the moving image.

  For example, when a subject that is a plurality of people or a plurality of objects is captured in the moving image, a frame 201 indicating the position of the subject having the highest degree of danger among the subjects of the moving image scene is highlighted. That is, for example, when a moving image includes a plurality of people or a plurality of objects, the position of a subject (person walking while looking at a mobile phone) having the highest risk of contact with the cleaning robot 22 Is highlighted.

  In addition, for example, when a moving image includes a plurality of people or a plurality of objects, a frame 201 indicating the positions of a predetermined number of subjects in descending order of the degree of danger among the subjects of the moving image scene. Highlighted. That is, for example, when a moving image includes a plurality of people or a plurality of objects, the position of a subject (person walking while looking at a mobile phone) having the highest risk of contact with the cleaning robot 22 A frame 201-2 indicating the position of the subject (the nearby garbage can) having the second highest risk of contact with the cleaning robot 22 is highlighted.

  That is, when a moving image includes a plurality of people or a plurality of objects, the frame 201 of the subject in a dangerous state in the moving image scene is a thick line or a double line, or It is highlighted such as red.

  By doing in this way, the subject who is monitoring the moving images displayed on the touch panel monitors 25-1 and 25-2, which subject is dangerous among the plurality of subjects reflected in the moving image. Can be recognized immediately.

  Returning to FIG. 16, in step S <b> 84, the learning data recording control unit 164 causes the recording unit 128 to record moving image imaging data for 5 seconds before and 5 seconds before the time when the stop is instructed. That is, when a stop is instructed by the supervisor, the learning data recording control unit 164 causes the recording unit 128 to record a moving image of a predetermined length including the time when the stop is instructed. At this time, the learning data recording control unit 164 records data indicating that the stop has been instructed in association with moving image data of the previous 5 seconds and the subsequent 5 seconds with reference to the time when the stop is instructed. Recorded in the unit 128. In step S85, the learning scheduler 165 reserves additional learning. That is, the learning scheduler 165 reserves execution of learning by the risk degree measurement unit 161 using moving image captured data recorded in the recording unit 128.

  If it is determined in step S82 that the monitoring has not been instructed to stop, the procedure proceeds to step S86, and the determination unit 166 has touched the inactive button 206 to determine whether the monitoring has instructed deceleration. Determine whether or not. If it is determined in step S86 that deceleration has been instructed by the supervisor, the procedure proceeds to step S87, and the command unit 167 transmits a control signal instructing deceleration to the cleaning robot 22. In this case, the transmission control unit 175 of the communication control unit 168 causes the communication unit 129 to transmit a control signal instructing deceleration to the cleaning robot 22.

  In step S <b> 88, the learning data recording control unit 164 causes the recording unit 128 to record moving image imaging data for 5 seconds before and 5 seconds before the point in time when deceleration is instructed. That is, the learning data recording control unit 164 causes the recording unit 128 to record a moving image of a predetermined length including the time when the deceleration is instructed when the monitoring is instructed to decelerate. At this time, the learning data recording control unit 164 records data indicating that deceleration has been instructed in association with moving image imaging data for the previous 5 seconds and the subsequent 5 seconds with reference to the point in time when the deceleration is instructed. Recorded in the unit 128. In step S89, the learning scheduler 165 reserves additional learning. That is, the learning scheduler 165 reserves execution of learning by the risk degree measurement unit 161 using moving image captured data recorded in the recording unit 128.

  If it is determined in step S86 that deceleration is not instructed by the supervisor, the procedure proceeds to step S90, where the active button 205 or the active button 206 is touched and acceleration is instructed by the supervisor. Determine whether or not. If it is determined in step S90 that acceleration has been instructed by the supervisor, the procedure proceeds to step S91, and the command unit 167 transmits a control signal instructing acceleration to the cleaning robot 22. In this case, the transmission control unit 175 of the communication control unit 168 causes the communication unit 129 to transmit a control signal instructing acceleration to the cleaning robot 22.

  For example, as shown in FIG. 13, the AI server 21 causes the touch panel monitor 25 to display the button 205 and the lamp display 213 in an active manner, and to display the lamp display 211 and the lamp display 212 in an inactive state, thereby cleaning the robot. When the message 221, such as “AUTO: STOP” and “Please TOUCH the monitor to release the robot if AI Server is mistaking”, is displayed, the monitor touches the touch panel monitor 25. When the risk measuring unit 161 determines that the person image shown in the poster is a person by looking at the moving image displayed on the poster, the monitor 205 displays the button 205 displayed actively. Is touched to instruct acceleration of the cleaning robot 22.

  After step S85, step S89 or step S91, the procedure returns to step S81, and the process according to the instruction from the supervisor is repeated.

  FIG. 19 is a flowchart for explaining the termination process. In step S181, the determination unit 93 of the instruction unit 86 acquires status information indicating the state of the cleaning operation from the cleaning operation control unit 61, and determines whether the cleaning operation of the cleaning robot 22 has been completed. If it is determined in step S181 that the cleaning operation has not been completed, the determination process is repeated.

  If it is determined in step S181 that the cleaning work has been completed, the procedure proceeds to step S182, and the instruction unit 86 notifies the AI server 21 of the completion of the work. That is, the transmission control unit 91 of the communication control unit 83 causes the communication unit 74 to transmit a work end notification to the AI server 21 in accordance with an instruction from the instruction unit 86.

  In step S <b> 281, the reception control unit 176 of the communication control unit 168 causes the communication unit 129 to receive a work end notification from the cleaning robot 22. In step S282, the reception control unit 176 of the communication control unit 168 causes the communication unit 129 to stop receiving imaging data that is streaming data.

  In step S183, the transmission control unit 91 of the communication control unit 83 stops transmission of imaging data that is streaming data. In step S184, the imaging control unit 81 causes the camera 43 to stop imaging. In this case, the encoding unit 82 stops encoding the moving image data.

  In step S185, the instruction unit 86 notifies the AI server 21 of power off. That is, the transmission control unit 91 of the communication control unit 83 causes the communication unit 74 to transmit a power-off notification to the AI server 21 in accordance with an instruction from the instruction unit 86.

  In step S283, the reception control unit 176 of the communication control unit 168 causes the communication unit 129 to receive a power-off notification from the cleaning robot 22. In step S284, the determination unit 166 determines whether or not the cleaning robot 22 is powered off based on whether or not the power-off notification is received from the cleaning robot 22. If it is determined in step S284 that the cleaning robot 22 is not powered off, the procedure returns to step S283, and the above-described processing is repeated.

  If it is determined in step S284 that the cleaning robot 22 is powered off, the procedure proceeds to step S285, and the risk measuring unit 161 ends the monitoring process for the cleaning robot 22 that has notified the end.

  In step S186, the control module 42 is stopped by the operation of the supervisor or operator. In step S187, the power of the cleaning robot 22 is turned off by the operation of the supervisor or the operator, and the end process ends.

  Next, the additional learning scheduling process reserved in step S85 or step S89 will be described with reference to the flowchart of FIG. In step S301, the learning scheduler 165 uses buttons or menus for selecting whether or not to perform additional learning on the output unit 127, which is a display, or the touch panel monitor 25, and a table or image in which date or time is arranged. Display the schedule for additional learning. In step S302, the learning scheduler 165 acquires an instruction as to whether or not to perform additional learning from an operation on the touch panel monitor 25 by a supervisor or an operation on the input unit 126 which is a keyboard or a mouse by an operator. For example, the learning scheduler 165 performs additional learning by causing a supervisor or operator to select a button or menu arranged in a table or image displayed on the output unit 127 or the touch panel monitor 25 that is a display. Get an indication of whether or not.

  In step S303, the learning scheduler 165 records, in the recording unit 128, data indicating whether or not to perform additional learning instructed by the supervisor or operator as additional learning schedule data.

  In step S304, the learning scheduler 165 acquires a date and time instruction for performing additional learning from an operation on the touch panel monitor 25 by the supervisor or an operation on the input unit 126 which is a keyboard or a mouse by the operator. For example, the learning scheduler 165 displays the date and time on the monitor or operator arranged in the table or image displayed on the output unit 127 or touch panel monitor 25 that is a display, on the output unit 127 or touch panel monitor 25 that is a display. By selecting the time, an instruction of the date and time for performing additional learning is acquired.

  In step S305, the learning scheduler 165 adds the data indicating the date and time for executing the additional learning instructed by the supervisor or operator to the additional learning schedule data, records the data in the recording unit 128, and schedules additional learning. This process ends.

  Next, with reference to the flowchart of FIG. 30, the additional learning process executed at regular intervals will be described. In step S321, the learning scheduler 165 reads from the additional learning schedule data recorded in the recording unit 128 an instruction as to whether or not to perform additional learning and the date and time for executing additional learning. In step S322, the learning scheduler 165 determines whether or not execution of additional learning is instructed from an instruction on whether or not to execute additional learning read from the additional learning schedule data. If it is determined in step S322 that execution of additional learning has been instructed, the procedure proceeds to step S323, and the learning scheduler 165 performs additional learning processing from a timing unit (not shown) provided in the AI server 21. Get today's date and current time when is running.

  In step S324, the learning scheduler 165 compares the date and time when the additional learning read from the additional learning schedule data is executed with the today's date and current time when the additional learning process is executed. Thus, it is determined whether or not the date and time for performing the additional learning have come. If it is determined in step S324 that the date and time for performing additional learning has come, the procedure proceeds to step S325, and the learning scheduler 165 instructs the risk measuring unit 161 to detect the object detection AI unit 171 and the risk. The degree measurement AI unit 173 is set to the learning mode. For example, in the learning mode, the learning scheduler 165 sets a batch size and an epoch (learning number) in the object detection AI unit 171 and the risk measurement AI unit 173. Further, for example, in the learning mode, the learning scheduler 165 selects an optimization function and a loss function used for learning of the object detection AI unit 171 and the risk measurement AI unit 173.

  In step S326, the object detection AI unit 171 and the risk measurement AI unit 173 instruct to stop or decelerate by the supervisor recorded in the recording unit 128 in the procedure of step S84 or step S88 according to the instruction from the learning scheduler 165. The learning is performed based on the moving image at that time and the data indicating that the stop or the deceleration associated with the moving image is instructed, and the additional learning process ends.

  For example, the object detection AI unit 171 and the risk measurement AI unit 173 learn by a batch method. In addition, for example, the object detection AI unit 171 and the risk measurement AI unit 173 may be a stop or a link associated with a moving image and a moving image when stoppage or deceleration is instructed by a monitor recorded in the recording unit 128. Supervised learning is performed by using a gradient calculation method such as an error back-propagation method using data indicating that deceleration has been instructed.

  If it is determined in step S322 that execution of additional learning is not instructed, or if it is determined in step S324 that the date and time for performing additional learning have not been reached, it is not necessary to perform learning. The additional learning process ends without learning.

  In this way, learning can be performed more easily while avoiding collision more reliably. Further, it is possible to know the progress of the process for obtaining the degree of danger of contact so that more effective correction or learning can be performed.

  As described above, the AI server 21 autonomously travels on the floor in the building and is captured in a moving image by machine learning from the moving image transmitted from the cleaning robot 22 that captures the moving image of the traveling destination. A degree-of-contact measuring unit 161 that measures the degree of danger of contact with a subject, a display control unit 170 that controls display of moving images on the touch panel monitor 25, and monitoring that monitors the displayed moving images An input control unit 169 that obtains a stop instruction by an operation by the operator, and when the measured danger level is equal to or greater than a predetermined threshold or when a stop is instructed by the supervisor, the cleaning robot 22 is instructed to stop. An instruction unit 167 that performs recording, a learning data recording control unit 164 that causes the recording unit 128 to record a moving image for a predetermined length of time including when the stop is instructed, when the stop is instructed by the supervisor, and recording Book execute learning risk measuring unit 161 by the recorded moving image 128 and a learning scheduler 165.

  When the input control unit 169 obtains a deceleration instruction by an operation from the supervisor, and when the instruction unit 167 is instructed to decelerate, the instruction is given to the cleaning robot 22, and the learning data recording control unit 164 performs monitoring. When deceleration is instructed by a person, a moving image for a predetermined length of time including when the deceleration is instructed can be recorded on the recording means. In this way, learning can be performed more easily.

  The learning scheduler 165 can make a reservation by scheduling a learning job of the risk measuring unit 161. In this way, learning can be performed more easily.

  The learning scheduler 165 determines whether or not learning can be performed and the time when the learning is executed in the scheduling of the learning job of the risk measuring unit 161 in accordance with an instruction from the supervisor or the operator. Can do. In this way, learning can be performed more easily.

  A drawing unit 174 is further provided for writing a character or an image indicating the degree of danger of contact between the cleaning robot 22 and the subject in the moving image, and the display control unit 170 indicates the degree of danger of contact between the cleaning robot 22 and the subject. The display on the touch panel monitor 25 of a moving image in which characters or images are written can be controlled. By doing so, collision can be avoided more reliably.

  The risk measuring unit 161 measures the distance from the cleaning robot 22 to the subject and the speed of the subject with respect to the cleaning robot 22, and the drawing unit 174 displays the distance from the cleaning robot 22 to the subject and the subject with respect to the cleaning robot 22 on the moving image. The display control unit 170 displays on the touch panel monitor 25 a moving image in which characters or images indicating the distance from the cleaning robot 22 to the subject and the speed of the subject with respect to the cleaning robot 22 are written. Can be controlled. By doing so, collision can be avoided more reliably.

  When the risk measurement unit 161 detects whether the subject is a person or something other than a person, and the command unit 167 detects that the subject is a person, a predetermined sound is output to the cleaning robot 22. Can be commanded. By doing so, collision can be avoided more reliably.

  In this way, the robot touches the subject in the moving image by machine learning from the moving image transmitted from the cleaning robot 22 that autonomously travels on the floor in the building and captures the moving image of the traveling destination. The degree of danger is measured, a moving image is displayed on the touch panel monitor 25, a stop instruction is obtained by an operation from a supervisor who is monitoring the displayed moving image, and the measured degree of danger is When it is equal to or greater than a predetermined threshold value or when a stop is instructed by the supervisor, the cleaning robot 22 is instructed to stop. Therefore, the degree of risk of contact with the subject in the moving image is temporarily measured. Even if it is wrong, the monitor who monitors the displayed moving image instructs the stop, so that the collision can be avoided more reliably. In addition, when a stop is instructed by the supervisor, a moving image of a predetermined length including the time when the stop is instructed is recorded in the recording unit 128, and the risk measurement by the moving image recorded in the recording unit 128 is performed. Since execution of learning by the unit 161 is reserved, even if the measurement of the degree of danger of contact with the subject in the moving image is wrong, it is not necessary to prepare learning data for each situation. Since learning can be performed easily, collision can be avoided more reliably after learning. In this way, learning can be performed more easily while avoiding collision more reliably.

  In this way, learning can be performed more easily while avoiding collision more reliably.

  In addition, the AI server 21 detects whether a subject is a person or a person other than a person from a moving image captured by the cleaning robot 22 by using machine learning. An object detection AI unit 171 for detecting the position of the subject in the image, an image processing unit 172 for determining a distance from the cleaning robot 22 to the subject and a speed of the subject with respect to the cleaning robot 22 based on the moving image and the position of the subject in the moving image; As a result of detecting whether the moving image, the subject is a person or something other than a person, the position of the subject in the moving image, the distance from the cleaning robot 22 to the subject, and the speed of the subject with respect to the cleaning robot 22 are determined by machine learning. A risk measurement AI unit 173 that measures the degree of danger of a moving image scene, and whether the subject is a person in the moving image As a result of detecting whether the object is other than a person, characters or images indicating the position of the subject in the moving image, the distance from the moving body to the subject, the speed of the subject relative to the moving body, and the degree of danger of the moving image scene are displayed. And a drawing unit 174 for writing.

  The object detection AI unit 171 detects whether the subject is a person or something other than a person by machine learning using a neural network, detects the position of the subject in the moving image, and detects the risk measurement AI unit 173. However, the degree of danger of contact between the cleaning robot 22 and the subject can be measured by machine learning using a neural network.

  When the image processing unit 172 detects that the subject is a person, the orientation of the subject with respect to the cleaning robot 22 or the orientation of the face of the subject with respect to the cleaning robot 22 can be obtained.

  If the degree of danger of contact between the cleaning robot 22 and the subject is equal to or greater than a predetermined threshold, a command unit 167 that instructs the cleaning robot 22 to stop or decelerate can be further provided.

  A communication unit 129 for receiving a moving image transmitted by streaming from the cleaning robot 22 is further provided, and the object detection AI unit 171, the image processing unit 172, and the risk measurement AI unit 173 are each detected by the received moving image. Or image processing or measurement.

  When a stop or deceleration instruction to the cleaning robot 22 is input from the outside, a recording unit 128 that records a moving image for a predetermined length of time including when the stop or deceleration is instructed can be further provided.

  As described above, the object detection AI unit 171 detects whether a subject is a person or a person other than a person from a moving image picked up by a moving object by machine learning. Then, the position of the subject in the moving image is detected, and the image processing unit 172 obtains the distance from the moving body to the subject and the speed of the subject with respect to the moving body from the position of the subject in the moving image and the moving image, and the risk measurement AI As a result of detecting whether the moving image, the subject is a person, or a non-human thing, the unit 173 determines from the position of the subject in the moving image, the distance from the moving body to the subject, and the speed of the subject relative to the moving body. The degree of danger of the scene of the moving image is measured by learning, and the drawing unit 174 detects that the subject is a person or something other than a person as a moving image. The characters or images that indicate the position of the subject to be moved, the distance from the moving object to the subject, the speed of the subject relative to the moving object, and the degree of danger of the moving image scene are written. If it is displayed, when the user views the moving image, the state of detection of whether the subject is a person or something other than a person, the state of detection of the position of the subject in the moving image, from the moving body to the subject Distance, the speed of the subject with respect to the moving object, and the degree of danger of the entire current situation as a moving image scene, so why does the user determine the degree of danger of the moving image scene? It is possible to know the clue of whether or not the judgment is correct. Thereby, for example, it is possible to perform a more effective operation such as instructing the moving body to stop or decelerate. In this way, the user can know a clue as to why such a determination has been made in the process of obtaining the degree of danger of the scene of the moving image, so that a more effective operation can be performed.

  In addition, when scheduling additional learning, a moving image used for learning by a supervisor or operator may be trimmed to a necessary area so that the object detection AI unit 171 and the risk measurement AI unit 173 learn.

  In addition, although it demonstrated that the AI server 21 controlled driving | running | working of the cleaning robot 22, not only this but the conveyance robot which conveys a document, a chemical | medical agent, foodstuffs, an apparatus, etc. can also be controlled. Furthermore, the AI server 21 can also control a moving body such as a vehicle such as an automobile or a train that moves indoors or outdoors, or an unmanned aircraft such as a multicopter.

  In addition, it has been described that the cleaning robot 22 stops or decelerates when the AI server 21 transmits a control signal that instructs the control module 42 to stop or decelerate, and the control module 42 instructs the travel operation control unit 51. Not limited to this, a pressing mechanism that presses a stop button or a deceleration button of the hardware of the cleaning robot 22 by a mechanical mechanism, provided with a pressing mechanism that operates according to a signal on the network, and the AI server 21 outputs a signal. By transmitting, the stop button or the deceleration button of the hardware of the cleaning robot 22 may be pressed to control the traveling of the cleaning robot 22.

  Although the AI server 21 has been described as instructing the cleaning robot 22 to stop or decelerate travel, the present invention is not limited to this, and the cleaning operation of the cleaning robot 22 is stopped, or a specific cleaning function, for example, the brush 65 is stopped. You may make it let it.

  In step S224, the determination unit 166 determines whether or not the degree of risk measured in step S53 is 40% or more and less than 60%. In step S226, the determination unit 166 is measured in step S53. Although it has been described that it is determined whether or not the degree of risk is 60% or more, the present invention is not limited to this. In step S224, the determination unit 166 determines whether or not the degree of risk measured in step S53 is 50% or more and less than 80%. In step S226, the determination unit 166 can set the determination threshold to a desired value, such as determining whether the degree of risk measured in step S53 is 80% or more.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  11 Cleaning System, 21 AI Server, 22, 22-1 and 22-2 Cleaning Robot, 23 Hub, 24, 24-1 to 24-3 Access Point, 25, 25-1 and 25-2 Touch Panel Monitor, 41 Main Body, 42 control module, 43 camera, 51 travel operation control unit, 52 sensor, 53 drive unit, 54 motor, 55 wheel, 61 cleaning operation control unit, 62 sensor, 63 drive unit, 64 motor, 65 brush, 71 speaker, 72 display Unit, 73 touch panel, 74 communication unit, 75 storage unit, 81 imaging control unit, 82 encoding unit, 83 communication control unit, 84 flag setting unit, 85 storage control unit, 86 instruction unit, 87 audio output control unit, 91 transmission Control unit, 92 reception control unit, 93 determination unit 121 CPU, 122 ROM, 123 RAM, 126 input unit, 127 output unit, 128 recording unit, 129 communication unit, 131 removable media, 161 risk measurement unit, 162 imaging data buffer, 163 decoding unit, 164 learning data recording control unit 165 Learning scheduler, 166 determination unit, 167 command unit, 168 communication control unit, 169 input control unit, 170 display control unit, 171 object detection AI unit, 172 image processing unit, 173 risk measurement AI unit, 174 drawing unit, 175 transmission control unit, 176 reception control unit, 201, 201-1 to 201-5 frame, 202, 202-1 to 202-5 type display, 203, 203-1 to 203-5 distance display, 204 status monitor, 205 And 206 , 207 distance such as a display, 211, 212 and 213 lamp display, 231 messages

Claims (8)

In an information processing system including a robot that autonomously travels on a floor in a building and picks up a moving image of the traveling destination, and an information processing device that controls the robot,
Measuring means for measuring the degree of danger of contact between the subject in the moving image and the robot by machine learning from the moving image;
Display control means for controlling display on the display means of the moving image;
An acquisition means for acquiring an instruction to stop the robot by an operation from a supervisor who is monitoring the displayed moving image;
When the measured degree of danger is equal to or greater than a predetermined threshold value or when a stop is instructed by the supervisor, command means for instructing the robot to stop,
An information processing system comprising: a recording control unit that causes the recording unit to record the moving image for a predetermined length of time including when the stop is instructed by the monitor. The information processing system according to claim 1,
The measuring means measures the degree of danger of contact with the subject in the moving image for each of the plurality of robots by using machine learning from the moving images captured by each of the plurality of robots,
The acquisition means acquires an instruction to stop any of the plurality of robots by an operation from a monitor who monitors the displayed moving image,
The command means has a degree of danger when the degree of danger measured for any of the plurality of robots is equal to or greater than a predetermined threshold or when the supervisor instructs the stop of any of the plurality of robots. The robot is instructed to stop or is instructed to stop,
The recording control means is the moving image transmitted from the robot instructed to stop, when the stop is instructed by any of the plurality of robots, and when the stop is instructed. An information processing system for causing the recording means to record the moving image for a predetermined length of time including: The information processing system according to claim 1,
The information processing system in which the measurement unit learns using data indicating that a stop is instructed and the moving image recorded in the recording unit. From the moving image transmitted from the robot that autonomously travels on the floor in the building and captures the moving image of the traveling destination, contact between the subject and the robot reflected in the moving image is performed by machine learning. A measuring means for measuring the degree of danger;
Display control means for controlling display on the display means of the moving image;
An acquisition means for acquiring an instruction to stop the robot by an operation from a supervisor who is monitoring the displayed moving image;
When the measured degree of danger is equal to or greater than a predetermined threshold value or when a stop is instructed by the supervisor, command means for instructing the robot to stop,
An information processing apparatus comprising: a recording control unit that causes the recording unit to record the moving image for a predetermined length of time including when the stop is instructed by the supervisor. The information processing apparatus according to claim 4,
The measuring means measures, from the moving images transmitted from each of the plurality of robots, the degree of danger of contact with the subject in the moving image for each of the plurality of robots by machine learning,
The acquisition means acquires an instruction to stop any of the plurality of robots by an operation from a monitor who monitors the displayed moving image,
The command means has a degree of danger when the degree of danger measured for any of the plurality of robots is equal to or greater than a predetermined threshold or when the supervisor instructs the stop of any of the plurality of robots. The robot is instructed to stop or is instructed to stop,
The recording control means is the moving image transmitted from the robot instructed to stop, when the stop is instructed by any of the plurality of robots, and when the stop is instructed. An information processing apparatus for causing the recording means to record the moving image for a predetermined length of time including The information processing apparatus according to claim 4,
The information processing apparatus, wherein the measurement unit learns using data indicating that a stop is instructed and the moving image recorded in the recording unit. From the moving image transmitted from the robot that autonomously travels on the floor in the building and captures the moving image of the traveling destination, contact between the subject and the robot reflected in the moving image is performed by machine learning. A measuring step to measure the degree of danger;
A display control step for controlling display on the display means of the moving image;
An acquisition step of acquiring an instruction to stop the robot by an operation from a supervisor who is monitoring the displayed moving image;
A command step for instructing the robot to stop when the measured degree of danger is equal to or greater than a predetermined threshold or when a stop is instructed by the supervisor;
A recording control step of causing the recording means to record the moving image for a predetermined length of time including when the stop is instructed when the monitor instructs the stop of the robot. From the moving image transmitted from the robot that autonomously travels on the floor in the building and captures the moving image of the traveling destination, contact between the subject and the robot reflected in the moving image is performed by machine learning. A measuring step to measure the degree of danger;
A display control step for controlling display on the display means of the moving image;
An acquisition step of acquiring an instruction to stop the robot by an operation from a supervisor who is monitoring the displayed moving image;
A command step for instructing the robot to stop when the measured degree of danger is equal to or greater than a predetermined threshold or when a stop is instructed by the supervisor;
When the stop is instructed by the supervisor, the computer performs information processing including a recording control step for causing the recording means to record the moving image for a predetermined length of time including when the stop is instructed. Program.

Images