JP2017047519A - Cloud robotics system, information processor, program, and method for controlling or supporting robot in cloud robotics system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an improved cloud robotics system.SOLUTION: A cloud robotics system comprising a cloud platform and a robot, in which the cloud platform and the robot respectively execute a service module having a common interface for executing a task. The common interface has a request for requesting execution of at least the task and a feedback for reporting about a state of execution of the task, and each service module has a working routine for executing the task, a failure detection sub-routine for detecting whether execution of the task is failed or not, and a rescue request routine for requesting the exterior for rescue.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system for controlling or supporting a robot that autonomously operates or moves autonomously, and more specifically, a robot that autonomously operates or moves autonomously, and a cloud server group that controls or supports these robots (hereinafter referred to as “cloud”). And a cloud robotics system including an information processing apparatus that can be connected to the cloud platform.

2. Description of the Related Art Conventionally, techniques for providing services by robots that autonomously work or move (hereinafter referred to as “autonomous robots” or simply “robots”) in ordinary homes and various facilities have been proposed. It was. These robots work or move to fulfill their duties by receiving a request directly from a user or receiving a task from a remote server to which the robot is connected via a network.
Various proposals have been made for a communication method between the remote server and the robot and a method of controlling the robot by the remote server.

  For example, a technique has been proposed in which when a robot executes a task based on a task execution request from a server, communication between the server and the robot is performed satisfactorily and task execution processing is performed reliably and smoothly (patent) Reference 1).

  That is, in Patent Document 1, a communication method in which a robot having a communication function and a server that makes a task execution request to the robot via a communication network perform data communication is transmitted from the server to the robot. A communication method is disclosed in which, with respect to request task information, the robot transmits estimate information including execution availability information for the request task information to the server.

  In addition, a robot (cloud robot) system that uses a cloud computing platform to calculate, store, and cooperate has been proposed (Patent Document 2).

  That is, Patent Document 2 discloses a cloud robot system including a cloud computing platform and at least one robot, and the cloud computing platform is transmitted by at least one robot in the system. Receiving work information including data, status and request regarding the one robot, processing the data and status work information, returning a processing result to the robot, and responding to the request. The robot sends its work information to the cloud computing platform and processes the work information processed by the cloud computing platform. It receives the results, performs its operation according to the control command transmitted, cloud robot system, characterized in that disclosed by the cloud computing platform.

  In addition, a cloud computing system having a shared robot knowledge base for efficiently executing tasks by effectively using the knowledge obtained by one robot has been proposed (patent) Reference 3).

  That is, Patent Document 3 discloses a cloud computing system having a shared robot knowledge base that collects data related to an object encountered by the robot from the robot and shares the collected data. Accumulating in the knowledge base or instructing the robot to touch the object, receiving feedback based on the contact between the robot and the object from the robot, and updating the shared robot knowledge base based on the feedback Alternatively, a system for instructing a robot to execute an application based on information stored in the shared knowledge base is disclosed.

JP 2006-007341 A Special table 2013-536095 gazette US Pat. No. 8,639,644

  However, the conventional robot control or support based on cloud computing has not been disclosed in detail as to how to construct a knowledge base database. In particular, further improvement is expected in the case where the cloud platform or the cloud computing environment cannot respond to the request from the robot even if the knowledge base is currently referred to.

  Therefore, a cloud robotics system according to an embodiment of the present invention is a cloud robotics system including a cloud platform and a robot, and each of the cloud platform and the robot has a common interface for executing a task. The service module is executed, and the common interface includes at least a request for requesting execution of a task and feedback for reporting a status of execution of the task, and each of the service modules includes At least a working routine for executing the task and a failure detection subroutine for detecting whether or not the execution of the task has failed.

The system further includes a user terminal, and the service module further includes a rescue request routine for requesting rescue to the outside, and the failure detection subroutine detects that the execution of the task has failed In addition, the rescue request routine may request rescue to the user terminal.

  According to a cloud robotics system or the like according to an embodiment of the present invention, even if a cloud platform or a cloud computing environment cannot respond to a request from a robot even if a current knowledge base is referred to There is an effect that it can lead to a solution.

It is explanatory drawing explaining the example of whole structure of the cloud robotics system containing the robot, information processing apparatus, and cloud platform concerning one Embodiment of this invention. It is explanatory drawing explaining the example of a variation of the robot in the cloud robotics system concerning one Embodiment of this invention. It is explanatory drawing explaining the functional block of the hardware which comprises the robot concerning one Embodiment of this invention. It is explanatory drawing explaining the example of an external appearance structure of the information processing apparatus in the cloud robotics system concerning one Embodiment of this invention. It is explanatory drawing explaining the functional block of the hardware which comprises the information processing apparatus concerning one Embodiment of this invention. It is explanatory drawing explaining the detailed functional block structure of the cloud platform concerning one Embodiment of this invention. It is explanatory drawing explaining the basic concept of the processing routine (component of a processing service) in the cloud robotics system concerning one Embodiment of this invention. It is explanatory drawing explaining the processing routine (component of a processing service) in the cloud robotics system concerning one Embodiment of this invention. It is explanatory drawing explaining the operation | movement concept of the cloud robotics system concerning one Embodiment of this invention. It is explanatory drawing explaining the example of an application of the cloud robotics system concerning one Embodiment of this invention. It is explanatory drawing explaining the example of an application of the cloud robotics system concerning other embodiment of this invention. It is explanatory drawing explaining the example of an application of the cloud robotics system concerning other embodiment of this invention. It is explanatory drawing explaining the example of an application of the cloud robotics system concerning further another embodiment of this invention. It is explanatory drawing explaining the example of an application of the cloud robotics system concerning further another embodiment of this invention.

(Introduction)
An object of the present invention is to realize a system that does not require high specifications for a robot, among robot control systems or support systems based on cloud computing. For example, if a lot of expensive sensors are mounted on the robot, it is natural that a high-performance robot can be realized (for example, if a $ 3,000 laser sensor is mounted on the robot, it will not be adversely affected by sunlight described later. That's it) However, not all high-performance but expensive robots are readily available. In addition, even if the robot has advanced functions, the cost for improving its performance increases exponentially (for example, the human ability is assumed to be 100%, and the robot function is improved from zero to 90%. The cost to increase from 90% to 95% may be much higher than the cost to increase

Therefore, while emphasizing the processing in the cloud platform, it can be said that it is a good idea to realize a robot computing environment in the direction of realizing robots at low cost, but how to make up the remaining 10% is the next problem. . The inventor believes that the key to this problem is to effectively use the knowledge of the cloud platform and the knowledge of other users (humans) for troubles in robot operations and operations.
In particular, human knowledge is highly concentrated unlike current robots, so if you are a human being, you can take appropriate measures at a glance (even if the cloud platform requires a large amount of computation). Often obtained.
Furthermore, if such human knowledge can be utilized in the future operation and work of the robot, a more useful cloud robotics system can be provided. The present invention has been devised by the inventor from such a viewpoint.

  Hereinafter, a cloud robotics system and the like according to an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows an example of the overall configuration of a cloud robotics system according to an embodiment of the present invention.

  As shown in FIG. 1, the cloud robotics system 10 includes a cloud platform 11 and various information processing apparatuses used by a user (exemplarily portable information terminals or tablet terminals 12 in the figure). , A mobile phone 13 and a PC 14. Hereinafter, the mobile phone 13 and the PC 14 may be collectively referred to as “terminals”) and robots 201a to 201e. As shown in FIG. 1, the cloud platform 11, various terminals, and the robot are connected to each other via a public line (wired line 37 to 39) such as a dedicated line or the Internet. Further, the line may be wired or wireless, and in the case of wireless, the mobile phone 13 and the portable information terminal or tablet terminal 12 enter the Internet 39 via a base station, an access point, etc. not shown wirelessly. Further, they are connected to the cloud platform 11 via the line 38 so as to be able to communicate with each other.

When the robots 201a to 201e are flying robots such as drones, the robots 201a to 201e are connected to the Internet 39 via wireless base stations or access points 41 and 42 as shown in FIG. That is, the robots 201a to 201e are also configured to be capable of mutual communication with the cloud platform 11 and various terminals via the Internet 39.
Here, the access point is a wireless device for connecting wireless terminals such as a PC and a smartphone to each other or to connect to another network. Typically, it is a device that operates with communication protocols of the first layer (physical layer) and the second layer (data link layer) in the OSI reference model.

  Note that many mobile phones, personal digital assistants, and tablets at the time of filing of the present application have processing capabilities (communication processing speed, image processing capability, etc.) comparable to personal computers (PCs). It is something to say.

  In addition, a program or software necessary for implementing the present invention is normally installed or stored in an HDD, an SSD, or the like in a storage unit of a PC or a portable information terminal. When executing the program or software, a memory in the storage unit is necessary. Are read as all or a part of the software modules, and are executed by the CPU.

  Alternatively, a browser-based computer or a portable information terminal can be employed. In this case, the program is distributed from another server or computer to the terminal as necessary, and the browser on the terminal executes the program.

  Also, the hardware configuration of the cloud platform 11 can basically adopt a PC. Note that the present invention is not limited to this, but the cloud platform 11 is configured so that a plurality of PCs (several tens to tens of thousands as an example) are arranged in parallel to increase the hardware specifications as necessary. It is also possible to adopt a configuration suitable for processing large-scale data.

  FIG. 2 shows a variation example of the robot in the cloud robotics system according to the embodiment of the present invention. A robot according to an embodiment of the present invention includes a self-propelled cleaning robot in addition to the flying robot 201 shown in FIG. 2A (corresponding to the flying robots 201a to 201e shown in FIG. 1). Many robots, such as 202 and the autonomous walking-type assistance robot 203, can be included. For example, in addition to the variations shown in FIG. 2, various vehicles, electronic devices, control machines, etc. that operate or move autonomously, such as airplanes, ships, automobiles, rockets, artificial satellites, searchers, rescue robots, etc. are included. It is.

  FIG. 3 illustrates a functional block diagram of hardware constituting the robots 201 to 203 according to the embodiment of the present invention. For example, the operations of the robots 201 to 203 are realized by individual operations of hardware described below, and a cooperative operation between the software and these hardware.

In FIG. 3, the robot 300 as a whole hardware block is roughly divided into a motor unit 301 for driving a propeller (in the case of a flying robot) and wheels (in the case of a land traveling robot) serving as moving means, and an arm. In addition to an RGB camera, an actuator unit 302 for moving a movable hook, a hand, and the like, a camera unit 303 composed of an RGB-D camera, an infrared camera, etc., a microphone unit 304 for recording audio, a thermal sensor, a temperature Sensing unit 305 such as a sensor or gas sensor, control unit 306 for controlling various signals, CPU 307 for performing comparison / arithmetic processing, storage unit 308 such as RAM, ROM, flash memory, lamp, speaker, etc. An output unit 309, a communication unit 310 (whether wireless or wired), and a time count for measuring time, etc. And parts 311, a power supply unit 312, a gyro 313, consisting GPS314 Prefecture.
However, some of these devices may be replaced with other devices or omitted depending on the type of robot.

  As the power supply unit 312, a battery unit such as a lithium polymer (Li-Po) battery or a hydrogen fuel battery is employed. The gyro 313 employs an inertial sensor (6-axis) unit capable of detecting (rotational motion) acceleration and (translational motion) acceleration, which is necessary particularly in the case of a flying robot. The GPS 314 employs a general-purpose GPS unit in which an antenna and a receiver are integrally mounted.

  These modules are connected by a communication bus or a power supply line as needed (in FIG. 3, for convenience, each line is collectively shown as a connection line 399).

  In addition, a program or software executed on the robot 300 necessary for implementing the present invention is normally installed or stored in a hard disk disk, SSD (Solid State Drive), flash memory, or the like that constitutes the storage unit 307. When executing the software, all or part of the software module is read into the memory in the storage unit 307 as necessary, and the CPU 312 performs an operation.

  It should be noted that the execution of the calculation is not necessarily performed by the central processing unit 312 such as the CPU, and an auxiliary arithmetic unit such as a digital signal processor (DSP) (not shown) may be used.

  FIG. 4 shows an external configuration of the tablet terminal 12 as an information processing apparatus in the cloud robotics system according to the embodiment of the present invention. In FIG. 4, the information processing apparatus (tablet terminal) 12 includes a housing part 121, a display 122, and a hardware button 123 provided at the lower center part of the housing 121. The display 122 is typically composed of a liquid crystal display (LCD) or the like, and can display various information such as characters, still images, and moving images. In addition, a menu button or a software keyboard is displayed on the display 122 and can be used as an instruction (command) to the tablet terminal 12 by touching it with a finger or a touch pen (not shown). In this respect, the hardware button 123 is not an essential component, but is implemented as a button having a certain function for convenience of explanation of the present invention. Of course, these hardware buttons 123 can be replaced with menu buttons displayed on a part of the display 122.

  The display 122 includes a multi-touch input panel, and touch input position coordinates on the touch input panel are transmitted to the processing system (CPU) of the tablet terminal 12 via an input device interface (not shown). It is processed. And this multi-touch input panel is comprised so that the several contact point with respect to a panel can be sensed simultaneously. This detection (sensor) can be realized by various methods, and is not necessarily limited to the contact sensor. For example, an indication point for the panel can be extracted using an optical sensor. Furthermore, in addition to a contact sensor or an optical sensor, a capacitive sensor that senses contact with human skin can be used as the sensor.

  Although not shown in FIG. 4, the tablet terminal 12 can include a microphone and a speaker. In this case, it is possible to determine the voice of the user picked up from the microphone and use it as an input command. Further, although not shown in FIG. 4, a camera device such as a CMOS is mounted on the back surface of the tablet terminal 12 or the like.

  Furthermore, there is a 3D gesture interface as an application example that can be implemented in the tablet terminal 12 or other information processing apparatuses. This is a three-dimensional motion capture system consisting of a small motion controller, which is externally attached or built in, and dedicated software.

  Mainly, it is possible to capture and analyze the movement of a fingertip or a hand, and use it as a computer instruction operation. Typically, in a certain three-dimensional space in front of the controller or display, the user's hands and fingertips are tracked at high speed, and commands corresponding to the movements are given to various applications in a sensational and direct manner. can do.

  For example, you can move a finger or hand on the map application to enlarge / reduce the map, rotate a 3D object displayed in the viewer, or track the movement of the fingertip to make detailed signatures It is also possible to capture handwriting. As an example, the movement of a fingertip or the like can be detected with an accuracy of about 1/100 mm.

  It is also effective to employ a wearable computer such as a display device (head mounted display) worn on the head as another interface device. In this case, for example, by linking the robot's field of view to a screen such as a head-mounted display, the movement (direction) of the head of the user wearing the head-mounted display and the field of view (direction) of the robot can be linked. .

  FIG. 5 illustrates a functional block diagram of hardware constituting the tablet terminal 12 according to the embodiment of the present invention. The operation of the tablet terminal 12 is realized by individual operations of hardware described below, and a cooperative operation between software and these hardware.

In FIG. 5, the tablet terminal 500 as a whole hardware block is roughly divided into a hardware button 123 in FIG. 4, a multi-touch input panel provided on the display 122, an input unit 501 including a microphone, and a program, Storage unit 502 configured with a hard disk, RAM, and / or ROM for storing data and data, etc., central processing unit 503 configured with a CPU that performs various numerical calculations and logical operations according to programs, a display 122, etc. A communication interface including a display unit 504 configured by a control unit 505, a control unit 505 for controlling a chip, an electric system, and the like, a slot for accessing the Internet, a port for optical communication, and a communication interface Part 506, speaker and vibration , An output unit 507 such as an infrared projector, a timing unit 508 for measuring time, a sensor unit 509 including an image sensor such as a CMOS, an infrared sensor, an inertial sensor, and the like, and supplies power to each module in the apparatus These modules are connected by a communication bus or a power supply line as needed (in FIG. 5, they are collectively shown as a connection line 599 in which each line is appropriately divided for convenience).
The sensor unit 509 may include a GPS sensor module for specifying the position of the tablet terminal 500 (12). In addition, a signal detected by an image sensor such as a CMOS or an infrared sensor constituting the sensor unit 509 can be processed as input information in the input unit 501.

  In addition, a program or software executed on the tablet terminal 500 necessary for implementing the present invention is normally installed or stored in a hard disk disk, SSD (Solid State Drive), flash memory, or the like constituting the storage unit 502, and the program Or, when executing the software, the CPU 503 reads out all or a part of the software modules in the memory in the storage unit 502 as necessary, and executes the calculation in the CPU 503.

  It should be noted that the execution of the calculation is not necessarily performed by the central processing unit 503 such as a CPU, and an auxiliary arithmetic unit such as a digital signal processor (DSP) (not shown) can be used.

  FIG. 6 shows a detailed functional block configuration of a cloud platform according to an embodiment of the present invention. FIG. 6 exemplarily shows two cloud platforms 600 and 650. The present invention is not limited to this, but a plurality of cloud platforms can be assumed.

  The cloud platform 600 includes a server group 610 and a server group 620, and has several databases 631 to 633.

  The server groups 610 and 620 are not necessarily two, but may be a group of servers or a larger number of server groups. In the figure, the server groups 610 and 620 are connected by a line 641 regardless of wired / wireless, and are configured to be able to communicate with each other.

Further, the number of the databases 631 to 633 is not limited and can be adaptively increased or decreased. These databases 631 to 633 are configured to be accessible from both server groups 610 and 620 via lines 642 to 645.
Furthermore, information in which privacy is appropriately protected is shared in the databases 631 to 633. In addition, a data format and a process can be defined (for example, information such as a robot type and specifications can be registered in the database 633 in advance).

The cloud platform 650 is another cloud resource (for example, a third party API) when viewed from the cloud platform 600. Similarly to the cloud platform 600, the cloud platform 650 is configured by hardware and software resources that can be adaptively increased and decreased, and is connected by a line 646 that can communicate bidirectionally regardless of wired / wireless.
In the figure, the cloud platform 650 illustratively includes a server (group) 651 and a database 652, and is connected by a line 653 that can communicate with each other.

  The cloud platforms 600 and 650 are constructed or configured as a secure and variable capacity platform.

  These cloud platforms (cloud resources) can communicate with various devices via a bidirectional standard communication interface 695.

  In the figure, robots 681 to 683 and an information processing apparatus (terminal) 684 are shown as examples of various devices. The robots 681 to 683 normally operate in cooperation with the cloud platform. However, when necessary, the robot 681 detects a network abnormality and switches to a complete and safe stand-alone system or device when an abnormality is detected. It can also be controlled.

  The information processing apparatus (terminal) is used by a user or an administrator, and can change the system configuration or monitor the status via, for example, a browser via the information processing apparatus. Furthermore, robot monitoring and control via a browser are possible via a cloud platform.

Next, with reference to FIGS. 7A and 7B, a basic concept of a processing routine (components of a processing service) in a cloud robotics system according to an embodiment of the present invention will be described.
The processing service is composed of several processing routines that are constituent elements. Details will be described below.

  FIG. 7A illustrates a software configuration of a processing routine as a single service. The service 70 includes internal software modules 720, 730, and 740 that can communicate with external modules via service interfaces 711 to 713. (As will be described later, the internal software module 720 is configured to be communicable with other external modules via the same interface as the service interfaces 711 to 713).

  The internal software modules 720, 730, and 740 are further divided into a working routine, a failure detection subroutine, and a rescue request routine as detailed role assignments. Hereinafter, the internal software modules 720, 730, and 740 are also referred to as a working routine 720, a failure detection subroutine 730, and a rescue request routine 740, respectively.

  The working routine 720 is a software module that executes a main task in accordance with a request received from an external software module. As will be described later, a request can be issued to another external module through an interface similar to the service interfaces 711 to 713.

  The failure detection subroutine 730 can determine by itself that the task has failed during the execution of the task that has received the request, and also determine that the request to the external module has failed from the feedback status from the external module. Can do.

  The rescue request routine 740 is a software module that calls for help to a cloud platform or the like for a request or task that is determined to have failed. In this case, if necessary, a software interface for requesting rescue to the cloud platform or the like may be provided separately from the service interfaces 711 to 713 (the rescue request is sent to the terminals 791 to 793 via the cloud platform. (Details will be described later).

  The service interfaces 711 to 713 are communication interfaces when a request is received from an external module (similarly, when a request is issued to the external module, the request is issued from the working routine 720. Connection between the two parties) The situation will be described later with reference to FIG. 7B).

  The service interface 711 is an interface for receiving a request from an external module, the service interface 712 is an interface for returning feedback to the external module that has issued the request, and the service interface 712 This is an interface for returning a result (typically a binary response such as true / false) to the issued external module.

  In the cloud robotics system according to the embodiment of the present invention, the relationship of receiving a request from another single service or issuing a request to another single service based on the single service shown in FIG. The main feature is that the whole process is advanced while building organically (dynamically).

  Here, when two or more services construct a connection relationship and proceed with processing, the service that issues a request is a high-level service, and the service that receives the request is a low-level service.

  FIG. 7B shows the relationship between the high-level service and the low-level service that have established such a connection relationship.

A typical interaction between the high-level service 750 and the low-level service 760 in the basic service connection structure shown in FIG. 7 follows the following model.
(A) Initialization within the high-level service 750 as necessary. This initialization includes setting a goal.
(B) A request or task request from the high-level service 750 to the low-level service 760 (in FIG. 7B, the request is transmitted from the working routine 7520 to the service interface 7611).
(C) Processing in the low-level service 760. Although not shown in FIG. 7B, the processing may be further broken down into more detailed tasks and requests and sent to other services or routines that are further lowered (in this case, the working routine) 7620 makes a request to an external service).
(D) Feedback from the low level service 760 to the high level service 750. The status detected by the low level service 760 is returned.
(E) Judgment at the high level service 750. For example, it is determined whether the progress of task processing toward the goal is preferable (whether it has failed) or whether external help is required. In this case, the determination of failure is determined by the failure detection subroutine 730, 7530, 7630 in FIG. 7, and the rescue request when it is determined that external help is necessary is determined by the rescue request routine 740, 7540, 7640 in FIG. To be implemented.
(F) Binary response from low-level service 760 to high-level service 750 (for example, binary response of true (1) or false (0))

  The models shown in FIGS. 7A and 7B are the simplest models, and there may be a model with a more complicated hierarchical structure (Fundamental Build Up Model). For example, a model having a three-layer structure, a five-layer structure, a model having a structure higher than that, and the like.

  Also, high-level services are always on the cloud platform side, and low-level services are not always on the robot side. It is possible that both are on the cloud platform side, and vice versa. Alternatively, some high-level services (or low-level services) may be on the cloud platform and others may be on low-level services.

Furthermore, in the process where the high-level service generated on the cloud platform makes a task request (service call) to the low-level service in the robot and executes the service on the robot, this time, the robot to the cloud platform A task request (service call) may be made.
Such high-level services and low-level services are dynamically generated in parallel and can have various hierarchical structures.

  In addition, the failure detection method (criteria for determining failure) is that there is no response after a certain period of time (so-called time-out) in addition to detecting that a specific event or status has returned. Is included.

  FIG. 8 shows a concept of a processing service in the cloud robotics system according to the embodiment of the present invention, which has a hierarchical processing structure (layer structure) based on the model described with reference to FIG. Is illustrated.

In the model shown in FIG. 8, there are four layers 1-4. “Goal G”, which is an ultimate service, is set in the layer 1, and there are three services 2A to 2C for achieving the “goal G” in the layer 2 that is at a lower level than the layer 1. .
From the service 2A of the layer 2, there is a service 3A in the layer 3 that is at a lower level than the layer 2, and from the service 2B, there is a service 3B in the layer 3.

  Furthermore, another service 3C for achieving the “goal G” exists in the layer 3 (note that this service 3C did not exist when the services 2A to 2C were generated. Details of processing) Will be described later). From the service 3C, there is a service 4A in the layer 4 that is at a lower level than the layer 3.

  Here, as a specific example in which the model shown in FIG. 8 is applied, a task execution instruction (request or request) that the cloud platform “provides a cup on the table” is given to the robot in the room. This is also referred to as a service call.

At this time, the goal G of layer 1 is

G: “Please bring a cup on the table”

It is a service call from the cloud platform to the robot.

Then, for example, the following services can be associated with the three services 2A to 2C that branch from G in layer 1 to layer 2 respectively. Both are service calls from the cloud platform to the robot.

2A: “Find the table”
2B: “Check and hold the cup on the table you find”
2C: “Come back with a glass”

The following feedbacks 1 to 3 are returned from the robot executing the service 2A to the cloud platform.

Feedback 1: “I am looking for an object in the room now”
Feedback 2: “I found a chair”
Feedback 3: “I found a chiffon”

This time, there is a service 3A which is a low level in layer 3 with the service 2A of layer 2 (find the table) as a high level service.

3A: “Is this a table (with image transmission)?”

This is a service call from the robot to the cloud platform. A binary response of Yes (true) or No (false) is returned from the cloud platform to the robot. Here, Yes (true) is returned, and the robot executes the service 2B (check and hold the cup on the found table). The following feedbacks 4 to 5 are returned from the robot executing the service 2B to the cloud platform.

Feedback 4: “I'm looking for an object on the table now”
Feedback 5: “I found the book”

Further, there is a service 3B which is a low level in the layer 3 with the service 2B of the layer 2 (please confirm and hold the cup on the table) as a high level service.

3B: "Is this a cup (with image transmission)?"

This is a service call from the robot to the cloud platform. A binary response of Yes (true) or No (false) is returned from the cloud platform to the robot. Here, it is assumed that Yes (true) is returned, and the robot returns the next feedback 6 in an attempt to further execute service 2B (check and hold the cup on the found table).

Feedback 6: “I do n’t know how much I should hold this glass.”

It is assumed that the cloud platform that has received the feedback 6 tries to find out the power of the cup on the database, but cannot find it. Then, from the robot,

Feedback 6: “I do n’t know how much I should hold this glass.”

The same feedback will be returned, or a certain time will elapse without feedback for a while (it will time out).

In this case, in the cloud platform, it is determined that the service 2B has reached a dead end, and as an example, after the service 2B has disappeared, a new service 3C is generated. The service 3C is generated as a new service call as a result of verification based on the result of the cloud platform inquiring another platform or the other user (human), and is transmitted to the robot.

3C: “Hold the cup with a force of 0.5N or more and less than 0.9N”

Then, there is a service 4A at a low level in the layer 4 with the service 3C of the layer 3 as a high level service.

4A: “Can I have this position (with sending image)?”

This is a service call from the robot to the cloud platform. Here, it is assumed that a binary response of No (false) is returned from the cloud platform to the robot (for the reason that the position is particularly fragile).

In this way, the robot starts to have the correct position at the correct force, and sequentially returns the following feedbacks 7 to 9 to the cloud platform.

Feedback 7: “I'm holding at 0.5kg”
Feedback 8: “Now hold at 0.8Kg”
Feedback 9: “Now the cup is lifted”

(At this point, “true” is returned as a binary response. The task is complete.)

Next, the robot executes the service 2C of layer 2 (return with a cup) and sequentially returns the following feedback 10 to 14 to the cloud platform.

Feedback 10: “We have changed direction now.”
Feedback 11: “I'm currently at (X ₁ , Y ₁ ).
Feedback 12: “I'm currently at (X ₂ , Y ₂ ).
Feedback 13: “I'm currently at (X ₃ , Y ₃ ).
Feedback 14: “I'm back now.”

(At this point, “true” is returned as a binary response. The task is complete.)

  Although not shown, another example is a “security system that ensures the safety of the home of the husband who is on a business trip.” The configuration of each layer and each service in that case is as follows.

Layer 001 (the next final goal exists)
001G: Ensure the safety of the husband who is on a business trip

Layer 002 (there are the following three services derived from 001G)
002A: Do not enter the thief (check for suspicious persons)
002B: Do not encounter fire (including checking neighbors)
002C: Prevent damage from typhoons and hurricanes (check weather information, etc.)

Layer 003 (the next service derived from 002A exists)
003A: Go around the house

Layer 004 (there are the following three services derived from 003A)
004A: Go around the house on Route A 004B: Go around the house on Route B 004C: Go around the house on Route C

  FIG. 9 shows an operation example of the cloud robotics system according to the embodiment of the present invention. In order to facilitate understanding, a more specific description will be given based on the above (a model executed by cloud robotics to ensure the safety of the home of the husband who is on a business trip). The premise of the camera device mounted on the robot in this embodiment will be briefly described.

(1) First, it is assumed that the robot in the figure has an RGB-D camera (sensor) mounted for navigation. From this RGB-D camera (sensor), two data of (a) RGB image and (b) depth information are streamed. These streaming data are used in a “navigation service” for moving the robot.
(2) Next, when the mobile environment is strongly influenced by sunlight, an empty value is transmitted from the sensor. The failure detection subroutine detects this state (as failure).
(3) The rescue request routine can make a rescue request to another user as necessary. For example, a specific user is allowed to remotely control the robot. An authorized user relies solely on the RGB image stream (i.e., without depth information) to remotely control the robot using his human experience.

  In addition, the outline of the operation will be described. In the figure, from time t1 to t4, an inquiry is made about a visit of a person who the robot has discovered through the window while traveling around route A and has not seen the robot itself. An operation example is shown. Specifically, a service call is made to the cloud platform together with the person image.

  In the same figure, from time t5 to t11, while the robot patrolled Route B, a camera malfunction (saturation phenomenon) occurred due to sunlight insertion or reflection, or it was not possible to move forward. Indicates remedies in case of danger when moved. Specifically, the robot in such a trouble returns feedback to the cloud platform that RGB information can be acquired but D information (depth information) cannot be acquired, so it cannot move (cannot move). In this case, in the cloud platform, for example, if there is a control program for moving the robot using only the RGB image, this program is started to control the robot from the cloud side. . When there are proposals from a plurality of users in response to this inquiry, the cloud platform verifies those proposals and, as an example, permits one user to perform a robot teleoperation. The robot adjusts the camera and changes the flight route (changes the altitude so as not to be affected by sunlight) by the user's teleoperation to solve the trouble. The contents of the operation and the improvement status are learned and used on the next day. In other words, on the next day, the robot that circulates Route B encounters the trouble that the camera becomes invisible due to sunlight insertion or reflection when it reaches the same position at the same time. As a result of learning, an appropriate camera adjustment and / or change of the flight altitude is performed, and this trouble is successfully avoided.

  At time t1 in FIG. 9, a service call 1 (a request to “travel around the house on route A”) is transmitted from the cloud platform to the robot (step S901).

  Next, at time t1 to t2, a request is executed by the robot, and sequential feedback is sent to the cloud platform (step S902). For example, the robot guards the inside of the master's home and reports its position and surrounding conditions to the platform at regular intervals.

  At time t2, since there was a visit of a person who cannot be analyzed by the robot itself, the cloud platform is requested to analyze the image taken from the window through the window (service call A in step S903). For example, a robot asks a person wearing an unfamiliar uniform with the question of “possibly a suspicious person” and sends an image of the person (which may be a moving image) to the cloud platform and requests analysis such as image recognition. .

  From time t2 to t3, object recognition processing is performed on the image (or moving image) sent from the robot on the cloud platform, and the person captured in the image is a regular postman Recognize It is determined that even a person wearing an unfamiliar uniform “accepts a legitimate visitor (and therefore does not take any special precautions)” (step S904).

  At time t3, the analysis processing on the cloud platform is completed, and a binary response is returned to the robot (step S906). In this case, since there is no particular problem, True is returned.

  From time t3 to t4, the service of service call 1 (circulation of route A) is continued by the robot, and successive feedback is sent to the cloud platform (step S907).

  At time t4, since the route A has ended, the robot reports completion to the cloud platform (a true binary response is returned).

  At time t4, the cloud platform accumulates and stores new data obtained from the current interaction with the robot (step S909), and the database is reconstructed (step S910).

  Next, after time t5, an operation example when the cloud platform needs help is shown.

  First, at time t5, service call 2 (go around the house on route B) is transmitted from the cloud platform to the robot (step S951).

  From time t5 to t6, a request is executed by the robot, and a sequential feedback is sent to the cloud platform (step S952). For example, the robot guards the inside of the master's home and reports its position and surrounding conditions to the platform at regular intervals.

  At time t6, it is difficult for the robot traveling around the house along the route B to block the field of view due to the insertion or reflection of sunlight and to move forward with nothing visible. Can be acquired, but D information (depth information) cannot be acquired, so feedback is returned to the cloud platform that it cannot move (cannot move) (step S953).

  From time t6 to t7, attempt to solve the problem by referring to the database in response to feedback from the robot on the cloud platform (for example, starting a control program that moves the robot using only RGB images) However, no appropriate treatment is found (step S954).

  Therefore, at time t7, it is determined that the cloud platform is another cloud platform or that a help request for another user (human) is necessary.

  Next, from time t7 to t8, the cloud platform performs examination processing such as what kind of inquiry should be made (step S955). For example, whether to make an inquiry to another cloud platform (for example, 650 in FIG. 6), or to make an inquiry to a user (human) widely as described later.

At time t8, the cloud platform determines that it is better to make a wide inquiry to third parties (other human users), and makes an inquiry to many user terminals (information processing devices) that can be connected to the cloud platform via the network. This is performed (step S956).
For example, the position and route map sent from the robot, the image or video of the site, etc. are distributed, and an inquiry such as “How should the robot proceed in this state?” Is made. At this time, processing for ensuring privacy is performed as necessary (described later with reference to FIGS. 13 to 14).

  Here, the camera used by the robot for its own movement is a single RGB-D camera or sensor that can obtain depth information, and when the flare caused by the reflection of sunlight described above occurs, this depth is used. Information cannot be obtained. On the other hand, an RGB image obtained from a normal RGB camera may not be significantly affected by the flare described above. In such a case, an on-site image or video by the normal RGB image may be displayed by another user (human). It can be fully expected that the situation can be judged by seeing.

  From time t8 to t9, answers or advice are returned from one or more users in response to inquiries from the cloud platform (step S957). For example, advice such as “just adjust the camera (RGB-D sensor) in this way”, “you only have to follow this route”, or “the effect of sunlight decreases with decreasing altitude”. Will be sent.

  Next, from time t9 to t10, the cloud platform verifies advice from one or more users sent from time t8 to t9 (step S958), and teleoperation is performed for one of the users P. Is permitted (step S959).

  From time t10 to time t11, the user P who is permitted to perform teleoperation teleoperates the robot to avoid trouble (step S960). For example, the robot's camera is adjusted or the flight altitude is appropriately lowered to improve the visibility of the robot (by the RGB-D sensor), and the movement of the area is manually cut by the user.

  Next, at time t11, it is reported that the trouble has been safely avoided from the robot to the cloud platform (step S961). In the cloud platform, information accumulated so far is organized and stored as new knowledge (step S962), and a database is reconstructed as necessary (step S963).

  For example, in the cloud platform, the date and place where the robot fell into trouble this time, the situation of the trouble (depth information became empty, that is, it was strongly affected by sunlight), and this trouble Measures taken by other users (operation histories such as changing camera settings or changing flight altitude) are stored in a database so that they can be searched.

  Through the processing (learning processing) from step S962 to step S963 described above in the cloud platform, the robot (even if it is another robot) is already in the database even if it passes the same area at the same time zone the next day. Since the trouble avoiding method in the same situation has been made into a database, the RGB-D sensor adjustment and the flight altitude are appropriately adjusted in advance based on the learning content without suffering the same trouble as this time You can safely pass through this area by changing to

  FIG. 10 shows an application example of a cloud robotics system according to an embodiment of the present invention.

FIG. 10A shows how two robots with cameras 1001 and 1002 collaborate to collect the three-dimensional space data of the room 1003 and build a virtual space on the cloud platform. ing. In the figure, white portions (portions lacking in white) are portions that have not yet been captured by the robots 1001 to 1002. The part captured once by the camera-equipped robot 1001 or 1002 basically does not need to be captured by the other robot.
Therefore, the entire virtual space can be constructed more quickly by performing the capture operation with three robots than three and four robots than three.

  Next, in FIG. 10B, a large number of robots with cameras (not shown) build a three-dimensional structure on the cloud platform that looks like a maze while mutually linking the buildings. Show the state. In the figure, the structure of the floor on the premises is partially analyzed, and the passages 1011 and 1012 arranged substantially in parallel, the room 1013 connected to the passages 1011 and 1012, and the other passages 1014 to 1016. Structural analysis is ongoing.

  In FIG. 10B, it is known that the passages 1014 and 1015 are divided through a partition. On the other hand, it is known that the passages 1015 and 1016 communicate with each other without a partition. Analysis by a large number of robots with cameras will continue, and it may be found that there is a door that can be opened and closed in the partition between the passages 1014 and 1015, and the connection relationship between other passages has also become clear. Will go.

  Data collected from each robot is collected in a cloud platform and organized and stored as 3D structure data on the premises.

  11 and 12 show examples of applications of a cloud robotics system according to another embodiment of the present invention. Specifically, a robot searches a certain premises, gets lost when trying to create a database on the cloud platform for directions from the start (Start) to the exit (Goal), and is rescued by the user (human) (help) ) Is shown.

Note that the application examples shown in FIGS. 11 and 12 are described based on a two-dimensional model for the sake of understanding of the present invention. In other words, with the current computer performance, it is easy to search the two-dimensional maze and reliably guide the robot from the start (Start) to the exit (Goal). Realistically, it may be appropriate to let the robot search from the first floor to a point on the higher floor in a high-rise building where each floor has a complex maze-like structure (the amount of calculation is Because it increases exponentially compared to the two-dimensional case).
However, in this embodiment, the description will proceed based on a simplified two-dimensional model for the sake of understanding of the present invention.

First, in FIG. 11 (A), a robot searches for a certain premises and tries to create a database on the cloud platform on the route from the entrance (Start) to the exit (Goal). The state of being closed is shown (the cross indicates the current position of the robot).
At point K, the robot returns feedback to the cloud platform that “I don't know where to go from here”.

  In the cloud platform, it is determined that the robot is lost based on the feedback of the robot (for example, the same feedback is returned or the next feedback is not returned within a certain time).

  The cloud platform searches its own database for other information about this premises, but cannot find it, but queries other cloud platforms, but still does not provide useful information.

  Therefore, the cloud platform determines to request relief from another user (human).

  Next, the cloud platform uses a map constructed based on the data provided by the robot and the route created by searching for the robot so far (from the Start in FIG. 11A to the point A → B → C → D → Visual information indicating E → F → G → H → I → J → K) is sent to one or more user terminals (information processing devices).

  One or more users who have received inquiries from the cloud platform have a map, the current position of the robot, and the route of the robot (FIG. 11A) shown on the display of their terminal (information processing device). Give advice.

  This advice is not necessarily compulsory for the user, but it can increase the number of advice by giving certain points or implementing elements of gamification. .

  FIG. 11B shows an example where the user advises a route or route by direct operation. In the figure, the user corrects and advises the robot's previous route in the visual information (visual data) displayed on his / her terminal so that the route becomes a more appropriate route by direct operation. be able to.

  For example, a point 1101 in the middle of the path is indicated and moved up and down, another point 1102 is moved left and right, or another point 1103 in the middle of the path is indicated in an oblique direction. For example, it can be moved up, down, left and right, or it can be moved up, down, left and right by indicating the end point 1104.

  In addition to this, the user can reconfigure a new route by picking and moving any point in the route or extending / shortening it to give advice to the cloud platform (or robot). Can do.

  In this example, the explanation is focused on the editing or correction of the route data in the two-dimensional plane. However, if the route data in the three-dimensional space is edited or corrected, it can be experienced through the 3D gesture interface. A direct instruction method can also be adopted.

12A to 12C show various routes that have been advised by other users for the visual information shown in FIG. 11A by the editing or correction method illustrated in FIG. 11B. It is shown.
The cloud platform verifies these advices, determines which advice is most appropriate, and sends the optimal advice to the robot. It is also possible to permit teleoperation of a lost robot to a user who has given optimum advice as necessary.

  In FIG. 12 (A), the route shown in FIG. 11 (A) is corrected by the user U1, and the dotted line route from Start to points A → B → C → D → E → F → G → H → N Has been advised as the best route.

In the cloud platform, this advice from the user U1 is verified, for example, the status of the route newly shown this time is verified by referring to the premises data constructed so far, for example, the path to the points H to N There is a staircase (in this figure, it is an area that expresses a step with a one-dot broken line), and this section derives that it cannot run with the specifications of the robot that is currently lost.
In this case, the route shown in FIG. 12A is not adopted.

  Next, in FIG. 12 (B), the route shown in FIG. 11 (A) is corrected by the user U2, and the dotted line route from Start to the points A → B → C → L → M → N is optimal. Advised as a route.

In the cloud platform, this advice from the user U2 is verified, for example, the status of the route newly shown this time is verified with reference to the premises data constructed so far, for example, the path to the points C to L Since the width of is narrower than the width of the robot, it is derived that this section cannot be run.
In this case, the route shown in FIG. 12B is not adopted.

  Next, in FIG. 12 (C), the route shown in FIG. 11 (A) is modified by the user U3, and the dotted line that continues from Start to the points A → B → C → D → E → F → M → N The route is advised as the best route.

  In the cloud platform, this advice from the user U3 is verified, for example, by referring to the data constructed so far, the status of the route newly shown this time is verified, and it is confirmed that there is no particular problem. The

Then, the route shown in FIG. 12C is adopted and transmitted to the robot. The robot can safely reach the exit (Goal) according to the route sent from the cloud platform (the route in FIG. 12C).
Note that the cloud platform can permit the user U3 to teleoperate the robot as necessary.

  FIGS. 13 and 14 show still other application examples. In particular, an example of processing at least a part of an image shown to other users from the cloud platform and protecting the privacy of an object or person included in the image. Show.

  FIG. 13A shows an image 131 in which an entrance (door) 1319 of a building is copied. A plate 1311 is installed on the left side of the door 1319. The cloud platform receives an inquiry without knowing how to open the door, and the image is sent as an image 131 to inquire about how to open the door by sending this image to many users without being able to solve it. (The image 132 in FIG. 13B and the image 133 in FIG. 13C are obtained by processing a part of the image 131 as described later).

The image 131 shown in FIG. 13A is an unprocessed image, whereas the image 132 shown in FIG. 13B has mosaic processing 1321 applied to the plate portion for privacy protection. It has been subjected. Further, the door 1329 is subjected to a coloring process (for example, green) different from the actual color (blue) of the door 1311.
In addition, the image 133 shown in FIG. 13C is subjected to concealment processing 1331 and 1332 with respect to objects and parts other than the door 1339 as not directly related to the inquiry matter. In addition to changing the transparency of the area to be concealed, this concealment process performs various processes such as mosaic processing as shown in Fig. 13 (B) and complete concealment with other patterns. It can be carried out.
Further, as described above, the color of the door shown to the user can be changed, or the color of another object can be colored different from the actual color.

  FIGS. 14A and 14B are images in which a certain person is imprinted. Illustratively, the person 1401 is a policeman and the person 1411 is Postman.

  In the cloud platform, the robot receives an inquiry without being able to identify these persons, and cannot resolve itself, but sends this person image to many users and cuts out the person images 1402 and 1412.

Here, in the image 1402 shown in FIG. 14A, a mosaic process 1403 is applied to the face portion of the person, and similarly, in FIG. 1412, the mosaic process 1413 is applied to the face portion of the person.
Such mosaic processing maintains the privacy of the person. Of course, in addition to the mosaic processing shown in the figure, various processing can be performed such as processing by covering the faces of cartoon characters and animals, or completely hiding with other patterns.

  As described above, the embodiments of the cloud robotics system, the information processing apparatus, and the like have been described based on the specific examples. However, as the embodiment of the present invention, the program is recorded in addition to the method or the program for implementing the system or the apparatus. It is also possible to take an embodiment as a storage medium (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card).

  Further, the implementation form of the program is not limited to an application program such as an object code compiled by a compiler and a program code executed by an interpreter, but may be a form of a program module incorporated in an operating system. good.

  Furthermore, it is not always necessary for the program to be executed only by the CPU on the control board, but if necessary, an expansion board attached to the board or another processing unit (DSP or the like) mounted on the expansion unit. ) May be partly or wholly implemented.

  These features are mutually exclusive for all of the components described in this specification (including claims, abstract, and drawings) and / or for all steps of all disclosed methods or processes. Except for the combination, any combination can be used.

  Also, each feature described in the specification (including the claims, abstract, and drawings) serves the same purpose, equivalent purpose, or similar purpose, unless expressly denied. Alternative features can be substituted. Thus, unless expressly denied, each feature disclosed is one example only of a generic series of identical or equivalent features.

  Furthermore, the present invention is not limited to any specific configuration of the above-described embodiment. The invention includes all novel features or combinations thereof described in the specification (including claims, abstract and drawings), or all novel methods or process steps described, or Can be extended to any combination.

10 Cloud Robotics System 11 Cloud Platform 12 Tablet Terminal (One Form of Information Processing Device)
13 Mobile phone (a form of information processing device)
14 PC (One form of information processing device)
201a to 201e flying robot (one form of robot)
41, 42 Access point (router, etc.)
37, 38 Communication line 39 Public line (private line, Internet, etc.)

Claims (18)

A cloud robotics system comprising a cloud platform and a robot,
The cloud platform and the robot each execute a service module having a common interface for executing a task,
The common interface has at least a request for requesting execution of a task and feedback for reporting a status of execution of the task,
Each of the service modules includes at least a working routine for executing the task and a failure detection subroutine for detecting whether the execution of the task has failed. The system further includes a user terminal having a user interface, and the service module further includes a rescue request routine for requesting rescue to the outside.
The system according to claim 1, wherein when the failure detection subroutine detects that the execution of the task has failed, the rescue request routine requests the user terminal to rescue.   The system according to claim 2, wherein the cloud platform permits the user terminal to operate the robot.   The system according to claim 3, wherein permitting the operation on the robot is an operation permission by direct operation on visual information displayed on a display of the user terminal.   The system according to claim 2, wherein the user interface is a 3D gesture interface.   The system according to claim 2, wherein the user interface includes a head mounted display.   The system according to any one of claims 2 to 6, wherein information obtained from the user terminal that requested the rescue is stored in a database so as to be searchable.   The information stored in the database includes a situation of the robot that has requested the rescue, and the information is used for rescue of another robot that falls into the situation. Item 8. The system according to any one of Items 2 to 7.   The image for display on the user terminal is transmitted in order to make the rescue request, and at least a part of the image is processed for privacy protection. The system according to item 1. A computer program executed in a cloud robotics system comprising a cloud platform and a robot, and when executed in the system,
Processing steps for causing the cloud platform and the robot to each execute a service module having a common interface for executing a task;
A processing step for causing the service module to make a request for at least requesting execution of a task via the common interface, and a processing step for causing feedback to report the status of execution of the task;
Where:
The service module further includes a processing step for causing the working routine to execute the task, and a processing step for causing the failure detection subroutine to detect whether or not the execution of the task has failed. The system further includes a user terminal having a user interface, and the service module further includes a rescue request routine for requesting rescue to the outside.
11. The program according to claim 10, further comprising a processing step of causing the rescue request routine to request rescue from the user terminal when the failure detection subroutine detects that execution of the task has failed.   The program according to claim 11, further comprising a processing step of allowing the user platform to allow the user terminal to operate the robot.   The program according to claim 12, wherein the processing step of permitting an operation on the robot is a processing step of permitting an operation by direct operation on visual information displayed on a display of the user terminal.   The program according to any one of claims 11 to 13, wherein the user interface is a 3D gesture interface.   The program according to any one of claims 11 to 14, wherein the user interface includes a head-mounted display.   The program according to any one of claims 11 to 15, further comprising a processing step of creating a database so that information obtained from the user terminal that has requested the rescue is searchable.   The information stored in the database includes a situation of the robot that has requested the rescue, and the information is used for rescue of another robot that falls into the situation. Item 17. The program according to any one of Items 11 to 16. In the cloud platform,
A processing step of transmitting an image to be displayed on the user terminal in order to make the rescue request;
The program according to claim 11, further comprising a processing step of executing processing for privacy protection on at least a part of the image.