JP6921448B1 - Control program and control method for new object operation robot, and new object operation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control program and a control method of a robot capable of stably operating a new object, and an object operation system. SOLUTION: When a computer receives an operation command indicating to operate an operation object, a determination unit for determining whether or not the operation object is an object to be newly operated, and the operation. When it is determined that the object is an object to be newly operated, the operation is performed by using the operation environment based on the information acquired by the sensor, the information about the operation object, and the behavior data learned in advance. A simulation unit that simulates the operation of the operation object by the robot body in response to a command, and the robot body is operated in real space so as to operate the operation object in response to the operation command based on the result of the simulation. A robot control program that functions as an actual operation control unit to be controlled is provided. [Selection diagram] FIG. 2B

Description

The present invention relates to a control program and a control method for a robot that operates a new object, and an object operation system that operates a new object.

In general, a simulation related to the operation of a specific object is performed offline in advance to predict a physical motion, and the object is operated. For example, in factories and the like, it is widely practiced to use a robot to transport an object (for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2006-21526

However, it is not always possible for such a robot to stably operate an object with respect to a new object. Moreover, although the development of artificial intelligence has been remarkable in recent years, it is not realistic to learn in advance for all objects.

An object of the present invention is to provide a control program and a control method for a robot capable of stably operating a novel object, and an object operation system.

According to one aspect of the present invention, when the computer receives an operation command indicating that the operation object is operated by the computer, it is determined whether or not the operation object is an object to be newly operated. When it is determined that the operation object is an object to be newly operated, the operation environment based on the information acquired by the sensor, the information about the operation object, and the pre-learned action data are provided. The robot uses a simulation unit that simulates the operation of the operation object by the robot body in response to the operation command, and the robot so as to operate the operation object in response to the operation command based on the result of the simulation. A robot control program that functions as an actual operation control unit that controls the main body in a real space is provided.

The computer may function as a learning unit that learns the result of the simulation by the simulation unit and registers it as new behavior data.

The learning unit may learn the result of the operation of the robot body in the real space and register it as new action data.

The actual operation control unit may control the robot body so as to operate the operation target object in response to the command based on the result of performing the simulation a plurality of times while changing the action data.

The actual operation control unit determines whether or not the result of the simulation and the state of the robot body controlled by the actual operation control unit are different according to a predetermined standard, and when it is determined that they are different. May stop the operation of the operation object by the robot body.

The operation of the operation object may be to stably place the operation object.

The operation of the operation object may be to carry the operation object to the destination indicated by the operation command.

When there are a plurality of successful results of moving to the destination as a result of the simulation, the actual operation control unit is based on the action data used in the simulation in which the moving time to the destination is the shortest. The robot body may be moved.

If there is no successful movement to the destination as a result of the simulation, the simulation unit may determine that the movement in response to the operation command is not possible.

The actual operation control unit determines whether or not the movement of the robot body differs from the result of the simulation during the movement of the robot body according to a predetermined criterion, and if it is determined that the movement differs from the result of the simulation. The movement of the robot may be stopped.

The information regarding the operating environment may include information on the floor surface to the destination, and the information regarding the operating object may include the weight of the operating object.

According to another aspect of the present invention, when an operation command indicating that the operation object is operated is received, it is determined whether or not the operation object is a newly operated object, and the simulation unit determines whether or not the operation object is a newly operated object. When it is determined that the operation object is a newly operated object, the operation environment based on the information acquired by the sensor, the information about the operation object, and the behavior data learned in advance are used. The robot simulates the operation of the operation object by the robot body in response to the operation command, and the actual operation control unit operates the operation object in response to the operation command based on the result of the simulation. A robot control method for controlling the main body in real space is provided.

According to another aspect of the present invention, when the sensor, the robot body, and the operation command indicating to operate the operation object are received, whether or not the operation object is an object to be newly operated. The determination unit for determining, information on the operation environment and the operation object based on the information acquired by the sensor when it is determined that the operation object is a newly operated object, and pre-learned behavior data. And, a simulation unit that simulates the operation of the operation object by the robot body in response to the operation command, and the operation object is operated in response to the operation command based on the result of the simulation. An object operation system including an actual operation control unit for moving the robot body in a real space is provided.

Safe movement can be achieved even when the environment or the object to be operated changes.

The flowchart explaining the decision-making process by this invention. The schematic block diagram of the object operation system which concerns on one Embodiment. A block diagram of a more detailed object manipulation system. Schematic block diagram of an object manipulation system in a virtual space where a simulation is performed. The flowchart which shows an example of the processing operation of the control computer 3.

Hereinafter, embodiments according to the present invention will be specifically described with reference to the drawings.

FIG. 1 is a flowchart illustrating a decision-making process according to the present invention. When it becomes a situation where some action should be decided (YES in step S21), it is confirmed whether or not there is sufficient information (step S22). If there is sufficient information, the action is selected by the physics simulator (step S25). On the other hand, when there is not enough information (NO in step S22), information in the real world is acquired (step S23). The information in the real world is information about an object to be operated (hereinafter referred to as "object to be operated") and information about an operation environment. Then, the acquired information is inserted into the physics simulator (step S24), and the action is selected by the physics simulator (step S25).

The present invention is characterized in that information is acquired in real time on the spot when there is not enough information at the time when the action should be decided, in other words, when the information acquired offline in advance is insufficient.

The object operation system according to the present embodiment relates to a robot that operates a new object (that is, an object that has never been operated before) as a main example. The operation is, for example, to stably transport one or more objects to a destination. Specific examples include transporting a glass containing water to a destination so that water does not spill, and transporting a glass on a tray to a destination so that the glass does not fall over. Alternatively, the operation is, for example, the stable placement of an object. As a specific example, an object having an arbitrary shape may be placed on a desk in a stable state without being tilted.

FIG. 2A is a schematic block diagram of an object manipulation system according to an embodiment. The object operation system includes a robot body 1, a sensor 2, and a control computer 3. These may be integrated to form a robot, or a control computer 3 may be provided separately from the robot main body 1 to remotely control the robot main body 1.

The robot body 1 has a function of manipulating an object according to a control from a control computer 3, and as a specific example, a function of holding an object, placing a held object, and moving the object.

The sensor 2 acquires information about the operating environment and the operating object as real-world information (hereinafter, the information acquired by the sensor 2 is referred to as “sensor acquisition information”) and transmits the information to the control computer 3. The sensor 2 is preferably attached to the robot body 1, but may be fixed to the environment in which the robot body 1 is located.

The control computer 3 has software (program) for realizing the recognition unit 42, the extraction unit 52, the learning unit 53, the simulation unit 60, and the actual operation control unit 70, and also has an object identification database 41 and an object identification database 41 used for processing. It has a learned behavior database 51. These details will be described in detail with reference to FIG.

When the robot body 1 operates the operation object, the control computer 3 receives the sensor acquisition information on the spot and performs a simulation when there is not enough information (NO in step S22 in FIG. 1). Control the main body 1. Specifically, the control computer 3 simulates the operation of the operation object by the robot body 1 in the virtual space using the operation command and the sensor acquisition information, and operates the operation object by the robot body 1 based on the simulation result. Control. Further, both the simulation result and the actual operation by the robot body 1 are learned and utilized for the subsequent simulation and the movement control of the robot body 1.

Hereinafter, a more specific description will be given.
FIG. 2B is a more detailed block diagram of the object manipulation system. As shown in the figure, the control computer 3 includes a determination unit 30, an object recognition unit 40, a physical world model learning unit 50, a simulation unit 60, and an actual operation control unit 70.

When an operation command (command indicating that the operation object is operated) is input from the outside, the determination unit 30 is an object that the operation object newly operates by referring to the object identification database 41 described later. Judge whether or not. Then, when the operation target is new, it is determined that there is not enough information (NO in step S22 of FIG. 1).

The object recognition unit 40 has an object identification database 41 and a recognition unit 42. Further, an operation command is input to the object recognition unit 40 from the outside, and sensor acquisition information is input from the sensor 2.

In the object identification database 41, object identification data learned in advance by machine learning, deep learning (for example, You only Look Once (YOLO), Single Shot MultiBox Detector (SSD)), or the like is registered.

The recognition unit 42 uses machine learning and deep learning to recognize information on the operating environment and information on the operating object required for simulation from the sensor acquisition information and the object identification database 41. Information about the operating environment and information on the operating object are transmitted to the physical world model learning unit 50 and the simulation unit 60.

The physical world model learning unit 50 has a learned behavior database 51, an extraction unit 52, and a learning unit 53.

Pre-learned behavior data is registered in the learned behavior database 51. The behavior data is data necessary for moving the robot body 1, and may include parameters such as acceleration, constant speed, and deceleration.

The extraction unit 52 extracts a plurality of behavior data registered in the learned behavior database 51 from the behavior data registered in the learned behavior database 51, based on the information on the operation environment and the operation target from the recognition unit 42, which are likely to succeed in the operation. .. The extracted behavior data is transmitted to the simulation unit 60.

The learning unit 53 learns the result of the simulation described later and registers new behavior data in the learned behavior database 51. Further, the learning unit 53 learns the result of the operation of the robot body 1 described later in the real space, and registers new action data in the learned action database 51. The newly registered behavior data can be subsequently extracted by the extraction unit 52 and used in the simulation.

The simulation unit 60 simulates the operation by the robot body 1 in response to the operation command based on the sensor acquisition information when it is determined that the operation target object is an object to be newly operated, and is a simulation environment setting unit. It has 61, a simulation parameter setting unit 62, a simulation execution unit 63, a simulation observation unit 64, and a loop determination unit 65.

The simulation environment setting unit 61 sets information about the robot main body 1 and the operation environment and the operation object in the virtual space for executing the simulation based on the information about the operation environment and the operation object from the recognition unit 42.

The simulation parameter setting unit 62 sets parameters for controlling the robot body 1 in the virtual space based on the action data from the extraction unit 52.

Based on the parameters set by the simulation parameter setting unit 62, the simulation execution unit 63 physically calculates the operation by the robot body 1 in the virtual space in which the robot body 1 and the like are arranged by the simulation environment setting unit 61, and causes the robot body 1 to perform physical calculations. Simulate the state.

The simulation observation unit 64 observes the state of the robot body 1 in the virtual space. Specifically, the simulation observation unit 64 observes whether or not the operation target can be stably operated, and if it can be operated, it is determined to be successful, and if it is not possible, it is determined to be a failure. The result of the observation is transmitted to the loop determination unit 65.

The loop determination unit 65 transmits the observed simulation result to the learning unit 53. As a result, the learning unit 53 can learn what kind of operation object can be operated in what kind of operation environment. The learning result is newly registered in the learned behavior database 51 as behavior data. Further, the loop determination unit 65 counts the number of times the simulation is executed, and repeats the execution and observation of the simulation a predetermined number of times while changing the behavior data.

When the simulation result is determined to be successful after repeating the predetermined times, the loop determination unit 65 determines that the operation according to the operation command cannot be performed, and terminates the command. When there is a simulation result determined to be successful, the loop determination unit 65 transmits the action data and the simulation result used in the simulation to the actual operation control unit 70. If two or more of the simulation results are determined to be successful, the loop determination unit 65 may select one.

Based on the simulation result, the actual operation control unit 70 causes the robot body 1 to operate the operation target object in response to the operation command. The actual operation control unit 70 includes a control parameter setting unit 71 and an actual operation observation unit 72. In addition, behavior data and simulation results are input from the simulation unit 60 to the actual operation control unit 70, and sensor acquisition information is input from the sensor 2.

The control parameter setting unit 71 sets the control parameters required for the robot body 1 to operate the operation target object based on the action data from the simulation unit 60. The robot body 1 operates the operation target according to this control parameter. The state of operation is detected by the sensor 2.

The actual operation observation unit 72 observes the state of the robot body 1 in the real space by using the sensor acquisition information. Then, the actual operation observation unit 72 compares the simulation result observed by the simulation unit 60 with the state of the robot body 1 in the real space based on the sensor acquisition information, and determines whether or not the two are significantly different. Judge according to. As a specific example, when the position of the robot body 1 at a specific time deviates by a predetermined distance or more between the simulation and the real space, the actual operation observation unit 72 determines that the two are different. In this case, the actual operation observation unit 72 stops the operation of the operation object by the robot body 1 and ends the command. Even when the robot main body 1 completes the operation without any difference between the two, the actual operation observation unit 72 ends the command.

Whether the robot body 1 is stopped in the middle of operating the operation target object or the operation is completed, the actual operation observation unit 72 sends the result of the operation in the real space to the learning unit 53. Send. As a result, the learning unit 53 can learn what kind of operation object is successfully operated in what kind of operation environment. The learning result is newly registered in the learned behavior database 51 as behavior data.

FIG. 3 is a schematic block diagram of an object manipulation system in a virtual space in which a simulation is executed. This simulation imitates the object operation system in the real space shown in FIG. 1, and includes a virtual robot main body 1V, a virtual sensor 2V, and a virtual control computer 3V. These correspond to the robot body 1, the sensor 2, and the control computer 3 of FIG. 1, respectively, but do not require a simulation function. Further, the output of the virtual sensor 2V corresponds to an observation of the state of operation by the virtual robot main body 1V in the virtual space, that is, the output of the simulation observation unit 64 in FIG.

Hereinafter, as an example of the operation, the transportation of the operation target to a predetermined destination will be described. In this case, the robot body 1 has a moving function. Specifically, the robot body 1 may be moved by two-wheel tires, or may be moved by multi-legged walking such as two legs or four legs. More specifically, the robot body 1 has a motor (not shown) for driving tires and feet, and the robot body 1 moves in response to control from the control computer 3. Further, the robot body 1 has a function of holding an object to be transported.

Further, the sensor 2 measures a camera that captures (at least a part of) the actual space from the current location of the robot body 1 to the destination, a LiDAR (laser range finder) that measures the unevenness of the floor surface, and the weight of the object to be operated. May include weight sensors. In addition, the sensor 2 may include a depth sensor, a tracking camera, an acceleration sensor, a gyro sensor, and the like.

In order to carry out transportation, the robot lifts the operation target, moves it while holding it, and lowers it at the destination. In the following, the point of moving while holding the object will be mainly described.

FIG. 4 is a flowchart showing an example of the processing operation of the control computer 3. Upon receiving an operation command indicating movement to a predetermined destination (step S1), the control computer 3 starts operating, and sensor acquisition information is input to the control computer 3. The destination shall be set within the range that can be photographed by the camera of the object operation system (if the final destination is not within the image range of the camera, the upper system that manages the movement of the robot , The movement path may be divided within the shooting range of the camera). Then, although there are no obstacles in the movement route to the destination, the floor surface is not necessarily flat and uneven (for example, due to a step at the boundary between rooms or deterioration over time).

When the operation command is received, the determination unit 30 determines whether or not the operation target is a newly operated object, that is, whether or not the information is sufficient (step S2). If the information is sufficient (YES in step S2), the process proceeds to step S4.

When there is not enough information (NO in step S2), the recognition unit 42 acquires the sensor acquisition information from the sensor 2. Then, the recognition unit 42 recognizes the information regarding the operation environment and the operation target required for the simulation from the object identification database 41 based on the sensor acquisition information (step S3). The information regarding the operating environment is the position of the unevenness of the floor surface, the material of the floor surface (friction coefficient parameter), the inclination angle, and the like. The information about the operation target is its weight, shape, center of gravity, and the like. As a result, the simulation environment setting unit 61 arranges the robot body 1, the object to be transported, and the floor surface to the destination in the virtual space where the simulation is executed.

Further, the extraction unit 52 extracts a plurality of action data having a high possibility of succeeding in moving to the destination from the learned action database 51 based on the recognized information (step S4). As a result, the simulation parameter setting unit 62 sets the parameters for moving the robot body 1 in the virtual space based on the action data from the extraction unit 52.

Then, the simulation unit 60 executes the simulation using the information about the operation environment and the operation object and the behavior data (step S5).

Specifically, the simulation execution unit 63 physically calculates the movement of the robot body 1 in the virtual space in which the robot body 1 and the like are arranged by the simulation environment setting unit 61 based on the parameters set by the simulation parameter setting unit 62. , Simulate the state of the robot body 1.

Then, the simulation observation unit 64 observes whether or not the object to be transported can be stably transported to the destination in the virtual space (whether or not the object to be transported does not fall, whether or not the robot body 1 does not fall, etc.). If it can be done, it is judged as success, and if it cannot be done, it is judged as failure. If the simulation observation unit 64 is successful, the simulation observation unit 64 observes the travel time to the destination. The result of the observation is transmitted to the loop determination unit 65.

For example, when the tray on which the glass containing water is placed is the object to be transported, the simulation observation unit 64 observes whether the glass falls on the tray or the water in the glass does not spill, and is safe. Determine if it could be transported.

In the simulation, the state of the robot body 1 is observed, learning is performed by the learning unit 53, and the learned behavior database 51 is updated (step S6). The simulation unit 60 repeats the above simulation a predetermined number of times while changing the behavior data (step S7).

When the repetition of a predetermined number of times is completed (YES in step S7), but all the simulation results are unsuccessful (NO in step S8), the simulation unit 60 determines that it cannot move to the destination in response to the operation command. And end the instruction.

When the repetition of a predetermined number of times is completed (YES in step S7) and one or more of the simulation results are successful (YES in step S8), the simulation unit 60 performs the simulation in which the travel time to the destination is the shortest. The behavior data used is selected (step S9). Then, based on this action data, the actual operation control unit 70 moves the robot body 1 in the real space (step S10). The moving state of the robot body 1 is acquired by the sensor 2.

From the start of movement of the robot body 1 to the arrival at the destination, the actual operation control unit 70 determines whether or not the state of the robot body 1 in the real space obtained from the sensor acquisition information differs from the simulation result. Judgment is made using the judgment criteria (step S11). This determination may be made at predetermined time intervals or every time the robot body 1 moves a predetermined distance.

If it is determined that they are different (YES in step S11), the actual operation control unit 70 stops the movement of the robot body 1 (step S12). If there is no difference (NO in step S11), the robot body 1 is moved until the robot body 1 reaches the destination (step S13).

Then, the learning unit 53 learns the result of the movement of the robot body 1 in the real space and updates the learned action database 51 (step S14). With the above, the correspondence to the operation command is completed.

As described above, in the present embodiment, when operating a new object, a simulation is performed using the information of the operating environment and the operating object acquired by the sensor 2 at that time, and the action is selected. Therefore, it is possible to operate a new object that has never been operated before.

As an example of the operation, when transporting the operation object, the operation environment to the destination acquired by the sensor 2 and the information of the operation object and the learned behavior data are used to perform simulation multiple times. Move the robot body 1. Therefore, even if the operating environment or the operating object changes, the robot body 1 holding the operating object can be safely moved to the destination. Further, since the simulation result and the result of the actual movement are learned and the behavior data is updated, the subsequent movement of the robot body 1 can be performed more safely.

In addition, by performing the simulation multiple times, in addition to obtaining the optimum behavior data for moving the robot body 1 in the real space, it is possible to learn the behavior data suitable for the operation target and the operation environment this time. can.

The above-described embodiment is described for the purpose of enabling a person having ordinary knowledge in the technical field to which the present invention belongs to carry out the present invention. Various modifications of the above embodiment can be naturally made by those skilled in the art, and the technical idea of the present invention can be applied to other embodiments. Therefore, the present invention is not limited to the described embodiments and should be the broadest scope according to the technical ideas defined by the claims.

1 Robot body 2 Sensor 3 Control computer 30 Judgment unit 40 Object recognition unit 41 Object identification database 42 Recognition unit 50 Physical world model Learning unit 51 Learned behavior database 52 Extraction unit 53 Learning unit 60 Simulation unit 61 Simulation environment setting unit 62 Simulation Parameter setting unit 63 Simulation execution unit 64 Simulation observation unit 65 Loop judgment unit 70 Actual operation control unit 71 Control parameter setting unit 72 Actual operation observation unit 1V Robot body 2V Sensor 3V Control computer 70V Virtual motion control unit

Claims (13)

Computer,
When the computer receives an operation command indicating that the operation object is operated, a determination unit that determines whether or not the operation object is a newly operated object, and a determination unit.
When it is determined that the operation object is a newly operated object, the operation environment based on the information acquired by the sensor, the information about the operation object, and the behavior data learned in advance are used. A simulation unit that simulates the operation of the operation object by the robot body in response to the operation command,
A robot control program that functions as an actual operation control unit that controls the robot body in a real space so as to operate the operation object in response to the operation command based on the result of the simulation. The robot control program according to claim 1, wherein the computer functions as a learning unit that learns the result of the simulation by the simulation unit and registers it as new behavior data. The robot control program according to claim 2, wherein the learning unit learns the result of an operation of the robot body in the real space and registers it as new action data. The first aspect of the present invention, wherein the actual operation control unit controls the robot body so as to operate the operation target object in response to the instruction based on the result of performing the simulation a plurality of times while changing the action data. Robot control program. The actual operation control unit determines whether or not the result of the simulation and the state of the robot body controlled by the actual operation control unit are different according to a predetermined standard, and when it is determined that they are different. Is the robot control program according to claim 1, wherein the operation of the operation target object by the robot main body is stopped. The robot control program according to any one of claims 1 to 5, wherein the operation of the operation object is to stably place the operation object. The robot control program according to any one of claims 1 to 5, wherein the operation of the operation object is to carry the operation object to a destination indicated by the operation command. When there are a plurality of successful results of moving to the destination as a result of the simulation, the actual operation control unit is based on the action data used in the simulation in which the moving time to the destination is the shortest. The robot control program according to claim 7, wherein the robot body is moved. The robot control program according to claim 7 or 8, wherein if the result of the simulation is that the movement to the destination is not successful, the simulation unit determines that the movement cannot be performed in response to the operation command. The actual operation control unit determines whether or not the movement of the robot body differs from the result of the simulation during the movement of the robot body according to a predetermined criterion, and if it is determined that the movement differs from the result of the simulation. The robot control program according to any one of claims 7 to 9, which stops the movement of the robot. The information regarding the operating environment includes information on the floor surface to the destination.
The robot control program according to any one of claims 7 to 10, wherein the information about the operation object includes the weight of the operation object. When the determination unit receives an operation command indicating that the operation object is to be operated, the determination unit determines whether or not the operation object is a newly operated object.
When the simulation unit determines that the operation object is a newly operated object, the operation environment based on the information acquired by the sensor, information on the operation object, pre-learned behavior data, and Is used to simulate the operation of the operation object by the robot body in response to the operation command.
A robot control method in which the actual operation control unit controls the robot body in a real space so as to operate the operation object in response to the operation command based on the result of the simulation. With the sensor
With the robot body
When an operation command indicating that the operation object is operated is received, a determination unit that determines whether or not the operation object is a newly operated object, and a determination unit.
When it is determined that the operation object is a newly operated object, the information about the operation environment and the operation object based on the information acquired by the sensor and the behavior data learned in advance are used. A simulation unit that simulates the operation of the operation object by the robot body in response to the operation command, and
An object operation system including an actual operation control unit that moves the robot body in a real space so as to operate the operation object in response to the operation command based on the result of the simulation.

Images