WO2021176902A1 - Learning processing device, device and method for robot control, and program - Google Patents

Abstract

Provided are a device and a method for executing robot control processing and learning processing using phase-only correlation information. This invention involves a phase-only correlation information calculation unit which generates phase-only correlation information between a camera-captured image and a model image. A learned model is generated, which receives input of the phase-only correlation information and outputs a control parameter used to control the position of an object captured in the camera-captured image. This invention provides a robot control device which uses the generated learned model to execute robot control. The phase-only correlation information between the camera-captured image of an object taken by the robot and the model image in which the object is arranged at an ideal position is input to the learned model, and a control parameter target value obtained as output from the learned model is used to control the robot.

Description

Learning processing equipment, robot control equipment, and methods, and programs

The present disclosure relates to learning processing devices, robot control devices, methods, and programs. Specifically, the present invention relates to a learning process for mounting a component in a predetermined position using a robot, a learning process device for performing a robot control process, a robot control device, a method, and a program.

In recent years, the number of manufacturing factories that assemble various products using robots has increased rapidly, and in factories, robots are now performing most of the work that was conventionally performed by humans.

There are various processes (tasks) performed by robots in factories, and one example is the following process.
The robot acquires the object a at the P1 point and moves the acquired object to the P2 point. Further, the acquired object a is attached to another object b located at the P2 point, and the like is executed.
Such processing (task) is an important task in, for example, work automation using an arm robot in a factory.

In order for the robot to accurately execute the above processing, it is necessary to control the movement of the robot with high accuracy. Learning processing is used as one method for constructing this control algorithm.
For example, by performing learning processing that analyzes images taken by a camera mounted on a robot or fixed at a specific position in a work place, for example, an algorithm for realizing automation of the above processing (task) can be used. Can be built.

However, the learning process requires inputting a large amount of sample data, for example, captured images with various different settings, and analyzing a large number of images, which requires a predetermined time and labor.

For example, there are many patterns of deviation from the assumed position / orientation of the object a to be acquired at the time of acquisition, and there are also many patterns of deviation of the position and orientation of the object b to be attached to the object a. In such a case, in order to generate control information corresponding to various deviation patterns, a very large number of sample data (camera-captured images) are required, and a considerable amount of time is required for the learning process.

Note that there is deep learning, which is a learning process using a deep neural network (DNN), as a learning process for causing a robot to execute an object position / orientation control task. If deep learning is used, a large amount of sample data is input, and the learning processing unit itself automatically extracts the feature amount from the large amount of data to generate an optimum solution corresponding to various data, for example, a robot control parameter. be able to.

However, in order to use this deep learning, it is necessary to prepare a large amount of sample data in advance. For example, in the above task example, a large number of pattern images of deviation from the assumed position / orientation at the time of acquisition of the object a to be acquired, pattern images of deviation of the position / posture of the object b to be attached to the object a, and the like are obtained. , It is necessary to take a picture in advance.

At a factory site for high-mix low-volume production where the specifications of the products to be produced are changed one after another, it is necessary to change the robot control algorithm every time the manufactured products are switched. In order to perform learning processing such as learning processing using the above deep learning for each product switching, it takes an enormous amount of time to take sample image data showing a large number of different deviation modes, which is realistic. Instead, as a result, there are situations in which robot control has to be given up and relied on by human hands.

On the other hand, Patent Document 1 (Japanese Unexamined Patent Publication No. 2015-102928) discloses a configuration using a phase-limited correlation method as a method for detecting the position and orientation of a robot.
This method is a method of detecting the position and orientation of an object in the input image by comparing the input image obtained by capturing the object and the template image by using the phase-limited correlation method.

By applying the method using this phase-limited correlation method, the position and orientation of the object in the input image can be detected with high accuracy.
However, this document does not disclose the learning processing method using this phase-limited correlation method or the robot control processing using the learning result.

Japanese Unexamined Patent Publication No. 2015-102928

The present disclosure has been made in view of the above problems, for example, and is a learning process for mounting a component in a predetermined position using a robot, a learning process device for performing a robot control process, a robot control device, and a method. , As well as the purpose of providing the program.

The first aspect of the disclosure is
An image correlation information calculation unit that generates image correlation information between a camera-photographed image and a model image,
The learning processing apparatus has a learning processing unit that executes a learning process of inputting the image correlation information and generating a learning model that outputs control parameters for controlling the position of an object captured by the camera image.

Further, the second aspect of the present disclosure is
A robot control unit that controls the robot and
An image correlation information calculation unit that generates image correlation information between a camera-photographed image of an object acquired by the robot and a model image in which the object is arranged at an ideal position.
It has a learning model that inputs the image correlation information and outputs a control parameter target value for controlling the position of the object captured in the image captured by the camera.
The robot control unit is in a robot control device that controls the robot by using a control parameter target value which is an output of the learning model.

Further, the third aspect of the present disclosure is
It is a learning processing method executed in the learning processing device.
The image correlation information calculation unit generates the image correlation information of the camera-captured image and the model image, and the image correlation information calculation step.
The learning processing unit has a step of executing a learning process of inputting the image correlation information and generating a learning model for outputting control parameters for controlling the position of an object captured by the camera image. ..

Further, the fourth aspect of the present disclosure is
It is a robot control method executed in a robot control device.
The steps in which the robot control unit controls the robot,
An image correlation information calculation step in which the image correlation information calculation unit generates image correlation information between the camera-photographed image of the object acquired by the robot and the model image in which the object is arranged at an ideal position.
A step of inputting the image correlation information to the learning model and outputting a control parameter target value for controlling the position of the object captured in the image captured by the camera.
The robot control unit is in a robot control method for recontrolling the robot by using a control parameter target value which is an output of the learning model.

Further, the fifth aspect of the present disclosure is
It is a program that executes learning processing in the learning processing device.
An image correlation information calculation step that causes the image correlation information calculation unit to generate image correlation information of a camera-photographed image and a model image, and
The program has a program for executing a step of inputting the image correlation information into the learning processing unit and executing a learning process of generating a learning model that outputs control parameters for controlling the position of an object captured by the camera captured image.

Further, the sixth aspect of the present disclosure is
A program that executes robot control processing in a robot control device.
Steps to let the robot control unit control the robot,
An image correlation information calculation step for causing the image correlation information calculation unit to generate image correlation information between the camera-photographed image of the object acquired by the robot and the model image in which the object is arranged at an ideal position.
A step of inputting the image correlation information to the learning model and outputting a control parameter target value for controlling the position of the captured object in the camera-captured image.
The program is for the robot control unit to recontrol the robot by using the control parameter target value which is the output of the learning model.

The program of the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium that is provided in a computer-readable format to an information processing device or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing according to the program can be realized on an information processing device or a computer system.

Still other objectives, features and advantages of the present disclosure will be clarified by more detailed description based on the examples of the present disclosure and the accompanying drawings described below. In the present specification, the system is a logical set configuration of a plurality of devices, and the devices having each configuration are not limited to those in the same housing.

According to the configuration of one embodiment of the present disclosure, a device and a method for executing learning processing and robot control processing using phase-limited correlation information are realized.
Specifically, for example, the phase-limited correlation information calculation unit that generates the phase-limited correlation information of the camera-captured image and the model image, and the phase-limited correlation information are input to control the position of the object captured in the camera-captured image. Generate a learning model that outputs the control parameters of. Furthermore, it is a robot control device that controls the robot using the generated learning model, and provides phase-limited correlation information between the camera-captured image of the object acquired by the robot and the model image in which the object is placed at an ideal position. The robot is controlled by inputting to the learning model and using the control parameter target value obtained as the output from the learning model.
With this configuration, a device and a method for executing learning processing and robot control processing using phase-limited correlation information are realized.
The effects described in the present specification are merely exemplary and not limited, and may have additional effects.

It is a figure explaining an example of a robot operation and control processing. It is a figure explaining the example of the state at the time of parts acquisition by a robot. It is a figure explaining the example of the state at the time of mounting a part by a robot. It is a figure explaining an example of the learning process for controlling a robot. It is a figure explaining an input data and a learning process for a learning processing part. It is a figure explaining an example of robot control using a learning model. It is a figure explaining the phase-limited correlation information (PoC information). It is a figure explaining the phase-limited correlation information (PoC information). It is a figure explaining an example of the structure and processing of the learning processing apparatus of this disclosure. It is a figure explaining an example of the structure and processing of the phase-limited correlation information (PoC information) calculation part of the learning processing apparatus of this disclosure. It is a figure explaining an example of the structure and processing of the learning processing part of the learning processing apparatus of this disclosure. It is a figure explaining an example of the structure and processing of the robot control device of this disclosure. It is a figure explaining the example of the input data in a learning process and the input / output data of a learning model. It is a figure explaining an example of the structure and processing of the learning processing part of the learning processing apparatus of this disclosure. It is a figure explaining an example of the structure and processing of the learning processing part of the learning processing apparatus of this disclosure. It is a figure explaining the example of the image for generating the phase-limited correlation information (PoC information) to apply to a learning process. It is a figure explaining an example of the structure and processing of the learning processing part of the learning processing apparatus of this disclosure. It is a figure explaining an example of the structure and processing of the learning processing part of the learning processing apparatus of this disclosure. It is a figure explaining an example of the structure and processing of the robot control device of this disclosure. It is a figure explaining the hardware configuration example of the learning processing apparatus and the robot control apparatus of this disclosure.

Hereinafter, the traveling robot of the present disclosure, the traveling robot control method, and the details of the program will be described with reference to the drawings. The explanation will be given according to the following items.
1. 1. About an example of robot control processing 2. About an example of general learning processing for building a robot control algorithm 3. Phase-limited correlation information between images 4. 5. Regarding the learning processing device and the robot control device of the present disclosure that perform processing using phase-limited correlation information. About the example using the photographed image at the part acquisition position 6. About the update process of the learning model 7. About hardware configuration example of learning processing device and robot control device 8. Summary of the structure of this disclosure

[1. About an example of robot control processing]
First, an example of robot control processing will be described with reference to FIG. 1 and below.
FIG. 1 is a diagram showing an example of processing executed by the robot 100.

Ring-shaped (hollow disk-shaped) parts 20 are piled up in bulk in the parts box 10. Further, the component mounting pin 40 is moved on the belt conveyor 30. The belt conveyor 30 is moving in the direction of the arrow shown in the figure.
The robot 100 acquires a component 20 from the component box 10 and executes a process (task) of inserting the component 20 into the component mounting pin 40 on the belt conveyor 30.
A suction unit 101 is provided at the tip of the arm of the robot 100, and the component 20 can be sucked and released by controlling the suction operation of the suction unit 101.

The robot 100 executes the following processes in the order of steps S1 to S03 shown in the figure.
(Step S01)
First, in step S01, the robot 100 sucks one part 20 from the parts box 10 in which the ring-shaped (hollow disk-shaped) parts 20 are piled up in bulk to the suction portion 101 provided at the tip of the arm of the robot 100. And get it.

(Step S02)
The robot 100 that has attracted one component 20 then rotates the arm (robot arm) of the robot 100 in step S02 to a position directly above one pin of the component mounting pin 40 that moves on the belt conveyor 30. Move the part 20.

(Step S03)
Next, in step S03, the robot 100 releases the suction of the component 20 sucked by the suction portion 101 at the tip of the arm of the robot 100, and mounts the component 20 on the pin of the component mounting pin 40.

After the process of step S03, the robot 100 rotates the arm (robot arm) of the robot 100 and performs the process of step S01, that is, the process of taking out the component 20 from the component box 10.
The robot 100 repeatedly executes the above steps S01 to S03.

In the processes of steps S01 to S03, the operations required of the robot 100 are a component picking operation for acquiring one component 20 from parts in a bulk state and a component for moving the acquired components on the belt conveyor 30. The operation of moving to the position directly above one pin of the mounting pin 40 and the center hole of the ring-shaped (bulk-shaped disk shape) component 20 are aligned with the pin position, and the suction of the component 20 is released at that position. , The operation is to mount the component 20 on the pin of the component mounting pin 40.

However, these operations are required to perform slightly different operations for each mounting of one component.
For example, in the part picking process for acquiring one part 20 from the parts in the bulk state, as shown in FIG. 2, the relative positions of the suction portion 101 of the robot 100 and the part 20 are various. That is, various different "deviations" occur for each component picking (adsorption) process.

Further, the component mounting pin 40 that moves on the belt conveyor 30 is moved to a position directly above one pin, and the central hole of the ring-shaped (hollow disk-shaped) component 20 is aligned with the pin position, and the position is adjusted. When the component 20 is released from the suction of the component 20 and the component 20 is mounted on the pin of the component mounting pin 40, as shown in FIG. 3, the position of the center hole of the component 20 and the relative position of the component mounting pin 40 Is various. That is, various different "deviations" occur for each component mounting process.

Therefore, in order to accurately execute the processes of steps S01 to S03 for each component, robot control assuming all various states as shown in FIGS. 2 and 3 is required.
The learning process is effective as a process for performing different robot control according to each state.
Hereinafter, an example of learning processing for robot control will be described.

[2. About an example of general learning processing for building a robot control algorithm]
Next, an example of a general learning process for constructing a robot control algorithm will be described.

In order to execute the robot component mounting process described with reference to FIG. 1 with high accuracy without mistakes, it is necessary to perform robot control according to the relative positions of the robot 100, the component 20, and the component mounting pin 40. It becomes.

That is, it is necessary to perform robot control peculiar to each state according to various component mounting patterns as shown in FIG. 2 and various component insertion patterns as shown in FIG. It is necessary to calculate the unique robot control parameters according to the state.
As one method for this, there is a learning process using images taken by a camera.

An example of the learning process using the image captured by the camera will be described with reference to FIG.
As shown in FIG. 4, the camera 120 captures an image of the robot 100 moving the component 20 onto the component mounting pin 40.
At this position, the robot 100 releases the suction of the component 20 and mounts the component 20 on the pin of the component mounting pin 40.

As shown in FIG. 4, the camera 120 captures the camera captured image 121. The camera-captured image 121 captures an image of a part of the component 20 and a part of the component mounting pin 40 observed from the hole in the center of the component 20.
As shown in the upper right (reference diagram) of FIG. 4, the image of a part of the part 20 included in the camera-captured image 121 is only a part of the hole in the center of the part and a part of the ring around it, and the part. The outer peripheral portion of the ring of 20 exists around the camera-captured image 121 shown in FIG.

The captured image 121 captured by the camera 120 is input to the learning processing unit 130.
Further, the robot control parameter 126 generated by the robot control unit 125 for controlling the robot 100 is input to the learning processing unit 130. The robot control parameter 126 is, for example, a parameter such as a control position and an angle of each arm of the robot 100, or a parameter set in a control command for setting these control positions and angles.
The learning processing unit 130 executes machine learning processing using these input data.

The learning processing unit 130 acquires the feature information of the captured image 121 captured by the camera 120. For example, image feature information such as pixel value distribution and edge information is acquired, learning processing is performed on these feature information, and a learning model 131 associated with the optimum control parameters of the robot corresponding to the image feature information is generated.

The learning model 131 generated by the learning processing unit 130 is
The input is an image, i.e., an image with the component 20 moved onto the component mounting pin 40.
The output is the robot control parameter target value, that is, the robot control parameter target value corresponding to various images.
Is.

The robot control parameter target value, which is the output of the learning model 131, is also a parameter such as a control position or angle of each arm of the robot 100, or a parameter set in a control command for setting these control positions or angles. Consists of.

As shown in FIG. 5, the learning processing unit 130 displays a number of different state images, that is, various different (A) learning images in a state where the robot 100 moves the component 20 onto the component mounting pin 40. It is input as learning data, feature information of these images is acquired, and a learning model 131 in which robot control parameter target values corresponding to each image are associated with each other is generated.

The learning process executed by the learning processing unit 130 is so-called machine learning, and various learning algorithms can be applied. Deep learning, which is a learning process using a deep neural network (DNN), is an example of a learning process for causing a robot to execute an object position / orientation control task. If deep learning is used, a large amount of sample data is input, and the learning processing unit itself automatically extracts the feature amount from the large amount of data to generate an optimum solution corresponding to various data, for example, a robot control parameter. be able to.

As a learning process using a deep neural network (DNN), for example, a learning process in which a convolutional neural network CNN (Convolutional Neural Network) is combined with a BN (Batch Rectification) or a ReLU (Rectified Linear Unit) is known. ing. The learning processing unit 130 shown in FIG. 5 can be configured as, for example, a learning processing unit using a deep neural network (DNN) having such a configuration.
The learning processing unit 130 can be realized as a learning processing unit to which various learning algorithms are applied.

A specific example of the learning process executed by the learning processing unit 130 will be described with reference to FIG. As shown in FIG. 5, the learning processing unit 130 has a number of images of different states in which the component 20 has been successfully inserted into the component mounting pin 40, that is, the “(A) learning image” shown in FIG. 5 and these images. The robot control parameters at the time of shooting, that is, the data corresponding to the “(B) robot control parameters at the time of successful component insertion” shown in FIG. 5 are input to the learning processing unit 130.

“(B) Robot control parameter when component insertion is successful” is composed of, for example, the position (x, y, z) of the robot arm and the roll, pitch, and yaw (roll, pitch, yaw) indicating the arm posture. Dimensional real-valued vectors (actions) are available.

In this way, as a learning sample, a large number of pairs of the camera image immediately before the successful insertion and the executed action are prepared, and these are input to the learning processing unit 130.
The learning processing unit 130 sets these input data as components of the neural network by using a learning process using a neural network in which the above-mentioned CNN is combined with BN and ReLU, for example, a stochastic gradient descent method. The learning process for determining the weight information to be performed is executed.

From this learning process, the learning processing unit 130 inputs an image showing the situation immediately before the component insertion and outputs an optimum robot control parameter for inserting the component 20 into the component mounting pin 40, that is, a robot control parameter target value. Generate a learning model 131 to be used.

By using the generated learning model 131, it is possible to control the actual robot 100.
FIG. 6 is a diagram showing a control example of the robot 100 using the learning model 131.

The camera-captured image 121 captured by the camera 120 provided at the same position as when the learning model was generated is input to the learning model 131.
The learning model 131 acquires feature information from the captured image 121, and outputs the optimum control parameter of the robot corresponding to the acquired feature information, that is, the robot control parameter target value 132 shown in FIG.

As shown in FIG. 6, the robot control parameter target value 132 is input to the robot control unit 125, and the robot control unit 125 calculates the difference between the current robot control parameter 126 and the robot control parameter target value 132, and the difference. If there is, the robot 100 is corrected and controlled so that the difference amount is zero or reduced.

By controlling the robot 100 using the robot control parameter target value 132 in this way, the position and orientation of the component 20 attached to the tip of the arm of the robot 100 are corrected and controlled, and then the component 20 is attracted. By releasing the component 20, the component 20 can be securely inserted into the component mounting pin 40.

However, in the configuration described with reference to FIGS. 5 and 6, a large number of images showing different states are input as learning data for the learning processing unit 130, and the learning processing unit 130 inputs a large number of images. It is necessary to execute the process of acquiring various feature information required to generate the robot control parameter target value from the image.

As described above with reference to FIGS. 2 and 3, the acquisition position (suction position) of the component 20 by the robot 100 is various, and the component 20 is attached to the component mounting pin 40. There are various states relative to the component mounting pin 40, and there are many variations indicating these situations.

In a situation where there are many variations like this, if the above learning process is to be performed, it is necessary to prepare tens of thousands of orders of learning sample data in order to cover the variations as widely as possible, and the data collection process. It takes a lot of time to build a learning model with robustness.

In particular, it takes a lot of time to construct an image analysis algorithm for analyzing the feature information of the image required to calculate an appropriate control parameter in the learning processing unit. The feature information that can be acquired from the image is enormous, and in order to construct a learning model with robustness, is it the feature information necessary to generate robot control parameters for each of the enormous image feature information? It is necessary to examine in detail how important the necessary feature information is.

If the accuracy of this image analysis algorithm is low, parameter calculation based on extra image feature information may be performed. Therefore, a more accurate image analysis algorithm is indispensable for constructing a learning model with robustness.
However, in order to construct a highly accurate image analysis algorithm, image analysis is required for a huge number of sample image data, and the processing cost and processing time for that purpose become enormous.

Specifically, for example, it may take several months to order. As a result, for example, at a factory site for high-mix low-volume production, there is a situation in which the construction of a learning model has to be given up and manual adjustment has to be made.

[3. About phase-limited correlation information between images]
The learning processing device and the robot control device of the present disclosure solve the above-mentioned problems, and for this solution, phase-limited correlation information, rotation-invariant phase-limited correlation information, and image phase spectrum are used. Image correlation information that can calculate image deviation information such as the existing image correlation information in pixel (bixel) units is used.

Phase-only Correlation information is also referred to as PoC information.
The phase-limited correlation information (PoC information) will be described with reference to FIG. 7.
The phase-limited correlation information (PoC information) is information having a translational movement amount (shift amount) of each object in two images.

For example, FIG. 7 shows two images, i.e.
(1a) Image a
(1b) Image b
These two images are shown.
These two images are images of the same subject, but their relative positions are deviated.
The amount of deviation (shift amount) between images is 12 pixels in the x direction and 13 pixels in the y direction.

By calculating the phase-limited correlation information (PoC information), for example, the amount of deviation between two images having such a positional deviation can be easily analyzed.

The calculation sequence of the phase-limited correlation information (PoC information) will be described.
The processing is performed in the order of steps S11 to S12 shown in FIG.
Each processing step will be described in sequence.

(Steps S11a, 11b)
First, a discrete Fourier transform (DFT) is performed on each of the images a and b.
By this discrete Fourier transform (DFT), the amplitude information A and the phase information E ^jθ corresponding to each of the images a and b are output.

(Step S12)
Next, the correlation between the phase information corresponding to each of the images a and b is calculated, and only the phase information is returned to the image space by IDFT (discrete Fourier transform).
The phase information returned to the image space is the phase-limited correlation information (PoC) 140 shown in FIG. 7.

One peak is detected from the phase-limited correlation information (PoC) 140 shown in FIG. 7.
This peak position (x, y) = (12, 13) indicates the amount of deviation (shift amount) between images. That is, the position coordinates (x, y) = (12,13) of the peak shown in the phase-limited correlation information (PoC) 140 is the amount of deviation (shift amount) between the image a and the image b (12 pixels in the x direction, y). The coordinates match 13 pixels in the direction).

By generating phase-limited correlation information (PoC) from the two images in this way, it is possible to easily analyze the amount of deviation between the two images.
In the example shown in FIG. 7, since the image a and the image b are in a relationship of shifting the entire image, only one peak is detected in the phase-limited correlation information (PoC) generated from the two images. ing.

For example, when there are a plurality of objects (subjects) having different deviation amounts in the image a and the image b, a plurality of peaks corresponding to the deviation amount of each subject are generated in the phase-limited correlation information (PoC). NS.

For example, as shown in FIG. 8, it is assumed that the image a and the image b are images taken at different timings, and a vehicle moving in the subject is photographed.
The amount of deviation (shift amount) between the image a and the entire image b is 12 pixels in the x direction and 13 pixels in the y direction, as in the example described with reference to FIG. 7.
However, the amount of deviation of the moving vehicle is different from the amount of deviation of the entire image.

In this case, in the phase-limited correlation information (PoC) generated from the images a and b, two peaks appear as shown in the phase-limited correlation information (PoC) 140 shown in FIG.
The coordinate position of one peak is the coordinates (x, y) = (12, 13) corresponding to the amount of deviation of the entire image of the images a and b. The coordinate position of the other peak is the coordinates (x, y) = (180, 13) corresponding to the amount of deviation of the vehicle in the images a and b.
In this way, when there are a plurality of objects (subjects) having different deviation amounts in the image a and the image b, the phase-limited correlation information (PoC) contains a plurality of peaks according to the deviation amount of each subject. Will be generated.

The learning processing device and the robot control device of the present disclosure execute learning processing and robot control processing using this phase-limited correlation information (PoC).

[4. About the learning processing device and the robot control device of the present disclosure that perform processing using phase-limited correlation information]
Next, the learning processing device of the present disclosure and the robot control device that perform processing using the phase-limited correlation information will be described.
The learning processing device and the robot control device of the present disclosure are not limited to phase-limited correlation information, but pixel (Vixel) image deviation information such as rotation-invariant phase-limited correlation information and image correlation information using an image phase spectrum. ) It is possible to perform processing using image correlation information that can be calculated in units. In the following, an example using phase-limited correlation information will be described as an example.

The learning processing unit 130 described above with reference to FIGS. 4 and 5 inputs a learning image, that is, a number of images in different states in which the component 20 is successfully inserted into the component mounting pin 40, and the input images. The image was analyzed and the feature information required to generate the robot control parameters was extracted.

On the other hand, the learning processing unit of the learning processing device of the present disclosure described below does not input an image of the component 20 or the component mounting pin 40.

Phase-limited correlation information that can be generated based on an image is input to the learning processing unit of the learning processing apparatus of the present disclosure.
Specifically, phase-limited correlation information is input between various images at the insertion position of the component 20 with respect to the component mounting pin 40 and a model image having a position and orientation that serves as a model for successful insertion of the component 20 into the component mounting pin 40. do.

A configuration and processing of an embodiment of the learning processing apparatus of the present disclosure will be described with reference to FIG.
FIG. 9 is a diagram illustrating the configuration and processing of the learning processing device 160 according to the first embodiment of the present disclosure.

As shown in FIG. 9, the learning processing device 160 has a phase-limited correlation information (PoC) calculation unit 161 and a learning processing unit 162.

A camera-captured image 121 captured by the same camera 120 as described above with reference to FIG. 4 is input to the phase-limited correlation information (PoC) calculation unit 161. The camera 120 captures an image of the robot 100 moving the component 20 onto the component mounting pin 40.
The camera-captured image 121 is an image in various states. That is, it is an image of various different states in which the relative positions of the component 20 and the component mounting pin 40 described above with reference to FIG. 3 are different.

Further, a model image 151 that has been captured in advance and stored in the storage unit 150 is input to the phase-limited correlation information (PoC) calculation unit 161.
The model image 151 is an image in which the relative positions of the component 20 and the component mounting pin 40 are set as model positions. That is, if the positions of the component 20 and the component mounting pin 40 are set at the same positions as the model image 151, the component 20 can be reliably inserted into the component mounting pin 40.

As described above, the phase-limited correlation information (PoC) calculation unit 161 is provided with the phase-limited correlation information (PoC) calculation unit 161.
Camera shot image 121,
Model image 151,
These pairs of images are input.

The phase-limited correlation information (PoC) calculation unit 161 calculates these two images, that is, the camera-captured image 121, the model image 151, and the phase-limited correlation information (PoC) corresponding to these two images.
FIG. 9 shows an example of the phase-limited correlation information (PoC) 171 generated by the phase-limited correlation information (PoC) calculation unit 161.

As explained earlier with reference to FIGS. 7 and 8, the phase-limited correlation information (PoC) generates peaks according to the shift of the object (subject) unit included in the two images. The coordinates of the peak position appearing in the phase-limited correlation information (PoC) are the coordinates indicating the amount and direction of the shift in the object (subject) unit.

In the example shown in FIG. 9, the camera-captured image 121 and the model image 151 include two objects, that is, a component 20 and a component mounting pin 40, and these two objects as subjects, and phase-limited correlation information (PoC). ) In the phase-limited correlation information (PoC) 171 generated by the calculation unit 161, a peak appears at a coordinate position corresponding to the amount of deviation between the images of these two objects. One of the two coordinates (x, y) corresponding to these two peaks corresponds to the deviation of the component 20, and the other corresponds to the deviation of the component mounting pin 40.

When the positions of the component 20 and the component mounting pin 40 in the model image 151 are set as reference positions (reference component position and reference pin position), the two peak coordinate positions of the phase-limited correlation information (PoC) 171 are images taken by the camera. It is data (positional deviation vector) indicating the amount and direction of the positional deviation from the reference position of the positions of the component 20 and the component mounting pin 40 in 121.

The phase-limited correlation information (PoC) 171 generated by the phase-limited correlation information (PoC) calculation unit 161 is input to the learning processing unit 162.
The learning processing unit 162 inputs the phase-limited correlation information (PoC) 171 generated by the phase-limited correlation information (PoC) calculation unit 161.
Further, the robot control parameter 126 generated by the robot control unit 125 for controlling the robot 100 is input to the learning processing unit 162. The robot control parameter 126 is, for example, a parameter such as a control position and an angle of each arm of the robot 100, or a parameter set in a control command for setting these control positions and angles.
The learning processing unit 162 executes machine learning processing using these input data.

The learning processing unit 162 performs learning processing by applying the phase-limited correlation information (PoC) 171 and the robot control parameter 126, and the optimum control parameter of the robot corresponding to various phase-limited correlation information (PoC) 171, that is, robot control. A learning model 172 associated with a parameter target value is generated.

The learning model 172 generated by the learning processing unit 162
The input is phase-limited correlation information (PoC), that is, the phase-limited correlation information of the two images generated based on the image in which the component 20 is moved onto the component mounting pin 40 and the model image. PoC),
The output is a robot control parameter target value, that is, a robot control parameter target value corresponding to various phase-limited correlation information (PoC).
Is.

The robot control parameter target value, which is the output of the learning model 172, is also a parameter such as a control position or angle of each arm of the robot 100, or a parameter set in a control command for setting these control positions or angles. Consists of.

Details of the processing executed by the phase-limited correlation information (PoC) calculation unit 161 and the processing executed by the learning processing unit 162 will be described with reference to FIGS. 10 and 11.

FIG. 10 is a diagram illustrating details of the process executed by the phase-limited correlation information (PoC) calculation unit 161.

The phase-limited correlation information (PoC) calculation unit 161 sequentially inputs a large number of images taken by the camera 120 while the robot 100 moves the component 20 onto the component mounting pin 40.
The robot 100 is repeatedly executed to generate the images captured by the camera (A) shown in FIG. 10, and these are sequentially input to the phase-limited correlation information (PoC) calculation unit 161.
Each of the images taken by the camera (A) shown in FIG. 10 is various images in which the relative positions of the component 20 and the component mounting pin 40 are different.

The phase-limited correlation information (PoC) calculation unit 161 sequentially executes the calculation processing of the phase-limited correlation information (PoC) with the model image 151 acquired from the storage unit 150 for each of the input camera-captured images. (A) The phase-limited correlation information (PoC) for learning corresponding to each of the images captured by the camera shown in FIG. 10 is generated.
(B) Phase-limited correlation information for learning (PoC) shown in FIG. 10.

Each of the (B) learning phase-limited correlation information (PoC) shown in FIG. 10 is a reference position (model) of the position of each of the component 20 and the component mounting pin 40 included in each of the (A) camera captured images shown in FIG. It is the PoC data in which the peak is generated at the coordinate position corresponding to the deviation position from the component 20 of the image 151 and the position of each of the component mounting pins 40).

The (B) learning phase-limited correlation information (PoC) shown in FIG. 10 is input to the learning processing unit 162.
Next, the details of the processing executed by the learning processing unit 162 will be described with reference to FIG.

As shown in FIG. 11, the learning processing unit 162
(B) Phase-limited correlation information for learning (PoC),
(C) Robot control parameters when insertion is successful,
Enter many sets of these data.

"(B) Phase-limited correlation information for learning (PoC)" is based on an image immediately before the component is inserted (camera image 121) when the component 20 is successfully inserted into the component mounting pin 40 and a model image 151. , Phase-limited correlation information (PoC) This is the phase-limited correlation information (PoC) calculated by the calculation unit 161.
"(C) Robot control parameter when insertion is successful" is robot control at the shooting timing of the image (camera shot image 121) applied to the calculation of the associated "(B) phase-limited correlation information for learning (PoC)". It is a parameter.

The "(C) robot control parameter when the insertion is successful" is composed of, for example, the position (x, y, z) of the robot arm and the roll, pitch, and yaw (roll, pitch, yaw) indicating the arm posture. Dimensional real-valued vector (action), etc.

In this way, as a learning sample, (B) a large number of pairs of phase-limited correlation information (PoC) for learning and (C) robot control parameters at the time of successful insertion are prepared, and these are input to the learning processing unit 162.
The learning processing unit 162 executes learning processing on these input data and generates a learning model 172.

The learning model 172 generated by the learning processing unit 162
The input is phase-limited correlation information (PoC).
The output is the robot control parameter target value, that is, the robot control parameter target value, which is the optimum robot control parameter for inserting the component 20 into the component mounting pin 40.
This is a learning model with the above settings.

The learning process executed by the learning processing unit 162 is so-called machine learning, and various learning algorithms can be applied. For example, the learning process using the deep neural network (DNN) described above can be applied.

As described above, for example, a learning process in which a convolutional neural network CNN (Convolutional Neural Network) is combined with a BN (Batch Rectification) or a ReLU (Rectified Linear Unit) can be applied.
The learning processing unit 162 can be realized as a learning processing unit to which various learning algorithms are applied.

The learning processing unit 162 learns the weight of the neural network by using, for example, a learning process using a neural network in which BN or ReLU is combined with the above-mentioned CNN, for example, a stochastic gradient descent method.
From this learning process, the learning processing unit 162 receives the learning phase-limited correlation information (PoC) generated from the image showing the situation immediately before the component insertion and the model image as input, and inserts the component 20 into the component mounting pin 40. A learning model 172 that outputs a robot control parameter target value, which is the optimum robot control parameter of the above, is generated.

By using the generated learning model 172, it is possible to control the actual robot 100.
FIG. 12 is a diagram illustrating the configuration and processing of the robot control device 180 that controls the robot 100 using the learning model 172.

As shown in FIG. 12, the robot control device 180 has a phase-limited correlation information (PoC) calculation unit 181, a learning model 172, and a robot control unit 125.

The phase-limited correlation information (PoC) calculation unit 181 includes a camera image 121 taken by the camera 120 with the robot 100 moving the component 20 onto the component mounting pin 40 and a model image stored in the storage unit 150. 151 is input and the phase-limited correlation information (PoC) 191 is calculated from these two images.

The camera-captured image 121 and the model image 151 include two objects, that is, a component 20 and a component mounting pin 40, and these two objects as subjects, and the phase-limited correlation information (PoC) calculation unit 181 In the generated phase-limited correlation information (PoC) 191, peaks appear at coordinate positions corresponding to the amount of deviation between the images of these two objects. One of the two coordinates (x, y) corresponding to these two peaks corresponds to the deviation of the component 20, and the other corresponds to the deviation of the component mounting pin 40.

The phase-limited correlation information (PoC) 191 calculated by the phase-limited correlation information (PoC) calculation unit 181 is input to the learning model 172.
The learning model 172 is a learning model 172 generated by the learning processing device 160 described above with reference to FIG.
This learning model 172
The input is phase-limited correlation information (PoC).
The output is the robot control parameter target value, that is, the robot control parameter target value, which is the optimum robot control parameter for inserting the component 20 into the component mounting pin 40.
This is a learning model with the above settings.

The learning model 172 acquires the phase-limited correlation information (PoC) 191 calculated by the phase-limited correlation information (PoC) calculation unit 181, and the optimum control parameter of the robot corresponding to the acquired phase-limited correlation information (PoC) 191. The robot control parameter target value 192 shown in FIG. 12 is output.

As shown in FIG. 12, the robot control parameter target value 192 is input to the robot control unit 125, and the robot control unit 125 calculates the difference between the current robot control parameter 126 and the robot control parameter target value 192, and the difference. If there is, the robot 100 is corrected (re-controlled) so that the difference amount is zero or reduced.

By controlling the robot 100 using the robot control parameter target value 192 in this way, the position and orientation of the component 20 attached to the tip of the arm of the robot 100 are corrected and controlled, and then the component 20 is attracted. By releasing the component 20, the component 20 can be securely inserted into the component mounting pin 40.

The learning model 160 generated by the learning processing device 160 described with reference to FIGS. 9 to 11 and used by the robot control device 180 described with reference to FIG. 12 can be set in various ways.

A plurality of setting examples will be described with reference to FIG.
FIG. 13 shows a plurality of setting examples for each of the following data.
(A) Input data of the learning processing unit of the learning processing device (M) Learning model input / output data of the robot control device

The setting example (1) is the following setting.
(A) The input data of the learning processing unit of the learning processing device is
(A1) Phase-limited correlation information between the camera-captured image and the model image when the component insertion is successful (a2) Robot control parameters at the capture timing of the camera-captured image when the component insertion is successful

(M) The learning model input / output data of the robot control device is
The training model input data is
(Min1) Phase-limited correlation information between the camera-captured image and the model image at the component insertion timing The learning model output data is
(Mout) Robot control parameter target value.
This setting example (1) is a setting example corresponding to the embodiment described with reference to FIGS. 10 to 12.

In addition to this setting, for example, the following settings can be made.
The setting example (2) is the following setting.
(A) The input data of the learning processing unit of the learning processing device is
(A1) Phase-limited correlation information between the camera-captured image of the component insertion timing and the model image (a2) Robot control parameters at the capture timing of the camera-captured image of the component insertion timing (a3) Insertion success / failure flag indicating success or failure of component insertion

(M) The learning model input / output data of the robot control device is
The training model input data is
(Min1) Phase-limited correlation information between the camera-captured image and the model image at the component insertion timing The learning model output data is
(Mout) Robot control parameter target value.

Unlike the setting example (1), this setting example (2) is based not only on the phase-limited correlation information based on the captured image when the component insertion is successful but also on the captured image when the component insertion fails in the learning process in the learning processing device. This setting also uses phase-limited correlation information.
By including the failure information as learning data in this way, it is possible to classify different phase-limited correlation information at the time of success and failure, and highly accurate learning model output data, that is, (Mout) robot control parameter target value can be obtained. It can be generated and output.

The setting example (3) is the following setting.
(A) The input data of the learning processing unit of the learning processing device is
(A1) Phase-limited correlation information between the camera-captured image and the model image when the component insertion is successful (a2) Robot control parameters at the capture timing of the camera-captured image when the component insertion is successful

(M) The learning model input / output data of the robot control device is
The training model input data is
(Min1) Phase-limited correlation information between the camera-captured image and the model image of the component insertion timing (Min2) Robot control parameters at the capture timing of the camera-captured image The learning model output data is
(Mout) The difference between the current robot control parameter and the target value.

In this setting example (3), the input data of the learning processing unit of the (A) learning processing apparatus is the same as the setting example (1), that is, the embodiment described with reference to FIGS. 10 to 12.
(M) As learning model input data of the robot control device,
(Min2) Robot control parameter at the shooting timing of the image shot by the camera The difference is that this control parameter is added.
Moreover,
(M) As learning model output data of the robot control device
(Mout) Difference between the current robot control parameter and the target value It is a setting to output such difference data.
The robot control unit 125 can correct and control the robot 100 based on the difference data.

The setting example (4) is the following setting.
(A) The input data of the learning processing unit of the learning processing device is
(A1) Phase-limited correlation information between the camera-captured image and the model image when the component is successfully inserted (a2) Robot control parameters at the capture timing of the camera-captured image when the component is successfully inserted (a3) The camera-captured image when the component is successfully inserted

(M) The learning model input / output data of the robot control device is
The training model input data is
(Min1) Phase-limited correlation information between the camera-captured image of the component insertion timing and the model image (Min2) The camera-captured image of the camera-captured image of the camera-captured image The learning model output data is
(Mout) The difference between the current robot control parameter and the target value.

This setting example (3) is the input data of the learning processing unit of the learning processing apparatus (A) with respect to the setting example (1), that is, the input data of the embodiment described with reference to FIGS. 10 to 12. ,
(A3) Image taken by the camera when the component is successfully inserted The difference is that this photographed image is added.
Also, as the learning model input data of the (M) robot control device,
(Min2) Camera-captured image of camera-captured image The difference is that this captured image is added.

As described above, the phase-limited correlation information (PoC) between the camera-captured image and the model image of the component insertion timing includes two objects, that is, the component 20 and the component mounting pin 40, and between the images of these two objects. Two peaks appear at the coordinate positions according to the amount of deviation.
As input data of the learning processing unit
(A3) Image taken by the camera when the component is successfully inserted By inputting this captured image data, the two peaks appearing in the phase-limited correlation information (PoC) are the component 20, the component mounting pin 40, and these two objects. It is possible to easily determine which object corresponds to.

Also in the robot control device, as learning model input data
(Min2) Camera-captured image of camera-captured image capture timing By adding this captured image, it is possible to easily determine the correspondence between the input camera-captured image and the peak in the phase-limited correlation information (PoC). It becomes possible and the processing becomes efficient.

FIG. 14 is a diagram showing a configuration example of the learning processing device 160 corresponding to the setting example (4).
The following data are input to the learning processing unit 162 of the learning processing device 160.
(A1) Phase-limited correlation information (PoC) 171 generated by the calculation unit 161,
(A2) Robot control parameter 126 generated by the robot control unit 125 for controlling the robot 100,
Moreover,
(A3) Camera-photographed image 121 taken by the camera 120

The learning processing unit 162 executes machine learning processing using these input data.
In this learning process, the learning process unit 162 ensures that the two peaks appearing in the phase-limited correlation information (PoC) correspond to the component 20, the component mounting pin 40, or which of these two objects. It is possible to perform highly accurate learning processing.

FIG. 15 is a diagram showing a specific example of input data for the learning processing unit 162 of the learning processing device 160 shown in FIG.
As shown in FIG. 15, the learning processing unit 162
(A) Camera captured image (B) Phase-limited correlation information for learning (PoC),
(C) Robot control parameters when insertion is successful,
Enter many sets of these data.

The “(A) camera-photographed image” is an image immediately before the member is inserted when the component 20 is successfully inserted into the component mounting pin 40.
"(B) Phase-limited correlation information for learning (PoC)" is based on an image immediately before the component is inserted (camera image 121) when the component 20 is successfully inserted into the component mounting pin 40 and a model image 151. , Phase-limited correlation information (PoC) This is the phase-limited correlation information (PoC) calculated by the calculation unit 161.
"(C) Robot control parameter when insertion is successful" is robot control at the shooting timing of the image (camera shot image 121) applied to the calculation of the associated "(B) phase-limited correlation information for learning (PoC)". It is a parameter.

In this way, as a learning sample, (A) a camera-captured image, (B) learning phase-limited correlation information (PoC), and (C) a large amount of data sets of robot control parameters at the time of successful insertion are prepared, and these are prepared. Input to the learning processing unit 162.

The learning processing unit 162 executes learning processing on these input data and generates a learning model 172.
The learning model 172 generated by the learning processing unit 162
Input,
Phase-limited correlation information for learning (PoC) and
Camera shot image (camera shot image just before inserting parts),
The output is the robot control parameter target value, that is, the robot control parameter target value, which is the optimum robot control parameter for inserting the component 20 into the component mounting pin 40.
This is a learning model with the above settings.

As described above, the learning processing unit 162 ensures that the two peaks appearing in the phase-limited correlation information (PoC) correspond to the component 20, the component mounting pin 40, and which of these two objects. It is possible to perform highly accurate learning processing.

[5. About the example using the photographed image at the part acquisition position]
Next, an example using the captured image at the component acquisition position will be described.

In the above-described embodiment, the image taken immediately before the component 20 is attached to the component mounting pin 40 is used as the image used for generating the phase-limited correlation information (PoC).

An image taken at another timing may be applied to the image input to the phase-limited correlation information (PoC) calculation unit of the learning processing device or the robot control device.
For example, as shown in FIG. 16, at the timing when the robot 100 takes out the component 20 from the component box 10, an image is captured by using the camera 210, and the component acquisition position camera captured image 211 as shown in FIG. 16 is acquired. ..

Similar to the above-described embodiment, the component acquisition position camera captured image 211 and the image immediately before the robot 100 attaches the component 20 to the component mounting pin 40 are captured by the camera 250, and the component mounting position camera captured image 251 shown in FIG. To get.
Two types of phase-limited correlation information (PoC) are calculated using these two images taken at different timings, and learning processing is performed.

FIG. 17 is a diagram showing an additional configuration in this embodiment with respect to the learning processing device 160 of the first embodiment described above with reference to FIG.
The phase-limited correlation information (PoC) calculation unit 231 of the learning processing device 230 shown in FIG. 17 has two images of the component acquisition position camera captured image 211 and the component acquisition position model image 221 stored in the storage unit 150. The phase-limited correlation information (PoC) is calculated.

The component acquisition position model image 221 stored in the storage unit 150 is an image showing the acquisition status of the ideal component 20, and is an image taken in advance.

Only the component 20 is photographed in the component acquisition position camera image 211 and the model image 221, and the component mounting pin 40 is not photographed as in the above-described embodiment.
Therefore, the phase-limited correlation information (PoC) 241 generated by the phase-limited correlation information (PoC) calculation unit 231 has one peak at the coordinate position corresponding to the positional deviation between the two images of the component 20 as shown in the figure. The image will only be detected.

The learning processing unit 232 executes the learning process using this image.
FIG. 18 is a diagram illustrating details of the processing executed by the learning processing unit 232.
As shown in FIG. 18, the learning processing unit 232 is
(A) Phase-limited correlation information (PoC) for learning corresponding to the component acquisition position
(B) Phase-limited correlation information (PoC) for learning corresponding to the mounting position of parts,
(C) Robot control parameters when insertion is successful,
Enter many sets of these data.

"(A) Phase-limited correlation information (PoC) for learning for component acquisition position" is a photographed image (camera image 211) at the component acquisition position when the component 20 is successfully inserted into the component mounting pin 40, and a model image. This is the phase-limited correlation information (PoC) calculated by the phase-limited correlation information (PoC) calculation unit 161 based on 151.
"(B) Phase-limited correlation information (PoC) for learning component mounting position correspondence" includes an image (camera image 121) immediately before component insertion when the component 20 is successfully inserted into the component mounting pin 40, and a model image 151. Based on the above, it is the phase-limited correlation information (PoC) calculated by the phase-limited correlation information (PoC) calculation unit 161.
"(C) Robot control parameter when insertion is successful" is robot control at the shooting timing of the image (camera shot image 121) applied to the calculation of the associated "(B) phase-limited correlation information for learning (PoC)". It is a parameter.

The "(C) robot control parameter when the insertion is successful" is composed of, for example, the position (x, y, z) of the robot arm and the roll, pitch, and yaw (roll, pitch, yaw) indicating the arm posture. Dimensional real-valued vector (action), etc.

In this way, as learning samples, (A) phase-limited correlation information (PoC) for learning for component acquisition position, (B) phase-limited correlation information (PoC) for learning for component mounting position, and (C) when insertion is successful. A large number of pairs of robot control parameters are prepared, and these are input to the learning processing unit 232.
The learning processing unit 232 executes learning processing on these input data and generates a learning model 242.

The learning model 242 generated by the learning processing unit 232 is
Inputs are phase-limited correlation information (PoC) corresponding to the component acquisition position and phase-limited correlation information (PoC) corresponding to the component mounting position.
The output is the robot control parameter target value, that is, the robot control parameter target value, which is the optimum robot control parameter for inserting the component 20 into the component mounting pin 40.
This is a learning model with the above settings.

The learning process executed by the learning processing unit 232 is so-called machine learning, and various learning algorithms can be applied. For example, the learning process using the deep neural network (DNN) described above can be applied.

As described above, for example, a learning process in which a convolutional neural network CNN (Convolutional Neural Network) is combined with a BN (Batch Rectification) or a ReLU (Rectified Linear Unit) can be applied.
The learning processing unit 162 can be realized as a learning processing unit to which various learning algorithms are applied.

The learning processing unit 232 learns the weight of the neural network by using, for example, a learning process using a neural network in which BN or ReLU is combined with the above-mentioned CNN, for example, a stochastic gradient descent method.
The learning processing unit 232 is subjected to this learning processing.
Inputs are phase-limited correlation information (PoC) corresponding to the component acquisition position and phase-limited correlation information (PoC) corresponding to the component mounting position.
The output is the robot control parameter target value, that is, the robot control parameter target value, which is the optimum robot control parameter for inserting the component 20 into the component mounting pin 40.
The learning model 242 set as described above is generated.

By using the generated learning model 242, it is possible to control the actual robot 100.
FIG. 19 is a diagram illustrating the configuration and processing of the robot control device 260 that controls the robot 100 using the learning model 242.

As shown in FIG. 19, the robot control device 260 has a phase-limited correlation information (PoC) calculation unit 261a, 261b, a learning model 242, and a robot control unit 125.

The phase-limited correlation information (PoC) calculation unit 261a is a component acquisition position captured by the camera 210 with the robot 100 taking out the component 20 from the component box 10, and a component acquisition position stored in the storage unit a150a. The model image 221 is input, and the component acquisition position corresponding phase-limited correlation information (PoC) 271a is calculated from these two images.

Further, the phase-limited correlation information (PoC) calculation unit 261b is attached to the component mounting position camera image 251 captured by the camera 250 while the robot 100 moves the component 20 onto the component mounting pin 40 and the storage unit b150b. The stored component mounting position model image 242 is input, and the component mounting position corresponding phase-limited correlation information (PoC) 271b is calculated from these two images.

The original image that generated the phase-limited correlation information (PoC) 271a corresponding to the component acquisition position includes one object, that is, the component 20 as a subject, and was generated by the phase-limited correlation information (PoC) calculation unit 261a. In the phase-limited correlation information (PoC) 271a corresponding to the component acquisition position, one peak appears at the coordinate position corresponding to the amount of deviation between the images for one object.

On the other hand, the original image that generated the phase-limited correlation information (PoC) 271b corresponding to the component mounting position includes two objects, that is, the component 20, the component mounting pin 40, and these two objects as subjects, and the phase-limited correlation. In the component mounting position-corresponding phase-limited correlation information (PoC) 271b generated by the information (PoC) calculation unit 261b, a peak appears at a coordinate position corresponding to the amount of deviation between the images of these two objects. One of the two coordinates (x, y) corresponding to these two peaks corresponds to the amount of deviation of the component 20, and the other corresponds to the amount of deviation of the component mounting pin 40.

The component acquisition position-corresponding phase-limited correlation information (PoC) 271a calculated by the phase-limited correlation information (PoC) calculation units 261a and 261b and the component mounting position-corresponding phase-limited correlation information (PoC) 271b are input to the learning model 242. ..

The learning model 242 is a learning model 242 generated by the learning processing unit 232 described above with reference to FIG.
This learning model 242
Inputs are phase-limited correlation information (PoC) corresponding to the component acquisition position and phase-limited correlation information (PoC) corresponding to the component mounting position.
The output is the robot control parameter target value, that is, the robot control parameter target value, which is the optimum robot control parameter for inserting the component 20 into the component mounting pin 40.
It is a learning model 242 set as described above.

The learning model 242 acquires the component acquisition position-corresponding phase-limited correlation information (PoC) 271a calculated by the phase-limited correlation information (PoC) calculation units 261a and 261b and the component mounting position-corresponding phase-limited correlation information (PoC) 271b. Optimal control of the robot corresponding to the component acquisition position corresponding phase limited correlation information (PoC) 271a calculated by the acquired phase limited correlation information (PoC) calculation units 261a and 261b and the component mounting position corresponding phase limited correlation information (PoC) 271b. The parameter, that is, the robot control parameter target value 273 shown in FIG. 19 is output.

As shown in FIG. 19, the robot control parameter target value 273 is input to the robot control unit 125, and the robot control unit 125 calculates the difference between the current robot control parameter 126 and the robot control parameter target value 273, and the difference. If there is, the robot 100 is corrected and controlled so that the difference amount is zero or reduced.

By controlling the robot 100 using the robot control parameter target value 273 in this way, the position and orientation of the component 20 attached to the tip of the arm of the robot 100 are corrected and controlled, and then the component 20 is attracted. By releasing the component 20, the component 20 can be securely inserted into the component mounting pin 40.

[6. About learning model update process]
Next, the learning model update process will be described.
The learning model generated by the learning processing device can be updated one after another.
For example, during the period when the robot 100 is actually operating in the manufacturing process, a new image is continuously captured by the camera, and the captured image is used to generate phase-limited correlation information (PoC) to generate the phase-limited correlation information (PoC). Correlation information (PoC) can be input to the learning processing unit to continuously execute the learning process to update the learning model.

As the input data for the learning processing unit, various settings can be made as described above with reference to FIG.
For example, as described with reference to setting (2) in FIG.
(A) As input data of the learning processing unit of the learning processing device
(A1) Phase-limited correlation information between the camera-captured image of the component insertion timing and the model image (a2) Robot control parameters at the capture timing of the camera-captured image of the component insertion timing (a3) Insertion success / failure flags indicating success or failure of component insertion These If it is set to input the data of, it is possible to use all the captured image data regardless of the success or failure of the component insertion.

By including the failure information as learning data in this way, it is possible to classify different phase-limited correlation information at the time of success and failure, and generate highly accurate learning model output data, that is, a robot control parameter target value. Can be output.

In the learning processing unit, for example, parameters are updated by the stochastic gradient descent method of a normal neural network. The neural network parameters after training are written to, for example, a file, and the parameters of the network constituting the training model are updated as appropriate using this file.

The learning process described above corresponds to regression learning that determines the control parameters (actions) of the robot using images taken by the camera, but is a reinforcement learning type that updates the learning model according to the reward. It may be a model of.

In the above-described embodiment, the learning processing device and the robot control device using the phase-limited correlation information have been described. However, as described above, the learning processing device and the robot control device of the present disclosure are phase-limited. Not limited to correlation information, a configuration that applies various image correlation information that can calculate image deviation information in pixel (bixel) units, such as rotation-invariant phase-limited correlation information and image correlation information using image phase spectra. It is possible.

[7. About hardware configuration example of learning processing device and robot control device]
Next, an example of the hardware configuration of the learning processing device and the robot control device will be described.

The learning processing device and the robot control device described in the above-described embodiment can be configured as individual devices, or can be configured as one device.
The learning processing device and the robot control device can be realized by using an information processing device such as a PC.
FIG. 20 is a block diagram showing a configuration example of the learning processing device of the present disclosure and an information processing device constituting the robot control device.

The CPU (Central Processing Unit) 301 functions as a control unit or a data processing unit that executes various processes according to a program stored in the ROM (Read Only Memory) 302 or the storage unit 308. For example, the process according to the sequence described in the above-described embodiment is executed. The RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. These CPU 301, ROM 302, and RAM 303 are connected to each other by a bus 304.

The CPU 301 is connected to the input / output interface 305 via the bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various switches, a keyboard, a mouse, a microphone, a sensor, etc., and an output unit 307 consisting of a display, a speaker, and the like. Has been done. The CPU 301 executes various processes in response to a command input from the input unit 306, and outputs the process results to, for example, the output unit 307.

The storage unit 308 connected to the input / output interface 305 is composed of, for example, a hard disk or the like, and stores programs executed by the CPU 301 and various data. The communication unit 309 functions as a transmission / reception unit for Wi-Fi communication, Bluetooth (registered trademark) (BT) communication, and other data communication via a network such as the Internet or a local area network, and communicates with an external device.

The drive 310 connected to the input / output interface 305 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and records or reads data.

[8. Summary of the structure of this disclosure]
As described above, the examples of the present disclosure have been described in detail with reference to the specific examples. However, it is self-evident that one of ordinary skill in the art can modify or substitute the examples without departing from the gist of the present disclosure. That is, the present invention has been disclosed in the form of an example, and should not be construed in a limited manner. In order to judge the gist of this disclosure, the column of claims should be taken into consideration.

The technology disclosed in the present specification can have the following configuration.
(1) An image correlation information calculation unit that generates image correlation information between a camera-photographed image and a model image,
A learning processing device having a learning processing unit that executes a learning process for inputting the image correlation information and generating a learning model that outputs control parameters for controlling the position of an object captured by the camera image.

(2) The learning processing device according to (1), wherein the image correlation information is phase-limited correlation information.

(3) The image correlation information generated by the image correlation information calculation unit is
The learning processing device according to (1) or (2), which is information having a peak at a coordinate position corresponding to a positional deviation between the camera-captured image and an object included in the model image.

(4) The image taken by the camera is
An image containing objects acquired by a robot
The learning processing device according to any one of (1) to (3), wherein the model image is an image having an ideal position of the object.

(5) The image taken by the camera is
It is an image including two objects, a part object acquired by a robot and an object to which the part object is mounted.
The learning processing device according to any one of (1) to (4), wherein the model image is an image having a positional relationship between two objects that can reliably mount the component object on the mounting destination object.

(6) The image correlation information generated by the image correlation information calculation unit is
The learning according to (5), which is information having individual peaks at coordinate positions corresponding to the positional deviations of the camera-captured image, the component object included in the model image, and the object corresponding to each of the mounting destination objects. Processing equipment.

(7) The learning processing unit
With the image correlation information
The learning processing apparatus according to any one of (1) to (6), wherein a learning process for generating the learning model is executed by inputting a control parameter for controlling the position of the object at the shooting timing of the image captured by the camera.

(8) The image captured by the camera is an image immediately before the successful mounting process of the component object acquired by the robot on the mounting destination object.
The learning processing unit
For inputting the image correlation information generated using the image immediately before the successful mounting process of the component object acquired by the robot to the mounting destination object, and controlling the position of the object captured in the camera captured image. The learning processing apparatus according to any one of (1) to (7), which generates a learning model that outputs control parameters.

(9) The camera-captured image includes both an image immediately before the successful mounting process of the component object to the mounting destination object acquired by the robot and an image immediately before the failure.
The learning processing unit
The image correlation information generated by using the image immediately before the successful mounting process of the component object acquired by the robot and the image correlation generated by using the image immediately before the failure. The learning process according to any one of (1) to (8), which inputs information and success / failure information of the mounting process and generates a learning model that outputs a control parameter for controlling the position of the object captured by the camera image. Device.

(10) The learning processing unit is
The learning processing apparatus according to any one of (1) to (9), which executes a learning process using a deep neural network (DNN).

(11) A robot control unit that controls the robot and
An image correlation information calculation unit that generates image correlation information between a camera-photographed image of an object acquired by the robot and a model image in which the object is arranged at an ideal position.
It has a learning model that inputs the image correlation information and outputs a control parameter target value for controlling the position of the object captured in the image captured by the camera.
The robot control unit is a robot control device that controls the robot by using a control parameter target value that is an output of the learning model.

(12) The robot control device according to (11), wherein the image correlation information is phase-limited correlation information.

(13) The image correlation information generated by the image correlation information calculation unit is
The robot control device according to (11) or (12), which is information having a peak at a coordinate position corresponding to a positional deviation between the camera-captured image and an object included in the model image.

(14) The image taken by the camera is
It is an image including two objects, a part object acquired by the robot and an object to which the part object is mounted.
The robot control device according to any one of (11) to (13), wherein the model image is an image having a positional relationship between two objects that can reliably mount the component object on the mounting destination object.

(15) The image correlation information generated by the image correlation information calculation unit is
The robot according to (14), which is information having individual peaks at coordinate positions corresponding to the positional deviations of the camera-captured image, the component object included in the model image, and the object corresponding to each of the mounting destination objects. Control device.

(16) This is a learning processing method executed in the learning processing device.
The image correlation information calculation unit generates the image correlation information of the camera-captured image and the model image, and the image correlation information calculation step.
A learning processing method comprising a step in which a learning processing unit executes a learning process of inputting the image correlation information and generating a learning model that outputs a control parameter for controlling the position of an object captured by the camera captured image.

(17) This is a robot control method executed in a robot control device.
The steps in which the robot control unit controls the robot,
An image correlation information calculation step in which the image correlation information calculation unit generates image correlation information between the camera-photographed image of the object acquired by the robot and the model image in which the object is arranged at an ideal position.
A step of inputting the image correlation information to the learning model and outputting a control parameter target value for controlling the position of the object captured in the image captured by the camera.
A robot control method in which the robot control unit recontrols the robot by using a control parameter target value which is an output of the learning model.

(18) A program that executes learning processing in a learning processing device.
An image correlation information calculation step that causes the image correlation information calculation unit to generate image correlation information of a camera-photographed image and a model image, and
A program that causes a learning process unit to execute a step of inputting the image correlation information and executing a learning process of generating a learning model that outputs control parameters for controlling the position of an object captured by the camera image.

(19) A program that executes robot control processing in a robot control device.
Steps to let the robot control unit control the robot,
An image correlation information calculation step for causing the image correlation information calculation unit to generate image correlation information between the camera-photographed image of the object acquired by the robot and the model image in which the object is arranged at an ideal position.
A step of inputting the image correlation information to the learning model and outputting a control parameter target value for controlling the position of the captured object in the camera-captured image.
A program that causes the robot control unit to recontrol the robot by using a control parameter target value that is an output of the learning model.

Note that the series of processes described in the specification can be executed by hardware, software, or a composite configuration of both. When executing processing by software, install the program that records the processing sequence in the memory in the computer built in the dedicated hardware and execute it, or execute the program on a general-purpose computer that can execute various processing. It can be installed and run. For example, the program can be pre-recorded on a recording medium. In addition to installing on a computer from a recording medium, it is possible to receive a program via a network such as LAN (Local Area Network) or the Internet and install it on a recording medium such as a built-in hard disk.

Further, the various processes described in the specification are not only executed in chronological order according to the description, but may also be executed in parallel or individually as required by the processing capacity of the device that executes the processes. Further, in the present specification, the system is a logical set configuration of a plurality of devices, and the devices having each configuration are not limited to those in the same housing.

As described above, according to the configuration of one embodiment of the present disclosure, a device and a method for executing learning processing and robot control processing using phase-limited correlation information are realized.
Specifically, for example, the phase-limited correlation information calculation unit that generates the phase-limited correlation information of the camera-captured image and the model image, and the phase-limited correlation information are input to control the position of the object captured in the camera-captured image. Generate a learning model that outputs the control parameters of. Furthermore, it is a robot control device that controls the robot using the generated learning model, and provides phase-limited correlation information between the camera-captured image of the object acquired by the robot and the model image in which the object is placed at an ideal position. The robot is controlled by inputting to the learning model and using the control parameter target value obtained as the output from the learning model.
With this configuration, a device and a method for executing learning processing and robot control processing using phase-limited correlation information are realized.

10 Parts box 20 Parts 30 Belt conveyor 40 Parts mounting pin 100 Robot 101 Suction unit 120 Camera 121 Camera image 125 Robot control unit 126 Robot control parameter 130 Learning processing unit 131 Learning model 132 Robot control parameter Target value 150 Storage unit 151 Model image 160 Learning processing device 161 Phase-limited correlation information (PoC) calculation unit 171 Phase-limited correlation information (PoC)
172 Learning model 180 Robot controller 181 Phase-limited correlation information (PoC) calculation unit 191 Phase-limited correlation information (PoC)
192 Robot control parameter Target value 210 Camera 211 Parts acquisition position Camera shot image 221 Parts acquisition position model image 230 Learning processing device 231 Phase-limited correlation information (PoC) Calculation unit 232 Learning processing unit 241 Phase-limited correlation information (PoC)
242 Learning model 250 Camera 251 Parts mounting position Camera image 260 Robot control device 261 Phase-limited correlation information (PoC) calculation unit 271 Phase-limited correlation information (PoC)
273 Robot control parameter target value 301 CPU
302 ROM
303 RAM
304 Bus 305 Input / output interface 306 Input unit 307 Output unit 308 Storage unit 309 Communication unit 310 Drive 311 Removable media

Claims (19)

1. An image correlation information calculation unit that generates image correlation information between a camera-captured image and a model image,
   A learning processing device having a learning processing unit that executes a learning process for inputting the image correlation information and generating a learning model that outputs control parameters for controlling the position of an object captured by the camera image.
2. The learning processing device according to claim 1, wherein the image correlation information is phase-limited correlation information.
3. The image correlation information generated by the image correlation information calculation unit is
   The learning processing device according to claim 1, which is information having a peak at a coordinate position corresponding to a positional deviation between the camera-captured image and an object included in the model image.
4. The image taken by the camera is
   An image containing objects acquired by a robot
   The learning processing device according to claim 1, wherein the model image is an image having an ideal position of the object.
5. The image taken by the camera is
   It is an image including two objects, a part object acquired by a robot and an object to which the part object is mounted.
   The learning processing device according to claim 1, wherein the model image is an image having a positional relationship between two objects that can reliably mount the component object on the mounting destination object.
6. The image correlation information generated by the image correlation information calculation unit is
   The learning according to claim 5, which is information having individual peaks at coordinate positions corresponding to the positional deviations of the camera-captured image, the component object included in the model image, and the object corresponding to each of the mounting destination objects. Processing equipment.
7. The learning processing unit
   With the image correlation information
   The learning processing apparatus according to claim 1, wherein a learning process for generating the learning model is executed by inputting a control parameter for controlling the position of the object at the shooting timing of the image captured by the camera.
8. The image captured by the camera is an image immediately before the successful mounting process of the component object acquired by the robot on the mounting destination object.
   The learning processing unit
   For inputting the image correlation information generated using the image immediately before the successful mounting process of the component object acquired by the robot to the mounting destination object, and controlling the position of the object captured in the camera captured image. The learning processing device according to claim 1, which generates a learning model that outputs control parameters.
9. The camera-captured image includes both an image immediately before the successful mounting process of the component object to the mounting destination object acquired by the robot and an image immediately before the failure.
   The learning processing unit
   The image correlation information generated by using the image immediately before the successful mounting process of the component object acquired by the robot and the image correlation generated by using the image immediately before the failure. The learning processing device according to claim 1, wherein the learning processing device generates a learning model that inputs information and success / failure information of the mounting process and outputs a control parameter for controlling the position of an object captured by the camera-captured image.
10. The learning processing unit
    The learning processing apparatus according to claim 1, which executes a learning process using a deep neural network (DNN).
11. A robot control unit that controls the robot and
    An image correlation information calculation unit that generates image correlation information between a camera-photographed image of an object acquired by the robot and a model image in which the object is arranged at an ideal position.
    It has a learning model that inputs the image correlation information and outputs a control parameter target value for controlling the position of the object captured in the image captured by the camera.
    The robot control unit is a robot control device that controls the robot by using a control parameter target value that is an output of the learning model.
12. The robot control device according to claim 11, wherein the image correlation information is phase-limited correlation information.
13. The image correlation information generated by the image correlation information calculation unit is
    The robot control device according to claim 11, which is information having a peak at a coordinate position corresponding to a positional deviation between the camera-captured image and an object included in the model image.
14. The image taken by the camera is
    It is an image including two objects, a part object acquired by the robot and an object to which the part object is mounted.
    The robot control device according to claim 11, wherein the model image is an image having a positional relationship between two objects that can reliably mount the component object on the mounting destination object.
15. The image correlation information generated by the image correlation information calculation unit is
    The robot according to claim 14, which is information having individual peaks at coordinate positions corresponding to the positional deviations of the camera-captured image, the component object included in the model image, and the object corresponding to each of the mounting destination objects. Control device.
16. It is a learning processing method executed in the learning processing device.
    The image correlation information calculation unit generates the image correlation information of the camera-captured image and the model image, and the image correlation information calculation step.
    A learning processing method comprising a step in which a learning processing unit executes a learning process of inputting the image correlation information and generating a learning model that outputs a control parameter for controlling the position of an object captured by the camera captured image.
17. It is a robot control method executed in a robot control device.
    The steps in which the robot control unit controls the robot,
    An image correlation information calculation step in which the image correlation information calculation unit generates image correlation information between the camera-photographed image of the object acquired by the robot and the model image in which the object is arranged at an ideal position.
    A step of inputting the image correlation information to the learning model and outputting a control parameter target value for controlling the position of the object captured in the image captured by the camera.
    A robot control method in which the robot control unit recontrols the robot by using a control parameter target value which is an output of the learning model.
18. It is a program that executes learning processing in the learning processing device.
    An image correlation information calculation step that causes the image correlation information calculation unit to generate image correlation information of a camera-photographed image and a model image, and
    A program that causes a learning process unit to execute a step of inputting the image correlation information and executing a learning process of generating a learning model that outputs control parameters for controlling the position of an object captured by the camera image.
19. A program that executes robot control processing in a robot control device.
    Steps to let the robot control unit control the robot,
    An image correlation information calculation step for causing the image correlation information calculation unit to generate image correlation information between the camera-photographed image of the object acquired by the robot and the model image in which the object is arranged at an ideal position.
    A step of inputting the image correlation information to the learning model and outputting a control parameter target value for controlling the position of the captured object in the camera-captured image.
    A program that causes the robot control unit to recontrol the robot by using a control parameter target value that is an output of the learning model.

Images