JP2020001127A - Picking system, picking processing equipment, and program - Google Patents

Abstract

To provide a picking system that makes an installation cost low and that is operated at high speed.SOLUTION: A picking system includes: a camera 21 for photographing an object that is a picking target; a distance sensor 22 for measuring a distance between it and the object; a first measurement part 112 for measuring either or both of coordinates of one or more points of the object photographed by the camera 21 and an attitude of the object at the one or more points; and a second measurement part 113 for measuring a distance in a depth direction, in terms of the one or more points of the object, on the basis of the result of measurement by the distance sensor 22.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a picking system, a picking processing device, and a program.

  Picking operations to pick up cluttered objects can be low-wage when performed by humans, despite severe physical labor.

  Therefore, the picking operation may be automated using a robot arm (which may be simply referred to as a “robot”).

  As a picking operation using a robot, there are, for example, a method using a three-dimensional vision sensor and a method using a two-dimensional vision sensor mounted on the tip of the robot.

JP 2013-043271 A JP 2013-101045 A

  However, since the three-dimensional vision sensor is expensive, there is a problem that the installation cost increases.

  In addition, since the two-dimensional vision sensor cannot estimate the distance in the depth direction with high accuracy, the two-dimensional vision sensor approaches the object to be picked while repeatedly estimating the distance in the depth direction. There is. In addition, in order to obtain the distance in the depth direction based on an estimated value instead of an actually measured value, it is necessary to use special software.

  In one aspect, the techniques described herein are directed to providing a picking system that operates at high speed with low installation costs.

  In one aspect, a picking system includes a robot having a hand mechanism for picking an object, wherein the picking system includes a camera that photographs an object to be picked, and a distance sensor that measures a distance between the object and the camera. A first measurement unit that measures one or both of coordinates of one or more points on the object captured by the camera and an orientation of the object at the one or more points, and measurement by the distance sensor. A second measuring unit that measures a distance in the depth direction for the one or more points on the object based on the result.

  According to the disclosed picking system, it is possible to provide a picking system that operates at high speed with low installation cost.

FIG. 4 is a diagram illustrating an example of an object to be picked in the embodiment. FIG. 1 is a diagram illustrating a first example of a system configuration of a picking system according to an embodiment. FIG. 4 is a diagram illustrating a second example of the system configuration of the picking system according to the embodiment. It is a figure showing the 3rd example of the system configuration of the picking system in an example. It is a figure showing the 4th example of the system configuration of the picking system in an example. FIG. 6 is a block diagram schematically illustrating a hardware configuration example and a functional configuration example of the picking processing device illustrated in FIGS. 2 to 5. 7 is a flowchart illustrating a process of estimating the coordinates and posture of a picking target object in the picking processing device illustrated in FIG. 6. 7 is a flowchart illustrating a learning process of a learning model using an actual bulk image in the picking processing device illustrated in FIG. 6. 7 is a flowchart illustrating a learning process of a learning model using a rendering image in the picking processing device illustrated in FIG. 6. 7 is a flowchart illustrating a learning process of a learning model using an actual bulk image and a rendered image in the picking processing apparatus illustrated in FIG. 6. 7 is a flowchart illustrating noise addition processing in the picking processing device illustrated in FIG. 6. It is a flowchart explaining the estimation process of the coordinate and attitude | position of the picking target object in the picking processing apparatus as a 1st modification. (A)-(C) is a figure explaining the generation processing of the surface texture in the picking processing apparatus as a 2nd modification. It is a flowchart explaining the production | generation process of the surface texture in the picking processing apparatus as a 2nd modification.

  Hereinafter, an embodiment will be described with reference to the drawings. However, the embodiment described below is merely an example, and there is no intention to exclude various modified examples and applications of technology not explicitly described in the embodiment. This embodiment can be implemented with various modifications without departing from the spirit thereof.

  In addition, each drawing is not intended to include only the components illustrated in the drawings, but may include other functions and the like.

  Hereinafter, in the drawings, portions denoted by the same reference numerals indicate similar portions.

[A] Example of Embodiment [A-1] Example of System Configuration FIG. 1 is a diagram illustrating an example of an object 32 to be picked in the example.

  As shown in FIG. 1, a large number of objects 32 to be picked (may be referred to as “work”) are randomly stored in a pallet 31 (may be referred to as “container”). In FIG. 1, reference numeral “32” is shown only for one of a plurality of objects, and reference numerals for other objects are omitted.

  In the example shown in FIG. 1, the object 32 has a cylindrical shape, but is not limited to this. The object 32 to be picked may have various shapes. The object 32 to be picked may not be accommodated in the pallet 31, and may be placed on, for example, a belt conveyor (not shown).

  Hereinafter, picking systems 100a to 100d, which are specific examples of the picking system 100, will be described with reference to FIGS.

  FIG. 2 is a diagram illustrating a first example of a system configuration of the picking system 100 according to the embodiment.

  In the example shown in FIG. 2, the picking system 100a includes a picking processing device 1, a camera 21 (may be referred to as a "two-dimensional vision sensor"), a distance sensor 22, a robot 23, a frame 24, a rail 25, and a pallet 31. Prepare.

  The picking processing device 1 controls the robot 23 based on the measurement results obtained by the camera 21 and the distance sensor 22. The details of the picking processing device 1 will be described later with reference to FIG.

  The camera 21 measures a position in one or more points of the object 32 in the x-axis and y-axis directions. The camera 21 measures the attitude of the object 32 in the x-axis and y-axis directions. In the picking system 100, a plurality of cameras 21 may be provided.

  The distance sensor 22 may be, for example, an ultrasonic sensor, a time of flight (TOF) sensor, or an electromagnetic wave sensor. The distance sensor 22 measures a distance in one or more points of the object 32 in the z-axis direction. In the picking system 100, a plurality of distance sensors 22 may be provided.

  The robot 23 grips the object 32 stored in the pallet 31 by the hand mechanism based on the control by the picking processing device 1. The hand mechanism may be any of various mechanisms for picking the object 32 by suction, a gripper, electromagnetism, or the like. The robot 23 may be, for example, a vertical articulated robot, a horizontal articulated robot, or an orthogonal robot.

  In the illustrated example, the robot 23 is attached to a base placed on the floor, but is not limited to this. The robot 23 may be attached to various places such as a top surface and a side surface of the frame 24.

  The frame 24 fixes the camera 21 and the distance sensor 22 above the pallet 31. The position at which the camera 21 and the distance sensor 22 are fixed to the frame 24 is not limited to the illustrated example, and may be obliquely upward with respect to the pallet 31 or laterally with respect to the pallet 31. .

  The rail 25 allows the distance sensor 22 to move in the x-axis direction and the y-axis direction.

  FIG. 3 is a diagram illustrating a second example of the system configuration of the picking system 100 according to the embodiment.

  In the example shown in FIG. 3, the picking system 100b includes a picking processing device 1, a camera 21, a distance sensor 22, a robot 23, a frame 24, and a pallet 31.

  3, the camera 21 is fixed to a frame 24, as in the example shown in FIG. On the other hand, the distance sensor 22 is attached near the hand mechanism of the robot 23.

  FIG. 4 is a diagram illustrating a third example of the system configuration of the picking system 100 according to the embodiment.

  In the example shown in FIG. 4, the picking system 100c includes a picking processing device 1, a camera 21, a distance sensor 22, a robot 23, and a pallet 31.

  In FIG. 4, the camera 21 and the distance sensor 22 are attached near the hand mechanism of the robot 23.

  FIG. 5 is a diagram illustrating a fourth example of the system configuration of the picking system 100 according to the embodiment.

  In the example shown in FIG. 5, the picking system 100d includes a picking processing device 1, a camera 21, a distance sensor 22, a robot 23, a frame 24, a pan / tilt table 26, and a pallet 31.

  The pan / tilt base 26 can tilt the attached distance sensor 22 in the x-axis direction and the y-axis direction.

  5, the camera 21 is directly fixed to the frame 24 as in the example shown in FIG. On the other hand, since the distance sensor 22 is fixed to the frame 24 via the pan / tilt table 26, the distance sensor 22 can measure a distance not only in a direction parallel to the z-axis but also in a direction other than the direction parallel to the z-axis.

  The shape of the frame 24 may be another shape such as an inverted L-shape such as a street light.

  FIG. 6 is a block diagram schematically illustrating a hardware configuration example and a functional configuration example of the picking processing device 1 illustrated in FIGS. 2 to 5.

  The picking processing device 1 includes a Central Processing Unit (CPU) 11, a memory 12, and a storage device 13. Further, the picking processing device 1 is communicably connected to the camera 21, the distance sensor 22, and the robot 23.

  The memory 12 is a storage device including a Read Only Memory (ROM) and a Random Access Memory (RAM).

  The storage device 13 is a device that stores data in a readable and writable manner. For example, a hard disk drive (HDD), a solid state drive (SSD), and a storage class memory (SCM) may be used. The storage device 13 stores the generated teacher data, learning model, and the like.

  The CPU 11 is a processing device that performs various controls and calculations, and realizes various functions by executing an Operating System (OS) or a program stored in the memory 12. That is, the CPU 11 may function as a learning model acquisition unit 111, a first measurement unit 112, a second measurement unit 113, and a machine learning processing unit 114, as shown in FIG.

  The CPU 11 is an example of a computer, and exemplarily controls the operation of the entire picking processing device 1. A device for controlling the entire operation of the picking processing device 1 is not limited to the CPU 11, and may be, for example, any one of an MPU, a DSP, an ASIC, a PLD, an FPGA, and a dedicated processor. Further, a device for controlling the operation of the entire picking processing device 1 may be a combination of two or more of a CPU, an MPU, a DSP, an ASIC, a PLD, an FPGA, and a dedicated processor. Note that MPU is an abbreviation for Micro Processing Unit, DSP is an abbreviation for Digital Signal Processor, and ASIC is an abbreviation for Application Specific Integrated Circuit. PLD is an abbreviation for Programmable Logic Device, and FPGA is an abbreviation for Field Programmable Gate Array.

  The learning model acquisition unit 111 acquires a learning model for the picking target object 32 stored in the storage device 13.

  The first measuring unit 112 measures a coordinate position and a posture on the xy plane of the object 32 to be picked based on the image captured by the camera 21 and the learning model acquired by the learning model acquiring unit 111. As the object 32 to be picked, for example, an object 32 in which no other object 32 exists above the plurality of objects 32 existing in the pallet 31 may be selected.

  In other words, the first measurement unit 112 measures one or both of the coordinates of one or more points on the object 32 captured by the camera 21 and the posture of the object 32 at the one or more points. In addition, the first measuring unit 112 may execute measurement on one or both of the coordinates and the posture based on the learning model acquired by the learning model acquiring unit 111.

  The second measuring unit 113 measures the distance in the z-axis direction between the hand of the robot 23 and the object 32 for the point on the object 32 measured by the first measuring unit 112 based on the information acquired from the distance sensor 22. I do.

  In other words, the second measuring unit 113 measures the distance in the depth direction at one or more points on the object 32 based on the measurement result by the distance sensor 22.

  The machine learning processing unit 114 creates teacher data based on the image captured by the camera 21 and the coordinates and orientation of the object 32, executes machine learning, and generates a learning model.

  In other words, the machine learning processing unit 114 combines one or both of the coordinates of a predetermined point of the object 32 and the posture of the object 32 at the predetermined point with an image captured by the camera 21. Thereby, it functions as a first machine learning processing unit that generates teacher data and executes machine learning.

  Further, the machine learning processing unit 114 may generate teacher data based on the 3D image generated by rendering, execute machine learning, and generate a learning model. At this time, the machine learning processing unit 114 may generate teacher data by adding noises such as scratches and dents actually occurring in the object 32 to the 3D image.

  In other words, the machine learning processing unit 114 functions as a second machine learning processing unit that generates teacher data and performs machine learning by adding noise to an image generated by rendering.

  Further, the machine learning processing unit 114 may generate teacher data based on both the image captured by the camera 21 and the 3D image generated by the rendering.

[A-2] Operation Example The process of estimating the coordinates and posture of the picking target object 32 in the picking processing apparatus 1 shown in FIG. 6 will be described with reference to the flowchart (steps S1 to S4) shown in FIG.

  The picking target object 32 is photographed by the camera 21 (step S1).

  The first measuring unit 112 estimates the (x, y) coordinates of the position at which the depth is to be measured, the (x, y) coordinates and the posture of the object 32 to be picked, by machine learning, based on the imaging result of the camera 21. (Step S2).

  The second measuring unit 113 moves the distance sensor 22 to an appropriate position and measures the distance to the object 32 (Step S3).

  The first measuring unit 112 and the second measuring unit 113 determine the (x, y, z) coordinates and posture of the object 32 to be picked, and control the robot 23 (step S4). Then, the process of estimating the coordinates and the posture of the object 32 to be picked ends.

  Next, the learning process of the learning model using the actual bulk image in the picking processing device 1 shown in FIG. 6 will be described with reference to the flowchart (steps S11 to S15) shown in FIG.

  The bulk environment is reproduced by the object 32 in the pallet 31 (step S11).

  The camera 21 captures images of the objects 32 stacked in bulk (step S12).

  The machine learning processing unit 114 determines whether the number of captured images has reached a sufficient number of data (step S13). The determination as to whether the captured image has reached a sufficient number of data may be made by determining whether the number of data has exceeded a threshold value, or may refer to the degree of variation in the captured image by humans or artificial intelligence. May be performed.

  If the number of data has not reached the sufficient number (see No route in step S13), the process returns to step S11.

  On the other hand, when the number of data has reached a sufficient number (see the Yes route in step S13), the machine learning processing unit 114 performs preprocessing on the data of the captured image (step S14).

  The machine learning processing unit 114 learns a learning model using the data on which the pre-processing has been performed (step S15). Then, the learning process of the learning model using the actual bulk image ends.

  Next, the learning process of the learning model using the rendered image in the picking processing device 1 shown in FIG. 6 will be described with reference to the flowchart (steps S21 to S26) shown in FIG.

  The machine learning processing unit 114 designs a 3D model of the picking target object 32 (step S21).

  The machine learning processing unit 114 reproduces the bulk environment using the 3D model (step S22).

  The machine learning processing unit 114 renders the bulk environment and stores the image and data such as the position and orientation of the point to be picked (step S23).

  The machine learning processing unit 114 determines whether the rendering image has reached a sufficient number of data (step S24). The determination as to whether the number of data in the rendering image has reached a sufficient number of data may be made by determining whether the number of data has exceeded a threshold value, or by human or artificial intelligence referring to the degree of dispersion of the rendering image. It may be.

  If the number of data has not reached the sufficient number (see the No route in step S24), the process returns to step S22.

  On the other hand, when the number of data reaches a sufficient number (see the Yes route in step S24), the machine learning processing unit 114 performs preprocessing on the data of the rendering image (step S25).

  The machine learning processing unit 114 learns a learning model using the data on which the pre-processing has been performed (step S26). Then, the learning process of the learning model using the rendering image ends.

  Next, the learning process of the learning model using the actual bulk image and the rendered image in the picking processing device 1 shown in FIG. 6 will be described with reference to the flowchart (steps S31 to S40) shown in FIG.

  The machine learning processing unit 114 designs a 3D model of the picking target object 32 (step S31).

  The machine learning processing unit 114 reproduces the bulk environment using the 3D model (step S32).

  The machine learning processing unit 114 renders the bulk environment and stores the image and data such as the position and orientation of the point to be picked (step S33).

  The machine learning processing unit 114 determines whether the rendering image has reached a sufficient number of data (step S34). The determination as to whether the number of data in the rendering image has reached a sufficient number of data may be made by determining whether the number of data has exceeded a threshold value, or by human or artificial intelligence referring to the degree of dispersion of the rendering image. It may be.

  If the number of data has not reached the sufficient number (see the No route of step S34), the process returns to step S32.

  On the other hand, when the number of data has reached a sufficient number (see the Yes route in step S34), the machine learning processing unit 114 performs preprocessing on the data of the rendering image (step S35).

  The bulk environment is reproduced by the object 32 in the pallet 31 (step S36).

  The objects 21 stacked in bulk are photographed by the camera 21 (step S37).

  The machine learning processing unit 114 determines whether the number of captured images has reached a sufficient number of data (step S38). The determination as to whether the captured image has reached a sufficient number of data may be made by determining whether the number of data has exceeded a threshold value, or may refer to the degree of variation in the captured image by humans or artificial intelligence. May be performed.

  If the number of data has not reached the sufficient number (see No route in step S38), the process returns to step S36.

  On the other hand, when the number of data has reached a sufficient number (see the Yes route in step S38), the machine learning processing unit 114 performs preprocessing on the data of the captured image (step S39).

  The machine learning processing unit 114 learns a learning model using the data on which the preprocessing has been performed (step S40). Then, the learning process of the learning model using the actual bulk image and the rendered image ends.

  Next, the noise addition processing in the picking processing apparatus 1 shown in FIG. 6 will be described with reference to the flowchart (steps S41 to S43) shown in FIG.

  The machine learning processing unit 114 reproduces the bulk environment using the 3D rendering software (step S41).

  The machine learning processing unit 114 applies a machine learning model to which noise such as a scratch or dent is added to the rendered image (step S42).

  The machine learning processing unit 114 learns a machine learning model for picking (step S43). Then, the noise addition processing ends.

[A-3] Effects According to the picking system 100 in the example of the above-described embodiment, for example, the following effects can be obtained.

  The camera 21 photographs the object 32 to be picked. The distance sensor 22 measures the distance to the object 32. The first measurement unit 112 measures one or both of the coordinates of one or more points on the object 32 captured by the camera 21 and the posture of the object 32 at the one or more points. The second measuring unit 113 measures the distance in the depth direction at one or more points on the object 32 based on the measurement result by the distance sensor 22.

  This makes it possible to provide the picking system 100 that operates at high speed with low installation costs. When the camera 21 and the distance sensor 22 are attached to the robot 23, the frame 24 is unnecessary in the picking system 100, and the installation space of the picking system 100 can be reduced.

  The machine learning processing unit 114 combines the one or both of the coordinates of the predetermined point of the object 32 and the posture of the object 32 at the predetermined point with the image captured by the camera 21 to convert the teacher data. Generate and perform machine learning. The first measuring unit 112 measures one or both of the coordinates and the posture based on the learning model generated by the machine learning by the machine learning processing unit 114.

  As a result, an image equivalent to the final picking can be used as teacher data, so that the reliability of the evaluation result can be improved. In addition, since randomly generated noise such as a scratch formed in the object 32 can be learned, the robustness can be improved.

  The machine learning processing unit 114 generates teacher data by adding noise to the image generated by the rendering, and executes the machine learning. The first measuring unit 112 measures one or both of the coordinates and the posture based on the learning model generated by the machine learning by the machine learning processing unit 114.

  Thus, since it is not necessary to prepare a shooting environment or an actual object 32, a bulk environment can be reproduced indefinitely at low cost. In addition, it is possible to reproduce the object 32 having a position or orientation that hardly occurs in probability in reality, and to obtain an accurate position or orientation of a specific portion of the object 32 without error. Furthermore, since randomly generated noise such as a scratch formed on the object 32 can be learned, the robustness can be improved.

  The machine learning processing unit 114 may generate the teacher data based on both the image captured by the camera 21 and the image generated by the rendering.

  Thus, since the data of the 3D model can be utilized, a large amount of data can be created at low cost, and random noise in the actual object 32 can be learned. Further, it is possible to learn from the 3D model a posture or the like that is difficult for a human to set with high accuracy.

[B] First Modification In the above-described embodiment, the posture of the object 32 is estimated using the camera 21 (in other words, a “two-dimensional vision sensor”). However, the present invention is not limited to this. Not something. The orientation of the object 32 may be estimated by measuring the surface of the object 32 by measuring three or more points on the object 32 with the distance sensor 22.

  In other words, the second measuring unit 113 measures the distance in the depth direction for three or more points on the object 32 based on the measurement result by the distance sensor 22. Then, the second measuring unit 113 estimates the posture of the object 32 based on the measured distance in the depth direction for three or more points.

  The process of estimating the coordinates and orientation of the picking target object 32 in the picking processing apparatus 1 as a first modification will be described with reference to the flowchart (steps S51 to S55) shown in FIG.

  The object 32 to be picked is photographed by the camera 21 (step S51).

  The first measuring unit 112 estimates the positions of a plurality of points (in other words, “three or more points”) whose depths are to be measured and the (x, y) coordinates of the picking target object 32 by machine learning. (Step S52).

  The second measuring unit 113 moves the distance sensor 22 to an appropriate position based on the measurement result by the first measuring unit 112, and measures the distance of a plurality of points (Step S53).

  The second measuring unit 113 estimates the posture of the object 32 using the information on the plurality of points whose distance has been measured (step S54).

  The first measuring unit 112 and the second measuring unit 113 determine the (x, y, z) coordinates and posture of the object 32 to be picked, and control the robot 23 (step S55). Then, the process of estimating the coordinates and the posture of the object 32 to be picked in the first modified example ends.

  The same effect as in the above-described embodiment can be obtained by the estimation processing of the coordinates and the posture of the picking target object 32 in the first modification.

[C] Second Modification The addition of noise such as a scratch or a dent to the rendered image may be performed by photographing the surface texture of the real object 32.

  (A) to (C) of FIG. 13 are diagrams illustrating a process of generating a surface texture in the picking processing device 1 according to the second modification.

  FIG. 13A shows a person's smiling face, and FIG. 13C shows a person's expressionless face. By acquiring the image data of the facial expressions shown in FIGS. 13A and 13C, the image data of the intermediate facial expression shown in FIG. 13B may be generated.

  The image data of the intermediate expression shown in FIG. 13B is not limited to one, and a plurality of image data may be generated.

  In the second modification, the machine learning processing unit 114 learns the shape of the flaws, the degree of reflection, and the like in the plurality of objects 32. Then, the machine learning processing unit 114 generates a 3D image of the object 32 having a myriad of pattern-shaped flaws, reflection conditions, and the like based on the learned wound shapes, reflection conditions, and the like of the plurality of objects 32.

  In other words, the machine learning processing unit 114 generates a first image in which noise is added to the object 32, and a second image in which noise different from the first image is added to the object 32. To get. Then, based on the acquired first and second images, the machine learning processing unit 114 includes a third noise including a noise of a characteristic between the noise characteristic of the first image and the noise characteristic of the second image. It functions as an example of a second machine learning processing unit that generates an image.

  The generation processing of the surface texture in the picking processing apparatus 1 as the second modification will be described with reference to the flowchart (steps S61 to S66) shown in FIG.

  The surface texture of the real object 32 is photographed by the camera 21 (step S61). Here, for example, the surface textures of the two objects 32 are photographed.

  The machine learning processing unit 114 performs pre-processing on the data of the photographed surface texture (step S62).

  The machine learning processing unit 114 learns a latent variable representing the surface texture (Step S63).

  The machine learning processing unit 114 generates a myriad of surface textures with the learned latent variables changed (step S64).

  The machine learning processing unit 114 generates an infinite number of unique 3D models using the generated surface texture (step S65).

  The machine learning processing unit 114 performs a learning process of a learning model using the rendering image illustrated in FIG. 9 (Step S66). Then, the surface texture generation processing ends.

  According to the surface texture generation processing in the second modified example, by learning the features of the surface texture in the plurality of real objects 32, an infinite number of intermediate surface textures obtained by combining the surface textures in the plurality of real objects 32 are obtained. Can be generated. In addition, by applying the generated surface texture to the 3D model, noise such as random scratches generated in the real object 32 can be learned using only the 3D model.

[D] Others The disclosed technology is not limited to the embodiments described above, and can be implemented with various modifications without departing from the spirit of the present embodiment. Each configuration and each process of the present embodiment can be selected as needed, or can be appropriately combined.

100: Picking system 1: Picking processing device 11: CPU
111: learning model acquisition unit 112: first measurement unit 113: second measurement unit 114: machine learning processing unit 12: memory 13: storage device 21: camera 22: distance sensor 23: robot 24: frame 25: rail 26: pan・ Tilt table 31: Pallet 32: Object

In one aspect, a picking system comprises a robot having a hand mechanism for picking an object, the picking system comprising: a camera for photographing an object to be picked; and a distance between one or more points on the object . , A noise learning processing unit that generates teacher data by performing noise learning by adding noise to the image generated by the rendering, and a noise learning processing unit that generates the noise learning processing unit by the noise learning processing unit. based on the learning model, and the coordinates of the one or more points in the object captured by the camera, and the attitude of the object in the one or more points, a first measuring unit for measuring either or both, Based on the measurement result by the distance sensor, the depth of the one or more points on the object is And a second measuring unit for measuring a distance.

Claims (7)

A picking system comprising a robot having a hand mechanism for picking an object,
A camera for shooting the object to be picked,
A distance sensor for measuring the distance between the object,
A first measurement unit that measures one or both of the coordinates of one or more points on the object captured by the camera and the orientation of the object at the one or more points,
Based on the measurement result by the distance sensor, for the one or more points on the object, a second measurement unit that measures the distance in the depth direction,
A picking system comprising: By combining one or both of the coordinates of a predetermined point of the object and the posture of the object at the predetermined point with an image captured by the camera, generate teacher data and execute machine learning. Further comprising a first machine learning processing unit that performs
The first measurement unit performs the measurement on one or both of the coordinates and the posture based on a learning model generated by machine learning by the first machine learning processing unit.
The picking system according to claim 1. The image processing apparatus further includes a second machine learning processing unit that generates teacher data and performs machine learning by adding noise to the image generated by the rendering,
The first measurement unit performs the measurement on one or both of the coordinates and the posture based on a learning model generated by machine learning by the second machine learning processing unit.
The picking system according to claim 1. The second machine learning processing unit acquires a first image to which noise is added to the object and a second image to which noise different from the first image is added to the object. Then, based on the acquired first and second images, a third image including noise of a characteristic between the noise characteristic of the first image and the noise characteristic of the second image is generated. ,
The picking system according to claim 3. The second measuring unit includes:
Based on the measurement result by the distance sensor, for three or more points on the object, measure the distance in the depth direction,
Based on the measured distance in the depth direction for the three or more points, to estimate the posture of the object,
The picking system according to claim 1. A first measurement unit that measures one or both of the coordinates of one or more points on the object captured by the camera that captures the object to be picked, and the orientation of the object at the one or more points,
Based on the measurement result by the distance sensor, for the one or more points on the object, a second measurement unit that measures the distance in the depth direction,
A picking processing device comprising: On the computer,
Coordinates of one or more points on the object photographed by a camera that photographs the object to be picked, and one or both of the posture of the object at the one or more points,
Based on the measurement result by the distance sensor, for the one or more points on the object, measure the distance in the depth direction,
A program that executes a process.

Images