JP7005388B2 - Information processing equipment and sorting system - Google Patents

Description

Embodiments of the present invention relate to an information processing apparatus and a sorting system.

In a sorting system that delivers goods (luggage) such as parcels, it is necessary to sort the collected packages by delivery destination. For example, a sorting system including a robot arm having a gripping mechanism for gripping an article, an arm mechanism for moving the gripping mechanism, and an information processing device for instructing the robot arm to operate has been put into practical use.

The information processing device acquires an image of an article mounted on the car (car image). The information processing device performs image recognition on the car vehicle image using a parameter (image recognition parameter) for recognizing the article in the image, and determines the area (article area) in which the article exists in the car car image. recognize. The information processing apparatus determines a position (picking position) for gripping the article by the robot arm based on the recognition result of the article area. The information processing device generates an motion plan indicating the picking position and the movement path of the gripping mechanism, and supplies the motion plan to the robot arm.

The robot arm moves the gripping mechanism by the arm mechanism based on the motion plan supplied from the information processing device. At the picking position, the robot arm grips the article by the gripping mechanism and moves the article to the supplier.

The articles sorted by the sorting system are not uniform in size and appearance. In order to improve the accuracy of image recognition by the information processing device, it is effective to perform machine learning and adjust the image recognition parameters. However, machine learning requires a large amount of learning data. The learning data has a question and an answer. In the above information processing device, the car car image corresponds to the problem, and the position of the article area on the car car image corresponds to the answer. That is, there is a problem that a large number of car vehicle images in which the article area is specified in advance are required for machine learning.

Japanese Unexamined Patent Publication No. 2017-64910

An object to be solved by the present invention is to provide an information processing device and a sorting system capable of easily collecting learning data.

According to the embodiment, picking position data indicating a picking position for gripping the article is supplied to a robot arm including a gripping mechanism for gripping the article placed in a predetermined range and an arm mechanism for moving the gripping mechanism. At the same time, the information processing device that supplies the learning data to the learning device that adjusts the image recognition parameter for recognizing the article region in which the article exists from the image in the predetermined range includes a control unit. The control unit moves the article by supplying the picking position data to the robot arm based on an input operation to the operation interface, and the first image which is an image of the predetermined range before the article moves. And the second image which is the image of the predetermined range after the article has moved, the training data is generated, and the generated learning data is supplied to the learning device, and the first image and the image are used. Based on the difference from the second image, a mask image showing an article region in which the article exists on the first image is generated, and the learning data in which the first image is a problem and the mask image is the answer. Is supplied to the learning device, and it is determined whether or not to supply the learning data to the learning device based on the article region and the picking position data.

FIG. 1 is an explanatory diagram for explaining a schematic configuration example of the sorting system according to the first embodiment. FIG. 2 is an explanatory diagram for explaining a configuration example of the operation terminal according to the first embodiment. FIG. 3 is an explanatory diagram for explaining a configuration example of the information processing apparatus according to the first embodiment. FIG. 4 is a flowchart for explaining an example of the operation of the robot arm according to the first embodiment. FIG. 5 is a flowchart for explaining an example of the operation of the information processing apparatus according to the first embodiment. FIG. 6 is an explanatory diagram for explaining an example of the operation of the information processing apparatus in the image recognition mode. FIG. 7 is an explanatory diagram for explaining an example of a car vehicle image generated by the distance sensor. FIG. 8 is an explanatory diagram for explaining the recognition result of the article region in the car car image. FIG. 9 is an explanatory diagram for explaining the picking position determined based on the recognition result of the article region in the car vehicle image. FIG. 10 is an explanatory diagram for explaining an example of the operation of the information processing apparatus in the manual input mode according to the first embodiment. FIG. 11 is an explanatory diagram for explaining an example of an input screen for prompting input of a picking position. FIG. 12 is an explanatory diagram for explaining an example of a car car image after the article has been moved. FIG. 13 is an explanatory diagram for explaining an example of a mask image. FIG. 14 is an explanatory diagram for explaining an example of the operation of the information processing apparatus in the combined mode. FIG. 15 is an explanatory diagram for explaining an example of the operation of the information processing apparatus in the manual input mode according to the second embodiment. FIG. 16 is an explanatory diagram for explaining an example of an input screen for prompting input of an article area. FIG. 17 is an explanatory diagram for explaining the relationship between the article area and the picking position.

Hereinafter, embodiments will be described with reference to the drawings.
(First Embodiment)
FIG. 1 is an explanatory diagram for explaining a schematic configuration example of the sorting system 1 according to the first embodiment.

The sorting system 1 is a system for sorting the articles 3 loaded on the car 2 to a predetermined sorting destination.

The car 2 is a container in which a storage unit for accommodating the article 3 and casters are combined. The car 2 is an example of a container on which the article 3 to be sorted is loaded, and may be any container such as a loading platform or a pallet that can load the article 3.

The supplier of the article 3 is, for example, a belt conveyor 4. The sorting destination is not limited to the belt conveyor 4, and may be a basket for sorting, a workbench, or the like.

The sorting system 1 includes a robot arm 11, an information processing device 12, and an operation terminal 13. Further, the sorting system 1 further includes a distance sensor 14. Further, the sorting system 1 further includes a storage 15 and a learning device 16. The robot arm 11, the information processing device 12, the operation terminal 13, the distance sensor 14, the storage 15, and the learning device 16 are configured to be able to communicate with each other via the network 17.

First, the configuration of the robot arm 11 will be described.
The robot arm 11 is a device that grips an article loaded on a car 2, lifts the gripped article 3, and supplies the article 3 to a supplier. The robot arm 11 includes a gripping mechanism 21, an arm mechanism 22, a contact sensor 23, and a controller 24.

The gripping mechanism 21 is a mechanism for gripping the article 3. The gripping mechanism 21 includes, for example, a suction pad that sticks to the article 3. The suction pad is in contact with the surface of the article 3 and the inside of the pad becomes a negative pressure under the control of the controller 24, so that the suction pad is attracted to the surface of the article 3 and grips the article 3. Further, the gripping mechanism 21 may be configured as a gripper including a plurality of fingers for gripping the article 3 by sandwiching the article 3 with two or more points of contact.

The arm mechanism 22 is a mechanism for moving the gripping mechanism 21. The arm mechanism 22 includes a plurality of arms and a joint mechanism for connecting the plurality of arms. The joint mechanism drives the arm by being moved by an actuator (not shown) controlled by the controller 24.

The contact sensor 23 is a sensor that detects the stress applied to the gripping mechanism 21. The contact sensor 23 detects, for example, the stress applied to the gripping mechanism 21 from the vertical direction. The contact sensor 23 transmits the detection result to the controller 24. Specifically, when the contact sensor 23 detects the stress applied to the gripping mechanism 21 from the vertical direction, the contact sensor 23 supplies an on signal to the controller 24. The contact sensor 23 may also be configured to detect the stress applied to the arm mechanism 22.

The controller 24 controls the operations of the gripping mechanism 21 and the arm mechanism 22 based on the operation plan supplied from the information processing apparatus 12. The controller 24 includes, for example, a memory (not shown) and an arithmetic element (not shown) that controls the operation of the gripping mechanism 21 and the arm mechanism 22 by executing a program stored in the memory. Further, the controller 24 may be configured as a sequencer.

The motion plan is information including picking position data indicating a picking position and a locus. The controller 24 moves the gripping mechanism 21 to the picking position indicated by the picking position data of the operation plan through the locus indicated by the operation plan.

The picking position data is information indicating a position where the article 3 is gripped by the gripping mechanism 21. For example, the picking position data indicates a position where the gripping mechanism 21 is moved when the article 3 is gripped by the gripping mechanism 21. More specifically, the picking position data is the coordinates on the three-dimensional space of the movement destination of the reference point of the gripping mechanism 21. The picking position indicated by the picking position data is simply referred to as a picking position.

The locus shows a path when the gripping mechanism 21 of the robot arm 11 is moved to the picking position. More specifically, the locus is a plurality of coordinates on the three-dimensional space indicating the movement path of the reference point of the gripping mechanism 21. The picking position may be included in the locus.

When the gripping mechanism 21 is configured as a suction pad, the reference point of the gripping mechanism 21 may be, for example, the center (center of gravity) of the suction pad or any point on the suction pad. When the gripping mechanism 21 is configured as a gripper, the reference point of the gripping mechanism 21 may be, for example, the center of a plurality of fingers constituting the gripper.

Next, the configuration of the distance sensor 14 will be described.
The distance sensor 14 is a sensor that measures the distance to an object. The distance sensor 14 acquires a distance image composed of a point cloud in which points indicating the distance between the object and the distance sensor 14 are arranged. That is, the distance image is a set of points in a space consisting of x-direction, y-direction, and z-direction orthogonal to each other. Alternatively, a distance image is a set of distance values arranged on an array of pixels. The distance sensor 14 supplies the distance image to the information processing apparatus 12 via the network 17.

The distance sensor 14 is a stereo camera that measures the distance to the object based on the parallax when the object is imaged from two different points (pupil positions), for example. That is, the distance sensor 14 includes two or more cameras in which a lens and an image pickup element that converts the light imaged by the lens into an image are combined. With such a configuration, the distance sensor 14 acquires a raster image in which coordinates (pixels) having color information are two-dimensionally arranged at the same time as the above distance image. The raster image may be a color image or a monochromatic image. The distance sensor 14 supplies the raster image to the information processing apparatus 12 via the network 17.

The angle of view of the lens of the distance sensor 14 is adjusted so that the area including the car 2 on which the article 3 lifted by the robot arm 11 is loaded can be photographed. For example, the lens of the distance sensor 14 is adjusted so that the optical axis faces the bottom surface of the car 2. For example, the lens of the distance sensor 14 is adjusted so that the optical axis of the lens of the distance sensor 14 is parallel to the z direction (vertical direction). That is, the distance sensor 14 captures a predetermined range including the car 2 from the direction facing the bottom surface of the car 2 and acquires a distance image and a raster image. An image in a predetermined range including the car 2 is referred to as a car image. The car car image may be either a distance image or a raster image, but in the present embodiment, it is assumed that the car car image is a distance image.

The distance sensor 14 may have a configuration in which light from an object is separated and incident on different positions of the image pickup device. Further, the distance sensor 14 may be a one-dimensional scanner using a laser.

Next, the operation terminal 13 will be described.
FIG. 2 is an explanatory diagram for explaining a configuration example of the operation terminal 13.

The operation terminal 13 is a device that supplies information based on the operation to the information processing device 12. That is, the operation terminal 13 functions as an operation interface of the information processing device 12. The operation terminal 13 includes a communication interface 31, a control unit 32, and a touch panel 33.

The communication interface 31 is an interface for communicating with a device other than the operation terminal 13. The communication interface 31 includes terminals and circuits corresponding to communication standards for communicating with the robot arm 11, the information processing device 12, and the distance sensor 14 via the network 17. The communication interface 31 communicates with the robot arm 11, the information processing device 12, the distance sensor 14, and the like based on the control of the control unit 32.

The control unit 32 is a processing unit that executes various processes. The control unit 32 includes a processor 34 and a memory 35.

The processor 34 is an arithmetic element that executes arithmetic processing. The processor 34 is configured as, for example, a CPU. The processor 34 performs various processes based on the program stored in the memory 35.

The memory 35 is a storage device for storing programs and data. The memory 35 includes, for example, one or more of a ROM, which is a read-only non-volatile memory, a RAM for temporarily storing data, and a storage for storing data.

The touch panel 33 is a device that displays a screen and generates an operation signal based on the operation. The touch panel 33 includes an integrally configured display 36 and a touch sensor 37. The operation terminal 13 may be configured to include a display that displays a screen instead of the touch panel 33, and an operation unit that generates an operation signal based on the operation. The operation unit may be any mouse, trackball, keyboard, trackpad, or the like.

The display 36 displays a screen based on display data (screen data) supplied from the control unit 32 or a graphic controller (not shown).

The touch sensor 37 generates an operation signal indicating the position touched by the operator who operates the operation terminal 13 on the screen displayed on the display 36.

Next, the configuration of the information processing apparatus 12 will be described.
FIG. 3 is an explanatory diagram for explaining a configuration example of the information processing apparatus 12.

The information processing device 12 is a device that supplies an operation plan to the robot arm 11. The information processing device 12 acquires a car car image, which is an image of the article 3 mounted on the car car 2. The information processing device 12 performs image recognition on the car vehicle image by using the image recognition parameter which is a parameter for recognizing the article 3 reflected in the car vehicle image, and the area in which the article 3 is shown in the car vehicle image. Recognize the article area that is. The information processing apparatus 12 determines the picking position, which is the position where the article 3 is gripped by the robot arm 11, based on the recognition result of the article area. The information processing apparatus 12 generates an operation plan showing a picking position and a locus which is a movement path of the gripping mechanism 21, and supplies the operation plan to the robot arm 11.

Further, the information processing apparatus 12 generates learning data for causing the learning apparatus 16 to perform machine learning by a method described later. The information processing device 12 supplies the generated learning data to the storage 15.

The information processing device 12 is composed of a device such as a personal computer (PC) capable of processing data and storing data. The information processing device 12 includes a communication interface 41 and a control unit 42.

The communication interface 41 is an interface for communicating with a device other than the information processing device 12. The communication interface 41 includes terminals and circuits corresponding to communication standards for communicating with the robot arm 11, the operation terminal 13, and the distance sensor 14 via the network 17. The communication interface 41 communicates with the robot arm 11, the operation terminal 13, the distance sensor 14, the storage 15, the learning device 16, and the like based on the control of the control unit 42.

The control unit 42 is a processing unit that executes various processes. The control unit 42 includes a processor 43 and a memory 44.

The processor 43 is an arithmetic element that executes arithmetic processing. The processor 43 is configured as, for example, a CPU. The processor 43 performs various processes based on the program stored in the memory 44.

The memory 44 is a storage device for storing programs and data. The memory 44 includes, for example, one or more of a ROM, which is a read-only non-volatile memory, a RAM for temporarily storing data, and a storage for storing data. For example, the memory 44 stores image recognition parameters.

Next, the storage 15 and the learning device 16 will be described.
The storage 15 is a storage device for storing learning data. The storage 15 stores learning data supplied from the information processing device 12. The storage 15 supplies learning data to the learning device 16 in response to a request from the learning device 16.

The learning device 16 acquires learning data from the storage 15, performs machine learning based on the learning data, generates an image recognition parameter, and supplies the image recognition parameter to the information processing device 12. The learning device 16 includes a communication interface for communicating with devices other than the learning device 16, a control unit for executing various processes, a memory, and the like.

The learning data used in the learning device 16 is data having a problem and an answer. For example, the learning data has a car car image in which the article 3 to be sorted is shown, and information indicating an article area which is a region in which the article to be sorted is shown in the car car image. The car image corresponds to the problem, and the article area in the car image corresponds to the answer. The learning device 16 is used when the information processing device 12 recognizes an object region from a car vehicle image by performing machine learning based on such learning data, for example, by using a neural network for segmentation or the like. Generate image recognition parameters. The image recognition parameter is a weight count in a neural network for segmentation. The learning device 16 may be configured to generate an image recognition parameter each time the learning data is acquired, or has already been generated by using the image recognition parameter generated based on the acquired learning data. The configuration may be such that the image recognition parameters are adjusted (updated). Further, the learning device 16 may be configured to limit the values that can be adjusted at one time when the already generated image recognition parameter is adjusted (changed).

Next, the operation of the robot arm 11 will be described.
FIG. 4 is a flowchart for explaining the operation of the robot arm 11.

The controller 24 of the robot arm 11 moves the arm mechanism 22 and the gripping mechanism 21 to an initial position outside the angle of view of the distance sensor 14 (step S11). For example, the controller 24 drives the arm mechanism 22 by moving the joint mechanism of the arm mechanism 22 by an actuator, and moves the arm mechanism 22 and the gripping mechanism 21 out of the photographing range. The distance sensor 14 acquires a car vehicle image at the timing when the arm mechanism 22 and the gripping mechanism 21 move out of the shooting range, and transmits the image to the information processing device 12.

When the controller 24 moves the arm mechanism 22 and the gripping mechanism 21 out of the shooting range, the controller 24 waits for the operation plan to be supplied from the information processing device 12 (step S12).

When the operation plan is supplied from the information processing apparatus 12 (step S12, YES), the controller 24 moves the gripping mechanism 21 based on the operation plan (step S13). For example, the controller 24 controls the arm mechanism 22 so as to move the gripping mechanism 21 to the picking position indicated by the motion plan based on the locus indicated by the motion plan.

The controller 24 sequentially confirms the position of the gripping mechanism 21 and determines whether or not the picking position indicated by the operation plan has been reached. When the gripping mechanism 21 reaches the picking position, the controller 24 shifts to the process of the next step S14. Further, the controller 24 sequentially confirms the detection result (contact detection result) of the contact sensor 23, and when the gripping mechanism 21 reaches the picking position and the contact detection result is ON, the process of the next step S14 is performed. It may be a configuration to be migrated.

When the controller 24 determines that the gripping mechanism 21 has reached the picking position, the gripping mechanism 21 grips the article 3 (step S14). For example, when the gripping mechanism 21 is configured as a suction pad, the controller 24 causes the gripping mechanism 21 to grip the article by making the inside of the suction pad a negative pressure by a vacuum pump or the like (not shown). Further, for example, when the gripping mechanism 21 is configured as a gripper, the controller 24 causes the gripping mechanism 21 to grip the article by pinching the article with the fingers of the gripper.

The controller 24 determines whether or not the article 3 has been normally gripped after the gripping mechanism 21 has performed the operation of gripping the article 3 (step S15). That is, the controller 24 determines whether or not the article 3 could be gripped by the gripping mechanism 21. For example, when the gripping mechanism 21 is configured as a suction pad, the controller 24 determines that the article 3 cannot be gripped by the gripping mechanism 21 when the inside of the suction pad does not have a negative pressure. Further, for example, when the gripping mechanism 21 is configured as a gripper, the controller 24 uses the gripping mechanism 21 when the stress presumed to be applied to the finger portion of the gripper does not occur by sandwiching the article 3 with the gripper. It is determined that the article 3 cannot be grasped.

When the controller 24 determines that the article 3 has been normally gripped by the gripping mechanism 21 (step S15, YES), the controller 24 moves the gripping mechanism 21 gripping the article 3 to a position corresponding to the belt conveyor 4 according to the supplier. The arm mechanism 22 is controlled so as to be moved (step S16).

When the gripping mechanism 21 is moved to a position corresponding to the belt conveyor 4 according to the supplier, the controller 24 controls the gripping mechanism 21 so as to release the gripping of the article 3 by the gripping mechanism 21 (step S17), 1 A completion notification indicating that the sorting of the two articles has been completed is transmitted to the information processing apparatus 12, and the process proceeds to step S11. As a result, the article 3 separated from the gripping mechanism 21 is put into the belt conveyor 4 of the supplier. Further, the controller 24 moves the arm mechanism 22 and the gripping mechanism 21 out of the angle of view of the distance sensor 14, and waits for the reception of the next motion plan.

Further, when the controller 24 determines that the gripping mechanism 21 has not normally gripped the article 3 (step S15, NO), the controller 24 transmits an error indicating that the gripping has not been normally performed to the information processing apparatus 12. (Step S18), the process proceeds to step S11.

When the end instruction is supplied from the information processing device 12, the controller 24 of the robot arm 11 moves the arm mechanism 22 and the gripping mechanism 21 to predetermined positions, and ends the process of FIG.

Next, the operation of the information processing apparatus 12 will be described.
FIG. 5 is a flowchart for explaining the operation of the information processing apparatus 12.

The control unit 42 of the information processing apparatus 12 operates in modes such as an image recognition mode, a manual input mode, and a combined mode. Its mode is indicated, for example, by information in a predetermined area of memory 44. Its own mode is switched by the information input via the communication interface.

The image recognition mode is a mode in which an article area is recognized from a car vehicle image by image recognition and a picking position is determined.

The manual input mode is a mode in which the operator of the operation terminal 13 determines the picking position based on the operation input to the operation terminal 13.

In the combined mode, the article area is recognized from the car car image by image recognition, the picking position is determined, and when the article area cannot be recognized from the car car image by image recognition, the operator of the operation terminal 13 operates the operation terminal. In this mode, the picking position is determined based on the operation input to 13.

First, the control unit 42 of the information processing apparatus 12 determines in which mode it is operating. For example, the control unit 42 determines whether or not it is operating in the image recognition mode (step S21). Further, the control unit 42 determines whether or not it is operating in the manual input mode (step S22). Further, the control unit 42 determines whether or not it is operating in the combined mode (step S23). When the control unit 42 determines that it is not any of the image recognition mode, the manual input mode, and the combined mode (steps S21 to S23, NO), it determines that an error occurs and ends the process.

When the control unit 42 determines that it is operating in the image recognition mode (step S21, YES), the control unit 42 performs processing according to the image recognition mode (step S24), and proceeds to the processing of step S27.

Further, when the control unit 42 determines that it is operating in the manual input mode (step S22, YES), the control unit 42 performs processing according to the manual input mode (step S25), and proceeds to the processing of step S27.

Further, when it is determined that the control unit 42 is operating in the combined mode (step S23, YES), the control unit 42 performs processing according to the combined mode (step S26), and proceeds to the process of step S27.

The control unit 42 determines whether or not the car 2 is empty (step S27). The control unit 42 may be configured to determine whether or not the car 2 has been emptied based on the image, or may be configured to determine whether or not the car 2 has been emptied based on other information. May be. For example, the control unit 42 acquires a car vehicle image, performs image recognition on the car vehicle image, and determines whether or not there is an article that can be sorted based on the detection result of the article area.

FIG. 6 is a flowchart for explaining the operation of the information processing apparatus 12 in the image recognition mode. That is, FIG. 6 is a flowchart corresponding to step S24 in FIG.

In the image recognition mode, the control unit 42 first waits for the car vehicle image to be supplied from the distance sensor 14 (step S31). As described above, the distance sensor 14 acquires a pair of image data by taking an image of the inside of the car 2 on which the article 3 lifted by the robot arm 11 is mounted by a stereo camera. Further, the distance sensor 14 calculates the distance from the distance sensor 14 for each predetermined area (point) on the image data based on the pair of image data, and generates the car vehicle image 51 as the point group data. The distance sensor 14 supplies the generated car vehicle image 51 to the information processing device 12.

FIG. 7 is an explanatory diagram for explaining an example of the car car image 51. As shown in FIG. 7, the distance sensor 14 has a predetermined range including the area where the car 2 is installed as the angle of view. Therefore, the car car 2 and the plurality of articles 3 are reflected in the car car image 51.

The car vehicle image 51 includes information indicating the distance between the surface of the car vehicle 2 and the plurality of articles 3 loaded in the car vehicle 2 and the distance sensor 14. That is, when the height at which the distance sensor 14 is installed, the focal length of the lens used for the distance sensor 14, and the dimensions of the image sensor used for the distance sensor 14 are known, the car vehicle image 51 shows. Based on this, the height from the floor on which the car car 2 is installed to the surface of the plurality of articles 3 can be calculated.

When the control unit 42 of the information processing apparatus 12 acquires the car vehicle image 51 from the distance sensor 14, the control unit 42 performs image recognition on the car vehicle image 51 using the image recognition parameters stored in the memory 44. As a result, the control unit 42 recognizes the article region, which is the region where the article 3 exists in the car vehicle image 51 (step S32). Specifically, the control unit 42 uses an image recognition parameter stored in the memory 44 for the car vehicle image 51 to create an article region by using a technique such as a neural network for object detection or segmentation. recognize. For example, the control unit 42 recognizes a region corresponding to the surface existing at the highest position in the car vehicle image 51 as an article region.

FIG. 8 is an explanatory diagram for explaining an example of the recognition result of the article region in the car car image 51. As shown in FIG. 8, in the car car image 51, the surface of the article 3 existing at the highest position is recognized as the article region 52. As a result, the control unit 42 recognizes the shape of the article 3 to be sorted.

The control unit 42 determines the picking position based on the recognition result of the article region 52 in the car vehicle image 51 (step S33). For example, the control unit 42 determines the coordinates of the center of gravity (or center) of the article region as the picking position.

FIG. 9 is an explanatory diagram for explaining an example of the picking position 53 determined based on the recognition result of the article region 52 of the car vehicle image 51. As shown in FIG. 9, the control unit 42 determines the center of gravity of the article region 52 as the picking position 53.

The control unit 42 generates an operation plan based on the determined picking position 53 (step S34). For example, the control unit 42 generates a locus of the gripping mechanism 21 of the robot arm 11 so as not to come into contact with another article 3 or a car 2 from the initial position to the picking position 53, and indicates an operation of showing the picking position and the locus. Generate a plan. Further, the control unit 42 transmits the generated motion plan to the controller 24 of the robot arm 11 (step S35), and ends the process. As a result, the control unit 42 shifts to the process of step S27 in FIG.

FIG. 10 is a flowchart for explaining the operation of the information processing apparatus 12 in the manual input mode. That is, FIG. 10 is a flowchart corresponding to step S25 in FIG.

In the manual input mode, the control unit 42 first waits for the car vehicle image to be supplied from the distance sensor 14 (step S41). In the manual input mode, the image of the car car in the state before grasping the article 3 to be sorted is referred to as the first car car image. That is, the control unit 42 acquires the first car vehicle image from the distance sensor 14 in step S41. The first car car image is the same as the car car image 51 shown in FIG. 7.

When the control unit 42 acquires the image of the first car, it outputs a guide prompting the input of the picking position (step S42). FIG. 11 shows an example of an input screen 61 for prompting input of a picking position. The control unit 42 draws a character string for prompting input of the picking position and a first car vehicle image on the input screen 61. For example, the control unit 42 transmits the input screen 61 to the operation terminal 13. As a result, the control unit 42 prompts the operator of the operation terminal 13 to input the picking position.

When the control unit 32 of the operation terminal 13 receives the input screen 61 from the information processing device 12, the control unit 32 displays the input screen 61 on the display 36 of the touch panel 33. Further, the control unit 32 generates coordinate information indicating the picking position based on the operation input by the touch sensor 37 being displayed on the input screen 61. The control unit 32 supplies the coordinate information of the picking position to the information processing device 12.

The control unit 42 of the information processing apparatus 12 determines whether or not the picking position has been input in the operation terminal 13 (step S43). When the control unit 42 receives the coordinate information of the picking position from the operation terminal 13, it determines that the picking position has been input in the operation terminal 13. The control unit 42 repeats the determination in step S43 until the coordinate information of the picking position is received from the operation terminal 13.

When the control unit 42 determines that the picking position has been input in the operation terminal 13 (step S43, YES), the control unit 42 generates an operation plan based on the coordinate information of the picking position supplied from the operation terminal 13 (step S44). .. Further, the control unit 42 transmits the generated motion plan to the controller 24 of the robot arm 11 (step S45).

When the operation plan is transmitted to the robot arm 11, the control unit 42 determines whether or not the article 3 is normally gripped by the gripping mechanism 21 of the robot arm 11 (step S46). For example, when the control unit 42 receives a completion notification from the robot arm 11 indicating that the sorting of one article has been completed, the control unit 42 determines that the article 3 has been gripped normally. Further, for example, when the control unit 42 receives an error from the robot arm 11 indicating that the article 3 was not normally gripped by the gripping mechanism 21, it is determined that the article 3 was not normally gripped. do. When the control unit 42 determines that the article 3 has not been gripped normally (step S46, NO), the control unit 42 proceeds to step S41, acquires the first car vehicle image again, and prompts the input of the picking position.

When the control unit 42 determines that the article 3 has been gripped normally (step S46, YES), the control unit 42 acquires the car vehicle image again from the distance sensor 14 (step S47). The fact that the article 3 was normally gripped indicates that the article 3 to be sorted, which was shown in the image of the first car, was moved. The car car image after the article 3 to be sorted is moved in this way is referred to as a second car car image. That is, the control unit 42 acquires the second car vehicle image in step S47.

Next, the control unit 42 detects the difference between the first car vehicle image and the second car vehicle image (step S48). For example, the control unit 42 generates a mask image showing a region where the difference between the first car image and the second car image exists, based on the difference between the first car image and the second car image. .. That is, the control unit 42 generates a mask image based on the difference between the car car image before the article 3 is moved by the robot arm 11 and the car car image after the article 3 is moved.

FIG. 12 is an explanatory diagram for explaining an example of the second car vehicle image 55. As described above, at the time of capturing the second car car image 55, the article 3 to be sorted shown in the first car car image is removed from the car car 2. Therefore, the portion hidden by the article 3 to be sorted is reflected in the second car car image 55. The region where the difference is generated between the first car vehicle image and the second car vehicle image 55 by removing the article 3 to be sorted in this way is referred to as a difference region 54.

FIG. 13 is an explanatory diagram for explaining an example of the mask image 71. In the example of FIG. 13, the mask image 71 is configured as a binary image in which the pixel value of the difference region 54 is “1” and the pixel value of the other region is “0”. The mask image 71 is not limited to such a configuration, and any configuration can be used as long as the region where there is a difference between the first car image and the second car image and the other regions can be distinguished. It may be a configuration. Further, the control unit 42 may be configured to generate a mask image by correcting the contour of a region where there is a difference between the first car vehicle image and the second car vehicle image.

The difference region 54 corresponds to a region to be detected as the article region 52 in the image recognition in step S32 of FIG. That is, the mask image 71 corresponds to the answer to the first car vehicle image in the learning data.

The control unit 42 generates learning data based on the first car vehicle image and the mask image (step S49). That is, the control unit 42 has a problem of the first car car image, and the second car car image and the first car car image taken after the article 3 to be sorted in the first car car image is removed. The training data is generated using the mask image showing the difference from the image as the answer. The control unit 42 transmits the generated learning data to the storage 15 (step S50), and ends the process. As a result, the control unit 42 shifts to the process of step S27 in FIG. When the process of step S25 is executed again, the control unit 42 may use the second car vehicle image acquired in the previous process as the first car vehicle image.

FIG. 14 is a flowchart for explaining the operation of the information processing apparatus 12 in the combined mode. That is, FIG. 14 is a flowchart corresponding to step S26 in FIG.

In the combined mode, the control unit 42 first tries to generate a motion plan by image recognition, and when the motion plan generated does not normally grip the article 3, the control unit 42 generates a motion plan based on manual input.

First, the control unit 42 waits for the car vehicle image to be supplied from the distance sensor 14 (step S61). The car car image in the state before gripping the article 3 to be sorted is referred to as a first car car image.

When the control unit 42 acquires the first car vehicle image from the distance sensor 14, the control unit 42 performs image recognition on the first car vehicle image using the image recognition parameter stored in the memory 44. As a result, the control unit 42 recognizes the article region, which is the region in which the article 3 to be sorted exists, in the first car vehicle image. At this time, the control unit 42 determines whether or not the article area can be recognized (step S62). When the control unit 42 determines that the article area cannot be recognized (step S62, NO), the control unit 42 shifts to the process of step S67 described later.

When the control unit 42 determines that the article region can be recognized (step S62, YES), the control unit 42 determines the picking position based on the recognition result of the article region in the first car vehicle image (step S63).

The control unit 42 generates an operation plan based on the determined picking position 53. At this time, the control unit 42 determines whether or not an operation plan can be generated (step S64). For example, the control unit 42 generates a locus of the gripping mechanism 21 of the robot arm 11 so as not to come into contact with another article 3 or a car 2 from the initial position to the picking position 53, and indicates an operation of showing the picking position and the locus. Generate a plan. If the control unit 42 cannot generate such an operation plan, the control unit 42 determines that the operation plan cannot be generated. When the control unit 42 determines that the motion plan cannot be generated (step S64, NO), the control unit 42 shifts to the process of step S67 described later.

When the control unit 42 determines that the motion plan can be generated (step S64, YES), the control unit 42 transmits the generated motion plan to the controller 24 of the robot arm 11 (step S65).

When the operation plan is transmitted to the robot arm 11, the control unit 42 determines whether or not the article 3 is normally gripped by the gripping mechanism 21 of the robot arm 11 (step S66). When the control unit 42 determines that the article 3 has been gripped normally (step S66, YES), the control unit 42 ends the process. As a result, the control unit 42 shifts to the process of step S27 in FIG.

Further, when it is determined that the article 3 has not been gripped normally (step S66, NO), the control unit 42 outputs a guide prompting the input of the picking position (step S67). For example, the control unit 42 prompts the operator of the operation terminal 13 to input the picking position by transmitting the input screen 61 for prompting the input of the picking position shown in FIG. 11 to the operation terminal 13.

The control unit 42 of the information processing apparatus 12 determines whether or not the picking position has been input in the operation terminal 13 (step S68). When the control unit 42 receives the coordinate information of the picking position from the operation terminal 13, it determines that the picking position has been input in the operation terminal 13. The control unit 42 repeats the determination in step S68 until the coordinate information of the picking position is received from the operation terminal 13.

When the control unit 42 determines that the picking position has been input in the operation terminal 13 (step S68, YES), the control unit 42 generates an operation plan based on the coordinate information of the picking position supplied from the operation terminal 13 (step S69). .. Further, the control unit 42 transmits the generated motion plan to the controller 24 of the robot arm 11 (step S70).

When the operation plan is transmitted to the robot arm 11, the control unit 42 determines whether or not the article 3 is normally gripped by the gripping mechanism 21 of the robot arm 11 (step S71). When the control unit 42 determines that the article 3 has not been gripped normally (step S71, NO), the control unit 42 proceeds to step S67 and prompts the input of the picking position again. The control unit 42 may acquire the first car vehicle image from the distance sensor 14 again when shifting to step S67, and may update the display on the input screen 61 with the acquired first car vehicle image.

When the control unit 42 determines that the article 3 has been gripped normally (step S71, YES), the control unit 42 acquires the car vehicle image again from the distance sensor 14 (step S72). The fact that the article 3 was normally gripped indicates that the article 3 to be sorted, which was shown in the image of the first car, was moved. The car car image after the article 3 to be sorted is moved in this way is referred to as a second car car image. That is, the control unit 42 acquires the second car vehicle image in step S72.

Next, the control unit 42 detects the difference between the first car vehicle image and the second car vehicle image (step S73). For example, the control unit 42 generates a mask image showing a region where the difference between the first car image and the second car image exists, based on the difference between the first car image and the second car image. .. The method of generating the mask image is the same as that described in FIG.

The control unit 42 generates learning data based on the first car vehicle image and the mask image (step S74). That is, the control unit 42 has a problem of the first car car image, and the second car car image and the first car car image taken after the article 3 to be sorted in the first car car image is removed. The training data is generated using the mask image showing the difference from the image as the answer. The control unit 42 transmits the generated learning data to the storage 15 (step S75), and ends the process. As a result, the control unit 42 shifts to the process of step S27 in FIG. When the process of step S26 is executed again, the control unit 42 may use the second car vehicle image acquired in the previous process as the first car vehicle image.

As described above, the sorting system 1 includes a robot arm 11 and an information processing device 12. The robot arm 11 has a gripping mechanism 21 for gripping an article 3 placed in a predetermined range, and an arm mechanism 22 for moving the gripping mechanism 21. The information processing device 12 supplies the picking position to the robot arm 11 and supplies learning data to the learning device 16 that adjusts the image recognition parameters for recognizing the article 3 from the images in a predetermined range.

Further, the control unit 42 of the information processing apparatus 12 moves the article 3 in a predetermined range by supplying the picking position to the robot arm 11 based on the input operation to the operation interface such as the operation terminal 13. Further, the control unit 42 has a first image (first car car image) which is an image of a predetermined range before the article 3 moves, and a second image (a second image) which is an image of a predetermined range after the article 3 moves. 2 The learning data is generated based on the car car image).

Further, the control unit 42 generates a mask image showing an article region in which the article 3 exists on the first car image based on the difference between the first car image and the second car image, and the first car. A car image and a mask image are generated as training data.

According to such a configuration, by instructing the picking position by the operation input, it is possible to acquire the problem and the answer of the learning data while surely moving the article 3 by the robot arm 11. That is, it is possible to collect learning data with little effort while actually operating the sorting system 1. As a result, it is possible to provide the information processing device 12 and the sorting system 1 that can easily collect learning data. Further, since the picking position is instructed by the operation input, the result of accurate processing by the robot arm 11 can be used to generate learning data. This facilitates the collection of accurate containment data.

(Second embodiment)
Next, the second embodiment will be described. The information processing device 12 of the sorting system 1 according to the second embodiment operates in the manual input mode differently from that of the first embodiment. Since each configuration of the sorting system 1 according to the second embodiment is the same as that of the first embodiment, detailed description of the configuration will be omitted.

The control unit 42 of the information processing device 12 of the sorting system 1 according to the second embodiment inputs the article area in which the article 3 exists in the car car image before moving the article. The control unit 42 determines the picking position based on the input article area. Further, the control unit 42 generates learning data using the input article area as an answer with the car vehicle image as a problem, and supplies the generated learning data to the storage 15.

FIG. 15 is a flowchart for explaining the operation of the information processing apparatus 12 in the manual input mode. That is, FIG. 15 is a flowchart corresponding to step S25 in FIG.

In the manual input mode, the control unit 42 first waits for the car vehicle image to be supplied from the distance sensor 14 (step S81).

When the control unit 42 acquires the car vehicle image, the control unit 42 outputs a guide prompting the input of the article area in which the article 3 to be sorted exists on the car vehicle image (step S82). FIG. 16 shows an example of an input screen 81 that prompts input of an article area. The control unit 42 draws a character string prompting the input of the article area and a car car image on the input screen 81. For example, the control unit 42 transmits the input screen 81 to the operation terminal 13. As a result, the control unit 42 prompts the operator of the operation terminal 13 to input the article area.

When the control unit 32 of the operation terminal 13 receives the input screen 81 from the information processing device 12, the control unit 32 displays the input screen 81 on the display 36 of the touch panel 33. Further, the control unit 32 generates information indicating the article area based on the operation input by the touch sensor 37 during the display of the input screen 81. The information of the article area may be the above-mentioned mask image or the locus of coordinates on the car car image. The control unit 32 supplies the information of the article area to the information processing device 12.

The control unit 42 of the information processing apparatus 12 determines whether or not the article area has been input in the operation terminal 13 (step S83). When the control unit 42 receives the information indicating the article area from the operation terminal 13, it determines that the article area has been input in the operation terminal 13. The control unit 42 repeats the determination in step S83 until the information on the article area is received from the operation terminal 13.

When the control unit 42 determines that the article area has been input in the operation terminal 13 (step S83, YES), the control unit 42 determines the picking position based on the input article area (step S84). For example, the control unit 42 determines the center of gravity (or center) of the article region as the picking position.

The control unit 42 generates an operation plan based on the determined picking position (step S85). Further, the control unit 42 transmits the generated motion plan to the controller 24 of the robot arm 11 (step S86).

When the operation plan is transmitted to the robot arm 11, the control unit 42 determines whether or not the article 3 is normally gripped by the gripping mechanism 21 of the robot arm 11 (step S87). For example, when the control unit 42 receives a completion notification from the robot arm 11 indicating that the sorting of one article has been completed, the control unit 42 determines that the article 3 has been gripped normally. Further, for example, when the control unit 42 receives an error from the robot arm 11 indicating that the article 3 was not normally gripped by the gripping mechanism 21, it is determined that the article 3 was not normally gripped. do. When the control unit 42 determines that the article 3 has not been gripped normally (step S87, NO), the control unit 42 proceeds to step S83, acquires the car vehicle image again, and prompts the input of the article area.

When the control unit 42 determines that the article 3 has been gripped normally (step S87, YES), the control unit 42 generates learning data based on the car car image and the information of the article area (that is, the mask image) (step S87, YES). Step S88). That is, the control unit 42 has a problem of the car car image before the article 3 to be sorted is moved, and uses the article area, which is the area where the article 3 to be sorted exists in the car car image, as an answer, and uses the learning data as an answer. Generate. The control unit 42 transmits the generated learning data to the storage 15 (step S89), and ends the process. As a result, the control unit 42 shifts to the process of step S27 in FIG.

In the process of FIG. 10 or FIG. 14, the control unit 42 generates a mask image based on the difference between the first car vehicle image and the second car vehicle image, and the mask image and the first car vehicle image are used. Although it was explained that the training data is generated based on the above, the present invention is not limited to this configuration. The control unit 42 may be configured to determine whether or not to generate learning data based on whether or not the generated mask image is reliable.

For example, the control unit 42 learns based on the difference area (article area) estimated based on the first car vehicle image and the second car vehicle image and the picking position determined by the operation input to the operation terminal 13. It is determined whether or not to generate the learning data, that is, whether or not to supply the learning data to the learning device 16. More specifically, when the difference between the center of gravity (or the center) of the article region and the picking position is less than a preset threshold value, the control unit 42 is for learning based on the first car vehicle image and the mask image. Generate data. Further, when the difference between the substantially center (or the center of gravity) of the article region and the picking position is equal to or more than a preset threshold value, the control unit 42 generates learning data based on the first car vehicle image and the mask image. Do not do.

For example, as shown in FIG. 17, when the article 3 is not moved by the robot arm 11, the article 3 falls into the car 2, and the position of the article 3 is displaced, the image of the first car and the second car The difference from the car image does not correspond to the article area. Further, even when the article 3 is moved by the robot arm 11 but the other article 3 collapses, the difference between the image of the first car and the image of the second car does not correspond to the article area. In such a case, as shown in FIG. 17, the center of gravity 91 of the difference region and the picking position 53 determined based on the input deviate from each other. When the difference between the center of gravity 91 of the difference region and the picking position 53 determined based on the input is equal to or greater than a preset threshold value, the control unit 42 removes the machine from the target of machine learning in the learning device 16. The accuracy of the learning data used for learning can be guaranteed.

Further, the control unit 42 recognizes the article area in which the article 3 to be sorted exists on the first car image based on the difference between the first car image and the second car image, and recognizes the article area. Based on the above, the dimensions of the moved article 3 may be estimated, and the estimation result may be stored in the memory 44. According to such a configuration, it is possible to obtain statistics on the dimensions of the article 3 sorted for each robot arm 11. The control unit 42 calculates the dimension by using the contour of the article area and the pixel value of the car car image. For example, the control unit 42 estimates the vertical and horizontal size of the article 3 from the contour of the article region. Further, the control unit 42 estimates the depth size of the article 3 based on the difference in the pixel values in the article region between the first car image and the second car image.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.
The following is a description of the scope of claims at the time of filing the application.
[C1]
A robot arm including a gripping mechanism for gripping an article placed in a predetermined range and an arm mechanism for moving the gripping mechanism is supplied with picking position data indicating a picking position for gripping the article, and is in the predetermined range. An information processing device that supplies learning data to a learning device that adjusts image recognition parameters for recognizing the article from an image.
Based on the input operation to the operation interface, the article is moved by supplying the picking position data to the robot arm, and the first image which is an image of the predetermined range before the article moves and the article. An information processing device including a control unit that generates learning data based on a second image that is an image in the predetermined range after the movement of the image, and supplies the generated learning data to the learning device.
[C2]
The control unit
Based on the difference between the first image and the second image, a mask image showing an article region in which the article is present on the first image is generated.
The information processing device according to claim 1, wherein the first image and the mask image are supplied to the learning device as the learning data.
[C3]
The information processing device according to claim 2, wherein the control unit determines whether or not to supply the learning data to the learning device based on the article area and the picking position data.
[C4]
The control unit
When the difference between the substantially center of the article region and the picking position data is less than a preset threshold value, the learning data is supplied to the learning device.
The information processing device according to claim 3, wherein when the difference is equal to or larger than a preset threshold value, the learning data is not supplied to the learning device.
[C5]
The control unit
Image recognition is performed on the first image using the image recognition parameter, and the image recognition is performed.
Recognizing the article area where the article is located on the first image,
The picking position data is determined based on the article area, and the picking position data is determined.
By supplying the picking position data to the robot arm, the article is moved.
When the article is not moved normally, the article is moved by supplying the picking position data to the robot arm based on the input operation to the operation interface.
Based on the first image and the second image, learning data is generated.
The information processing device according to claim 1, wherein the generated learning data is supplied to the learning device.
[C6]
The control unit
Based on the difference between the first image and the second image, the article region in which the article exists on the first image is recognized.
The information processing apparatus according to any one of claims 1 to 5, which estimates the dimensions of the article based on the article area and stores the estimation result in a memory.
[C7]
Information is supplied to a robot arm provided with a gripping mechanism for gripping an article placed in a predetermined range and an arm mechanism for moving the gripping mechanism, and image recognition for recognizing the article from an image in the predetermined range is performed. An information processing device that supplies learning data to a learning device that adjusts parameters.
In the first image, which is an image of the predetermined range before the article moves, the article area in which the article exists is specified based on an input operation to the operation interface, and the first image and the article area are designated. An information processing device including a control unit that generates learning data based on the learning data and supplies the generated learning data to the learning device.
[C8]
A robot arm for moving an article placed in a predetermined range and picking position data indicating a picking position for gripping the article are supplied to the robot arm, and image recognition for recognizing the article from an image in the predetermined range is provided. It is a sorting system equipped with an information processing device that supplies learning data to a learning device that adjusts parameters.
The robot arm
A gripping mechanism for gripping the article placed in the predetermined range,
An arm mechanism that moves the gripping mechanism based on the picking position data, and
Equipped with
The information processing device is
Based on the input operation to the operation interface, the article is moved by supplying the picking position data to the robot arm, and the first image which is an image of the predetermined range before the article moves and the article. A sorting system including a control unit that generates learning data based on a second image that is an image in the predetermined range after the robot has moved, and supplies the generated learning data to the learning device.

1 ... Sorting system, 2 ... Car, 3 ... Article, 4 ... Belt conveyor, 11 ... Robot arm, 12 ... Information processing device, 13 ... Operation terminal, 14 ... Distance sensor, 15 ... Storage, 16 ... Learning device, 17 ... Network, 21 ... Grip mechanism, 22 ... Arm mechanism, 23 ... Contact sensor, 24 ... Controller, 31 ... Communication interface, 32 ... Control unit, 33 ... Touch panel, 34 ... Processor, 35 ... Memory, 36 ... Display, 37 ... Touch Sensor, 41 ... communication interface, 42 ... control unit, 43 ... processor, 44 ... memory.

Claims (5)

A robot arm including a gripping mechanism for gripping an article placed in a predetermined range and an arm mechanism for moving the gripping mechanism is supplied with picking position data indicating a picking position for gripping the article, and is in the predetermined range. An information processing device that supplies learning data to a learning device that adjusts image recognition parameters for recognizing an article region in which the article exists from an image.
Based on the input operation to the operation interface, the article is moved by supplying the picking position data to the robot arm, and the first image which is an image of the predetermined range before the article moves and the article. A control unit is provided which generates training data based on a second image which is an image in the predetermined range after the robot has moved, and supplies the generated training data to the learning device .
The control unit
Based on the difference between the first image and the second image, a mask image showing the article region in which the article is present on the first image is generated.
The learning data having the first image as a problem and the mask image as the answer is supplied to the learning device.
An information processing device that determines whether or not to supply the learning data to the learning device based on the article area and the picking position data . The control unit
When the difference between the substantially center of the article region and the picking position data is less than a preset threshold value, the learning data is supplied to the learning device.
The information processing device according to claim 1 , wherein the learning data is not supplied to the learning device when the difference is equal to or larger than a preset threshold value. The control unit
Image recognition is performed on the first image using the image recognition parameter, and the image recognition is performed.
Recognizing the article area where the article is located on the first image,
The picking position data is determined based on the article area, and the picking position data is determined.
By supplying the picking position data to the robot arm, the article is moved.
When the article is not moved normally, the article is moved by supplying the picking position data to the robot arm based on the input operation to the operation interface.
Based on the first image and the second image, learning data is generated.
The information processing device according to claim 1, wherein the generated learning data is supplied to the learning device. The control unit
Based on the difference between the first image and the second image, the article region in which the article exists on the first image is recognized.
The information processing apparatus according to any one of claims 1 to 3 , which estimates the dimensions of the article based on the article area and stores the estimation result in a memory. A robot arm for moving an article placed in a predetermined range and picking position data indicating a picking position for gripping the article are supplied to the robot arm, and an article region in which the article exists is recognized from an image of the predetermined range. It is a sorting system equipped with an information processing device that supplies learning data to a learning device that adjusts image recognition parameters for performing.
The robot arm
A gripping mechanism for gripping the article placed in the predetermined range,
An arm mechanism that moves the gripping mechanism based on the picking position data, and
Equipped with
The information processing device is
Based on the input operation to the operation interface, the article is moved by supplying the picking position data to the robot arm, and the first image which is an image of the predetermined range before the article moves and the article. A control unit is provided which generates training data based on a second image which is an image in the predetermined range after the robot has moved, and supplies the generated training data to the learning device .
The control unit
Based on the difference between the first image and the second image, a mask image showing the article region in which the article is present on the first image is generated.
The learning data having the first image as a problem and the mask image as the answer is supplied to the learning device.
A sorting system that determines whether or not to supply the learning data to the learning device based on the article area and the picking position data .

Images