JP7178954B2 - Object sorting system - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act On June 1, 2018, in the Robotics and Mechatronics Lecture 2018 Lecture Proceedings (The Japan Society of Mechanical Engineers), a patent invented by Natsuki Shirai, Satoshi Nakamura, and Yutaka Kumagai was published. We presented the development of robot vision that recognizes piles of standard waste.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object sorting system for sorting out a specific object from a moving group of overlapping objects.

2. Description of the Related Art Conventionally, an object sorting system that sorts out a specific object from a plurality of objects is known (see, for example, Patent Document 1).

Patent Document 1 discloses a recognition device that outputs an output that enables identification of a piece of wood among waste on a steel conveyor, a handling mechanism that can hold the piece of wood on the steel conveyor, and a piece of wood that identifies the piece of wood according to the output of the recognition device. , and a processing unit for controlling a handling mechanism to remove the identified pieces of wood from the steel conveyor.

JP 2015-128763 A

However, since the waste sorting treatment facility of Patent Document 1 cannot sort out specific wastes from a group of overlapping wastes, a vibrating sieve for flattening the piled up form of wastes is installed in the front stage of the recognition device. You have to have the machine.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an object sorting system capable of sorting out an object in the uppermost layer from a group of overlapping objects.

In order to achieve the above object, the object sorting system of the present invention is an object sorting system for sorting out a specific object from a group of overlapping objects moving in a conveying section, wherein a 3D scanner photographs the group of objects. an entire area identification unit that identifies an entire area of the object group based on the image; an edge area generation unit that generates an edge area of an object included in the object group based on the entire area; a region-divided image generation unit that generates a region-divided image based on the region-divided image; a large region calculation unit that calculates a large region having a predetermined area or more based on the region-divided image; and a top layer area calculation unit that calculates a top layer area, and a robot hand unit that selects the top layer area as a specific object.

Here, in the object sorting system of the present invention, the images are a laser luminance image and a range image, and the edge area generation unit generates a range image edge area of the object group extracted based on the range image. , and a luminance image edge region of the object group extracted based on the laser luminance image.

Further, in the object sorting system of the present invention, an area camera arranged adjacent to the 3D scanner extracts the top layer region from a color image of the group of objects, and generates a colored top layer region. a colored top layer area generation unit; and a material determination unit that determines whether or not the colored top layer area is a target material. can be used to sort out specific objects.

Further, in the object sorting system of the present invention, the material determination unit performs color correction on the color image of the object based on a learned model that has performed machine learning using a corrected color image. It may be determined whether the top layer region is the material of interest.

Further, in the object sorting system of the present invention, the learned model may perform machine learning using information obtained by equalizing height information of the object based on the range image of the object.

Further, in the object sorting system of the present invention, a 3D scanner object group height of the object group on the range image photographed by the 3D scanner, and a 3D scanner object group height of the object group on the range image photographed by the 3D scanner. a scanner object group width, a first distance from the area camera to the transport unit, an area camera object group width of the object group on the color image photographed by the area camera, and the distance from the center of the area camera to the Based on a second distance to the center of the object group and a third distance between the center of the object group and the center of the object group photographed by the 3D scanner, the color image and the distance image. An image correcting unit that corrects the position and size of the object group may be provided.

Further, the edge region extraction program of the present invention is an edge region extraction program for extracting an edge region of an object included in a group of objects, wherein the distance of the object extracted from a distance image obtained by photographing the group of objects by a 3D scanner a computer as edge region generation means for generating an edge region of the object by integrating an image edge region and a luminance image edge region of the object extracted from a laser luminance image of the object group photographed by the 3D scanner; It is characterized by functioning.

The object sorting system of the present invention configured in this way includes an entire area specifying unit that specifies the entire area of the object group based on the image of the object group photographed by the 3D scanner, and an object group based on the entire area. An edge region generation unit that generates an edge region of a contained object, a region division image generation unit that generates a region division image based on the edge region, and a large region with a predetermined area or more is calculated based on the region division image. a large area calculation unit that calculates the uppermost layer area based on the calculated large area and the image; and a robot hand unit that selects the uppermost layer area as a specific object. As a result, the object in the uppermost layer can be sorted out from the group of overlapping objects.

Further, in the object sorting system of the present invention, the edge area generation unit generates a distance image edge area of the object group extracted based on the distance image and a luminance image edge area of the object group extracted based on the laser luminance image. , to generate As a result, an edge region that could not be obtained from the laser luminance image can be obtained from the range image. Therefore, the edge region T can be obtained with high accuracy.

Further, in the object sorting system of the present invention, a colored top layer region is extracted from a color image of a group of objects captured by an area camera placed adjacent to the 3D scanner, and a colored top layer region is generated. An upper layer region generation unit and a material determination unit that determines whether or not the colored top layer region is a target material. sort out. As a result, the material of the object in the uppermost layer can be selected from the overlapping object group.

Further, in the object sorting system of the present invention, the material determination unit performs machine learning on the color image of the object using a corrected color image that has undergone color correction. , to determine whether or not it is the target material. As a result, it is possible to emphasize the color features of the target material and increase the difference in features from the surrounding objects, improve the accuracy of the trained model, and increase the accuracy of the determination of the target material by the material determination unit. can be raised.

Further, in the object sorting system of the present invention, the trained model executes machine learning using information obtained by equalizing the height information of the object based on the distance image of the object. This makes it possible to flatten (uniformize) the brightness and improve the accuracy of the trained model.

Further, the object sorting system of the present invention is provided with an image correction unit that corrects the position and size of the object group in the color image and the range image, so that the 2D image and the 3D image are accurately combined. be able to.

Further, in the edge area extraction program of the present invention, the distance image edge area of the object extracted from the distance image of the object group photographed by the 3D scanner and the object extracted from the laser intensity image of the object group photographed by the 3D scanner. The computer is caused to function as an edge region generating means for generating an edge region of an object by integrating the luminance image edge region. As a result, an edge region that could not be obtained from the laser luminance image can be obtained from the distance image, and the edge region can be obtained with high accuracy.

1 is a diagram illustrating the configuration of an object sorting system according to Example 1; FIG. 3A and 3B are diagrams showing a laser luminance image, a distance image, and a color image in Example 1; FIG. 2 is a block diagram showing the functional configuration of the object sorting system of Example 1; FIG. FIG. 4A is a diagram illustrating image processing by the object sorting system of the first embodiment, FIG. 4A is a diagram showing a laser intensity image of a group of objects photographed by a 3D scanner, and FIG. FIG. 4 is a diagram showing an entire area specified by an specifying unit; 5A and 5B are diagrams for explaining image processing by the object sorting system of the first embodiment, FIG. 5A is a diagram showing an edge region generated by an edge region generation unit, and FIG. FIG. 10 is a diagram in which the edge region is superimposed on the range image. 6A and 6B are diagrams for explaining image processing by the object sorting system of Example 1, FIG. FIG. 4 is a diagram showing a large area calculated by a calculator; 7A and 7B are diagrams for explaining image processing by the object sorting system of the first embodiment, FIG. 7A is a diagram showing the top layer area calculated by the top layer area calculation unit, It is an explanatory view explaining an area calculation part. FIG. 3 is an explanatory diagram for explaining an image correction unit of Example 1; FIG. 4 is a diagram for explaining image processing by the object sorting system of Example 1, and is a diagram showing a colored top layer region generated by a colored top layer region generation unit; FIG. FIG. 10 is a diagram illustrating uniformity of height information according to the first embodiment; 4A and 4B are diagrams for explaining images used for learning in Example 1; FIG. 4A and 4B are diagrams for explaining color correction according to the first embodiment; FIG. 5 is a flow chart showing the flow of object sorting processing by the object sorting system of embodiment 1;

Hereinafter, an embodiment for realizing an object sorting system according to the present invention will be described based on Example 1 shown in the drawings.

Embodiment 1 will explain an object sorting system that sorts out a piece of wood in the uppermost layer from a plurality of stacked objects (object group). In the first embodiment, an example will be described in which the object is a waste product generated in demolition work of a building or the like.

[Configuration of object sorting system]
FIG. 1 is a diagram for explaining the configuration of an object sorting system according to the first embodiment. 2A and 2B are diagrams showing a laser luminance image, a distance image, and a color image in Example 1. FIG. The configuration of the object sorting system according to the first embodiment will be described below with reference to FIGS. 1 and 2. FIG.

The object sorting system 1 is composed of a transport section 2 , an imaging section 10 , a control section 20 and a sorting section 40 .

(Conveyor)
The transport unit 2 can be, for example, a transport belt such as a belt conveyor. An object group G of a plurality of objects M in an overlapping state is placed on the transport unit 2 . The transport section 2 is configured to circulate.

(Shooting department)
The photographing unit 10 includes a photographing box 13 provided on the conveying unit 2, a 3D scanner 11, and an area camera 12. As shown in FIG.

The photographing box 13 is formed in a cylindrical shape and installed at a predetermined position of the transport section 2 so as to cover the upper portion of the transport section 2 . Inside the photographing box 13, an LED lighting 13a and a reflecting plate 13b for diffusing the light of the LED lighting 13a are attached.

The 3D scanner 11 can be, for example, Gocator2730 manufactured by LIM TECHNOLOGIES. The 3D scanner 11 acquires a distance image and a laser brightness image by a light section method. The 3D scanner 11 is installed above the photography box 13 .

The area camera 12 can be, for example, VCC-5CXP3R manufactured by CIS. The area camera 12 acquires color images. The area camera 12 is installed adjacent to the 3D scanner 11 above the photographing box 13 .

In the photographing unit 10 configured in this way, the group of objects G that is placed on and conveyed by the conveying unit 2 passes through the photographing box 13 while being illuminated by the LED illumination 13a. The image is captured by the camera 12 .

Then, a laser brightness image S1 as shown in FIG. 2(a), a distance image S2 as shown in FIG. 2(b), and a color image S3 as shown in FIG. 2(c) are acquired.

(control part)
The control unit 20 can be a PC (personal computer). Note that the functional configuration of the control unit 20 will be described later.

(Sorting department)
The sorting unit 40 includes a controller 42 that receives information from the control unit 20 via the control panel 41 and a robot hand unit 43 whose driving is controlled by the controller 42 .

The control panel 41 can be, for example, a PLC (Programmable Logic Controller). The controller 42 controls the robot hand section 43 based on information from the control panel 41 .

The robot hand unit 43 can be, for example, a vertically articulated 6-axis robot. The robot hand section 43 is arranged along the transport section 2 . The robot hand unit 43 is arranged downstream of the imaging unit 10 in the transport direction of the transport unit 2 . The robot hand unit 43 is configured to grip the object M that is placed on and transported by the transport unit 2 and to recover the object M to the recovery box 44 .

The object sorting system 1 configured in this way acquires the position of the transport unit 2 by an encoder, and selects the robot hand unit from among the object group G photographed by the photographing unit 10 based on the control information of the control unit 20. 43 selects a particular object M;

[Functional Configuration of Object Sorting System]
FIG. 3 is a block diagram showing the functional configuration of the object sorting system according to the first embodiment; 4 to 7 are diagrams for explaining image processing by the object sorting system of the first embodiment. FIG. 8 is an explanatory diagram for explaining the image correction unit 26 of the first embodiment. FIG. 9 is a diagram for explaining image processing by the object sorting system according to the first embodiment. The functional configuration of the object sorting system according to the first embodiment will be described below with reference to FIGS. 3 to 9. FIG.

The object sorting system 1 mainly includes a 3D scanner 11, an area camera 12, a transport section 2, a control section 20, a controller 42, and a robot hand section 43, as shown in FIG.

The 3D scanner 11 obtains a laser luminance image S1 and a distance image S2 of the object group G being transported by the transport unit 2 by the light section method. The laser brightness image S1 and the distance image S2 are transmitted to the control unit 20. FIG.

The area camera 12 acquires a color image S<b>3 of the object group G being transported by the transport unit 2 . The color image S3 is transmitted to the control section 20. FIG.

The transport unit 2 is connected to an encoder (not shown), and the transport amount of the transport unit 2 is transmitted to the control unit 20 .

The control unit 20 includes an entire area identifying unit 21, an edge area generating unit 22, an area divided image generating unit 23, a large area calculating unit 24, a top layer area calculating unit 25, an image correcting unit 26, and a coloring unit. It includes a top layer region generation unit 27 , a material determination unit 28 , a color correction unit 29 , a height information equalization unit 30 , and a storage unit 31 .

The entire area specifying unit 21 specifies the entire area E of the object group G shown in FIG. 4(b) based on the laser brightness image S1 of the object group G photographed by the 3D scanner 11 shown in FIG. 4(a). The entire area specifying unit 21 sets a predetermined brightness threshold to separate the transport unit 2, which is the background area, from the object group G, which is the waste area, and specifies the entire area E of the object group G.

Note that the entire area specifying unit 21 may specify the entire area E of the object group G based on the distance image S2 obtained by capturing the object group G by the 3D scanner 11 . However, it is preferable for the entire area specifying unit 21 to specify the entire area E of the object group G based on the laser luminance image S1, because the color difference between the conveying unit 2, which is the background, and the object group G is more likely to occur.

As shown in FIG. 5(a), the edge region generation unit 22 uses filters such as a canny filter, a sobel filter, and a lanser filter to extract the edge region of the object M included in the object group G from the entire region E. Generate T.

The edge area generation unit 22 generates a distance image edge area Ta and a luminance image edge area Tb as edge areas T. FIG. The edge area generation unit 22 generates a distance image edge area Ta of the object M extracted based on the distance image S2 and a luminance image edge area Tb of the object M extracted based on the laser luminance image S1. ,Integrate.

Further, the edge region generator 22 extracts the central axis of the edge region T. FIG. The edge area generator 22 functions as an edge area generator that generates and integrates the distance image edge area Ta and the luminance image edge area Tb.

The segmented image generator 23 overlaps the integrated edge region T on the distance image S2 as shown in FIG. 5(b). As shown in FIG. 6A, the segmented image generation unit 23 segments the edge region T superimposed on the distance image S2 into a plurality of segmented regions M1 having similar characteristics by the region growing method. to generate the segmented image P1.

Note that the segmented image generation unit 23 may generate segmented images by performing segmentation using a region growing method based on the edge region T superimposed on the laser luminance image S1. However, if the segmented image P1 is generated based on the edge region T superimposed on the distance image S2, the color information of the surface of the object M is not affected, and segmentation can be performed with high accuracy.

As shown in FIG. 6(b), the large area calculator 24 calculates a large area M2 having a predetermined area or more based on the divided area image P1. The predetermined area refers to a size that can be grasped by the robot hand section 43 . Note that the large area calculation unit 24 may calculate a large area having a predetermined width or more based on the area-divided image P1.

As shown in FIG. 7A, the top layer area calculator 25 calculates the top layer area M3 based on the large area M2 and the distance image S2. Specifically, first, the large area M2 is superimposed on the distance image S2. Next, as shown in FIG. 7B, for each large region M2, a point P1 of the object M and a point P2 of the large region M2, which are at the minimum distance F from the surrounding object M, are extracted. Next, the average height of the area A1 of the object M surrounded by the constant radius R centered at the point P1 and the average height of the area A2 of the large area M2 surrounded by the constant radius R centered on the point P2 and compare.

When the average height of the area A2 of the large area M2 is higher than the average height of the area A1 of all the objects M surrounding the large area M2, the top layer area calculator 25 determines the large area M2 as the top layer area. M3 is determined.

On the other hand, when the average height of the area A2 of the large area M2 is lower than the average height of the area A1 of the object M surrounding the large area M2, the top layer area calculator 25 determines the large area M2 as the top layer area. It is judged that it is not M3. In this way, a second distance image P2 is generated in which the top layer region M3 is reflected in the distance image S2.

By the way, the distance image S2 has a shadowed area that cannot be obtained. By defining the distance between the large region M2 and the object M in the range image S2 surrounding it, even if there is a shadowed region that cannot be obtained, the top layer region calculation unit 25 calculates the top layer region M3. can be calculated.

A second distance image P2 in which the uppermost layer region M3 is reflected in the distance image S2 is used for machine learning (deep learning) described later and determination of the posture of the robot hand unit 43. FIG.

By the way, between the color image S3 acquired from the area camera 12 and the distance image S2 and the laser brightness image S1 acquired from the 3D scanner 11, the size and position of the photographed object are shifted due to the difference in the principle of photographing.

The image correction unit 26 corrects the size and positional deviation of the object between the 2D (two-dimensional) image and the 3D (three-dimensional) image. Specifically, as shown in FIG. 8A, the height of the object group G on the distance image S2 captured by the 3D scanner 11 is set as the 3D scanner object group height h. The width of the object group G on the distance image S2 captured by the 3D scanner 11 is defined as the 3D scanner object group width w.

As shown in FIG. 8B, the distance from the area camera 12 to the transport unit 2 is defined as a first distance WD. The width of the object group G on the color image S3 captured by the area camera 12 is defined as the area camera object group width w1. The distance from the center 12a of the area camera 12 to the center Ga of the object group G is defined as a second distance d. The distance (shift amount) between the center Ga of the object group G and the center Gb of the object group G captured by the 3D scanner 11 is defined as a third distance d1.

The area camera object group width w1 and the third distance d1 are calculated by the following equations.
w1=w×WD/(WD−h) Formula 1
d1=d×WD/(WD−h)−d Formula 2

The image correction unit 26 refers to Equations 1 and 2 to enlarge the uppermost layer region M3 obtained from the 3D scanner 11 by w1/w times, obtain the third distance d1, perform affine transformation, and obtain the size and Correct the position.

That is, the image correction unit 26 determines the 3D scanner object group height h of the object group G on the distance image S2 photographed by the 3D scanner 11 and the 3D scanner object group G of the object group G on the distance image S2 photographed by the 3D scanner 11. The object group width w, the first distance WD from the area camera 12 to the transport unit 2, the area camera object group width w1 of the object group G on the color image S3 captured by the area camera 12, and the center of the area camera 12 12a, based on the second distance d to the center Ga of the object group G and the third distance d1 between the center Ga of the object group G and the center Gb of the object group G photographed by the 3D scanner 11, color The position and size of the object group G in the image S3 and the distance image S2 are corrected.

As shown in FIG. 9, the colored uppermost layer region generation unit 27 generates a color image S3 of the object group G photographed by the area camera 12, in which the position and size of the object group G are corrected by the image correction unit 26. The top layer region M3 is extracted to generate a colored top layer region M4. The colored uppermost layer region M4 is used for material determination by the material determination unit 28, but may be used for machine learning, which will be described later.

The material determination unit 28 uses the learned model 32 stored in the storage unit 31 to determine whether or not the colored top layer region M4 is wood, which is the target material. The trained model 32 is generated by pre-learning using material-labeled images (teacher data). The material can be wood, plastic, concrete, glass, or the like.

Specifically, the learning software uses a convolutional neural network (CNN) to perform material determination learning with the labeled image of the object M, thereby generating the trained model 32 . Note that the learned model 32 may be updated using extracted data of the colored uppermost layer region M4 obtained after operation.

Determination information from the material determination unit 28 is transmitted to the controller 42 via the control panel 41 . The determination information includes information on the colored top layer region M4 made of wood, information on the orientation of the robot hand unit 43 calculated from the normal vector of the surface of the colored top layer region M4 made of wood, and information on the position of the center of gravity of the colored top layer region M4.

The controller 42 controls the operation of the robot hand section 43 based on the determination information from the material determination section 28 . That is, the robot hand unit 43 selects the colored uppermost layer region M4, which is the target material, as the specific object M. As shown in FIG.

[Machine learning]
FIG. 10 is a diagram illustrating uniformization of height information according to the first embodiment. 11A and 11B are diagrams for explaining images used for learning in Example 1. FIG. FIG. 12 is a diagram for explaining color correction according to the first embodiment. Machine learning according to the first embodiment will be described below with reference to FIGS. 10 to 12. FIG.

(Equalization of height information)
FIG. 10(a) is a cross-sectional view passing through the center point of a rectangle formed to circumscribe the object M in plan view. As shown in FIG. 10A, the height of the highest point a at the edge of the object M is the highest height Ha, the height of the lowest point b is the lowest height Hb, and the height of the center c of the object M is is the central height Hc.

The height information equalization unit 30 subtracts the maximum height Ha from the maximum point a, subtracts the minimum height Hb from the minimum point b, and obtains a corrected central height Hcnew, which is the height of the central portion c after correction. to equalize the height information.

The corrected center height Hcnew is calculated by the following formula.
(Ha-Hb): ab = (Ha-Hc'): ac
Hc′=Ha−ac/ab*(Ha−Hb)
Hc new = Hc - Ha + ac/ab * (Ha - Hb)

The distance ab is the horizontal distance between the highest point a and the lowest point b. The distance ac is the horizontal distance between the highest point a and the central portion c. The height Hc' is an intermediate height between the maximum height Ha and the minimum height Hb. Note that the height information uniformizing unit 30 may calculate Hcnew for a plurality of points for one object M. FIG.

The height information equalization unit 30 transmits information obtained by equalizing the height information of the object M to the storage unit 31 based on the distance image S2 of the object M. FIG. Then, the learned model 32 that is referred to by the material determination unit 28 is generated using the information obtained by equalizing the height information as input data.

(color correction)
As shown in FIG. 11, the color correction unit 29 enhances the color features of the RGB image by image processing. Specifically, as shown in FIG. 12(b), the color correction unit 29 multiplies the color image of wood as the object M by, for example, a color filter of wood color, thereby enhancing the characteristics of the wood color. A color image V1 is generated.

As shown in FIG. 12(c), the color correction unit 29 multiplies the color image of the wood as the object M by, for example, the color of the image itself to generate a corrected color image V2 in which the color of the image itself is enhanced. . As shown in FIG. 12(d), the color correction unit 29 multiplies the color image of the wood as the object M by, for example, a color filter of the complementary color of the wood, and reduces the saturation of the color of the wood only to obtain a corrected color image. Generate V3.

Note that FIG. 12(a) shows the color image of the object M or the original image V0 of the colored top layer region M4. The color correction unit 29 can thus generate the trained model 32 with high determination accuracy by correcting the colors and enhancing the color features of the RGB image by image processing.

Also, the color correction unit 29 may set an optimum Multi to improve the determination accuracy of the learned model 32 . Assuming that the gray value of the original image is G1, the gray value of the filter is G2, the gray value after multiplication is G3, and the multiplication coefficient is Multi, the multiplication formula of the filter is as follows.
G3=(G1+G2)*Multi

FIG. 12(e) is a color image W0 of wood of the original object M, and FIG. 12(f) is a color image W1 of wood of the object M when Multi is 0.3. (g) is a color image W2 of wood of object M when Multi is 0.6, and FIG. 12(f) is a color image W3 of wood of object M when Multi is 0.9.

(multi-channel)
When generating the learned model 32, as shown in FIG. 11, in addition to using color images of three channels of RGB, channels of the distance image S2 acquired by the 3D scanner 11 and the laser brightness image S1 are added. Therefore, machine learning may be performed on 4-channel or 5-channel images. As a result, the accuracy of the learned model 32 can be improved by taking into account the unevenness information on the surface of the object M.

[Flow of object sorting process]
FIG. 13 is a flow chart showing the flow of object sorting processing by the object sorting system of the first embodiment. The flow of the object sorting process according to the first embodiment will be described below with reference to FIG. 13 .

As shown in FIG. 13, when the object sorting process is started, the entire area specifying unit 21 specifies the entire area E of the object group G based on the laser brightness image S1 obtained by capturing the object group G by the 3D scanner 11 ( step S101).

Next, the edge region generator 22 generates an edge region T of the object M included in the object group G based on the entire region E (step S102). At this time, the edge region generation unit 22 generates a distance image edge region Ta of the object M extracted based on the distance image S2 and a brightness image edge region Tb of the object M extracted based on the laser brightness image S1. An edge region T is generated by integration.

Next, the region-divided image generation unit 23 superimposes the edge region T obtained by integrating the distance image edge region Ta and the luminance image edge region Tb on the distance image S2 (step S103).

Next, based on the edge region T superimposed on the distance image S2, the segmented image generating unit 23 segments the segmented region M1 into segmented regions M1 by the region growing method to generate the segmented image P1 (step S104).

Next, the large region calculator 24 calculates a large region M2 having a predetermined area or more based on the segmented image P1 (step S105).

Next, the top layer area calculator 25 calculates the top layer area M3 based on the large area M2 and the distance image S2 (step S106).

Next, the image correction unit 26 corrects the size and positional deviation of the uppermost layer region M3 so that the 2D image and the 3D image are matched (step S107).

Next, the colored top layer region generation unit 27 extracts the top layer region M3 from the color image S3 of the object group G photographed by the area camera 12, which has been corrected by the image correction unit 26, and extracts the top layer region M3. M4 is generated (step S108).

Next, the material determination unit 28 uses the information of the learned model 32 stored in the storage unit 31 to determine whether or not the colored top layer region M4 is wood, which is the target material (step S109). If it is determined that the colored top layer region M4 is not made of wood (NO in step S109), the object sorting process ends. On the other hand, if it is determined that the colored top layer region M4 is wood (YES in step S109), the process proceeds to step S110.

Next, the control unit 20 calculates the posture of the robot hand unit 43 from the normal vector of the colored uppermost layer region M4 determined as wood (step S110). The posture of the robot hand unit 43 is a posture for gripping the collared uppermost layer region M4 determined as wood.

Next, the control unit 20 detects the center-of-gravity position of the colored uppermost layer region M4 made of wood (step S111).

Next, the control unit 20 transmits the posture of the robot hand unit 43 and the center-of-gravity position of the collared uppermost layer region M4 made of wood to the controller 42 via the control panel 41 . Then, the controller 42 causes the robot hand unit 43 to grasp the colored uppermost layer area M4 of the timber to collect it in the collection box 44 (step S112), and ends the object sorting process.

[Action of Object Sorting System]
Next, the operation of the object sorting system 1 of the first embodiment will be described. An object sorting system 1 according to the first embodiment is an object sorting system that sorts out a specific object M from a group of objects G in a state of being superimposed and moved by a conveying unit 2 . The object selection system 1 includes an entire area specifying unit 21 that specifies an entire area E of the object group G based on an image (laser brightness image S1) of the object group G captured by the 3D scanner 11, and based on the entire area E , an edge region generation unit 22 for generating an edge region T of an object M included in an object group G, a region division image generation unit 23 for generating a region division image P1 based on the edge region T, and the region division image P1. a large region calculation unit 24 that calculates a large region M2 having a predetermined area or more based on and a robot hand unit 43 that selects the uppermost layer region M3 as a specific object M (FIG. 3).

As a result, the object M on the uppermost layer can be sorted out from the object group G in the overlapping state. Therefore, the object M in the uppermost layer can be sorted out without going through a pretreatment process for breaking down the object group G in the overlapping state and making it into a non-overlapping state. As a result, it can be easily introduced into existing lines.

In the object sorting system 1 of the first embodiment, the images are the laser luminance image S1 and the distance image S2, and the edge area generator 22 generates the distance image edge area Ta of the object M extracted based on the distance image S2 and the , and a brightness image edge region Tb of the object M extracted based on the laser brightness image S1 (FIG. 5).

Thereby, the distance image edge area Ta obtained from the distance image S2 and the luminance image edge area Tb obtained from the laser luminance image S1 can be used. Therefore, an edge region that could not be obtained from the laser luminance image S1 can be obtained from the distance image S2. As a result, the edge region T can be obtained with high accuracy.

Incidentally, when the object group G is transported by the transport unit 2, it becomes difficult to extract the edge region T of the object M having the same color as that of the transport unit 2 from the laser luminance image S1. On the other hand, in Example 1, when the object group G is transported by the transport unit 2, the edge region T of the object M having the same color as that of the transport unit 2 is extracted from the distance image S2. be able to.

In the object sorting system 1 of the first embodiment, the top layer region M3 is extracted from the color image S3 of the object group G photographed by the area camera 12 arranged adjacent to the 3D scanner 11, and the colored top layer region M4 is obtained. and a material determination unit 28 that determines whether or not the colored top layer region M4 is a target material. A specific object M is sorted out based on the determination information of 28 (FIG. 3).

Thereby, the material of the object M in the uppermost layer can be sorted out from the object group G in the overlapping state. Therefore, the material of the object M in the uppermost layer can be sorted out without collapsing the object group G in the overlapping state so that the object group G does not overlap.

In the object sorting system 1 of the first embodiment, the material determination unit 28 performs machine learning on the color image S3 of the object M using the corrected color images V1 to V3 that have undergone color correction, based on the learned model 32. , colored uppermost layer region M4 is determined whether or not it is the target material (FIG. 11).

As a result, the color feature of the target material can be emphasized, and the feature difference from the surrounding object M can be increased. Therefore, the accuracy of the learned model 32 can be improved, and the accuracy of determination of the target material by the material determination unit 28 can be increased.

In the object sorting system 1 of Example 1, the trained model 32 performs machine learning using information obtained by equalizing the height information of the object M based on the distance image S2 of the object M (FIG. 10).

By the way, in the object group G in which the objects M are overlapped, the brightness of the object M at the top is increased by the height of the object M.

In Example 1, the height information of the object M can be made uniform. Therefore, it is possible to flatten (uniformize) the brightness and improve the accuracy of the learned model 32 . As a result, the accuracy of determination of the target material by the material determination unit 28 can be improved.

In the object sorting system 1 of the first embodiment, the 3D scanner object group height h of the object group G on the range image S2 captured by the 3D scanner 11 and the object group G on the range image S2 captured by the 3D scanner 11 3D scanner object group width w, first distance WD from area camera 12 to transport unit 2, area camera object group width w1 of object group G on color image S3 captured by area camera 12, area camera 12 Based on the second distance d from the center 12a of the object group G to the center of the object group G, the center Ga of the object group G, and the third distance d1 between the center Ga of the object group G captured by the 3D scanner 11, An image correction unit 26 for correcting the position and size of the object group G in the color image S3 and the distance image S2 is provided (FIG. 8).

By the way, in the 2D color image S3 taken by the area camera 12, the object M may appear large or the coordinates of the center of the object M may be shifted due to perspective. Therefore, the 2D color image S3 captured by the area camera 12 is out of alignment with the 3D distance image S2 and laser brightness image S1 captured by the 3D scanner 11 .

On the other hand, in Example 1, it is possible to correct the size deviation of the object M and the deviation of the center position of the object M between the 2D color image S3 and the 3D range image S2 or laser luminance image S1. Therefore, a 2D image and a 3D image can be combined with high accuracy.

The edge area extraction program of the first embodiment is an edge area extraction program for extracting the edge area T of the object M included in the object group G, and is extracted from the distance image S2 obtained by photographing the object group G A depth image edge region Ta of the object M and a luminance image edge region Tb of the object M extracted from the laser luminance image S1 of the object group G photographed by the 3D scanner 11 are integrated to generate an object group edge region T. The computer is made to function as an edge region generation means (FIG. 3).

Thereby, the distance image edge area Ta obtained from the distance image S2 and the luminance image edge area Tb obtained from the laser luminance image S1 can be used. Therefore, an edge region that could not be obtained from the laser luminance image S1 can be obtained from the distance image S2. As a result, the edge region T can be obtained with high accuracy, and the object in the uppermost layer can be accurately selected from the group of overlapping objects.

The object sorting system of the present invention has been described above based on the first embodiment. However, the specific configuration is not limited to this embodiment, and design changes, additions, etc. are permitted as long as they do not deviate from the gist of the invention according to the claims.

In the first embodiment, the robot hand unit 43 selects the colored uppermost layer region M4, which is the target material determined by the material determination unit 28, as a specific object. However, the robot hand section may select the uppermost layer region M3 as a specific object.

In Example 1, the trained model 32 showed an example generated by pre-learning. However, the trained model may be updated using extracted data of the colored top layer region M4 obtained after operation.

In the first embodiment, wood is used as the material determined by the material determination unit 28 . However, the material determined by the material determining unit can be resin, slate material, concrete, glass, reinforcing steel, or the like.

In Example 1, an example in which the conveying section 2 is a belt conveyor is shown. However, the conveying unit is not limited to a belt conveyor, and may be, for example, a roller conveyer or any other unit capable of conveying a group of objects.

Embodiment 1 shows an example in which the 3D scanner 11 is of the light section method. However, the 3D scanner can also be of the optical projection type.

In the first embodiment, an example is shown in which the robot hand unit 43 is a vertically articulated 6-axis robot. However, the robot hand section may be a parallel link robot. Also, there may be a plurality of robot hand units.

In Example 1, an example was shown in which the present invention is applied to an object sorting system that sorts out the top layer of wood from a plurality of stacked wastes. However, the present invention can be applied to inspection of bar arrangement in an overlapped state, finished shape control, and the like.

1 Object Sorting System 2 Conveying Unit 11 3D Scanner 12 Area Camera 21 Entire Area Specifying Unit 22 Edge Area Generating Unit 23 Area Divided Image Generating Unit 24 Large Area Calculating Unit 25 Top Layer Area Calculating Unit 26 Image Correcting Unit 27 Colored Top Layer Area Generation unit 28 Material determination unit 32 Trained model 43 Robot hand unit E Whole area G Object group M Object M2 Large area M3 Top layer area P1 Area divided image S1 Laser intensity image (an example of an image)
S2 distance image (an example of an image)
S3 Color image T Edge area Ta Range image edge area Tb Luminance image edge area M4 Colored uppermost layer area V1 to V3 Corrected color image

Claims (5)

An object sorting system for sorting out a specific object from a group of overlapping objects moving in a conveying unit,
an entire area identifying unit that identifies an entire area of the object group based on an image of the object group captured by a 3D scanner;
an edge region generation unit that generates an edge region of an object included in the object group based on the entire region;
a region-divided image generation unit that generates a region-divided image based on the edge region;
a large area calculation unit that calculates a large area having a predetermined area or more based on the area-divided image;
a top layer region calculation unit that calculates a top layer region based on the large region and the image;
a robot hand unit that sorts out the top layer region as a specific object,
The images are a laser luminance image and a range image,
The edge region generation unit
a distance image edge region of the object extracted based on the distance image;
An object sorting system, wherein an edge region of the object is generated by integrating a brightness image edge region of the object extracted based on the laser brightness image. a colored top layer region generation unit that extracts the top layer region from a color image of the object group captured by an area camera arranged adjacent to the 3D scanner and generates a colored top layer region;
a material determination unit that determines whether the colored top layer region is a target material,
The object sorting system according to claim 1, wherein the robot hand section sorts out a specific object based on determination information from a material determination section. The material determination unit determines that the colored top layer region is a target material based on a trained model that has performed machine learning using a corrected color image that has undergone color correction on the color image of the object. 3. The object sorting system according to claim 2, characterized by determining whether or not. 4. The object sorting system according to claim 3, wherein the trained model executes machine learning using information obtained by equalizing the height information of the object based on the distance image of the object. . a 3D scanner object group height of the object group on the range image captured by the 3D scanner;
a 3D scanner object group width of the object group on the range image captured by the 3D scanner;
a first distance from the area camera to the transport unit;
an area camera object group width of the object group on the color image captured by the area camera;
a second distance from the center of the area camera to the center of the object group;
Based on the third distance between the center of the object group and the center of the object group photographed by the 3D scanner,
5. The object sorting system according to any one of claims 2 to 4, further comprising an image correction unit that corrects the position and size of the object group in the color image and the distance image.

Images